Duplicate mounting of `/run/secrets` not handled

[bergmannf]

ISSUE TYPE

• Bug Report

container.yml

version: "2"
settings:

  conductor:
    # The Conductor container does the heavy lifting, and provides a portable
    # Python runtime for building your target containers. It should be derived
    # from the same distribution as you're building your target containers with.
    base: centos:7
    # roles_path:   # Specify a local path containing Ansible roles
    # volumes:      # Provide a list of volumes to mount
    # environment:  # List or mapping of environment variables

  # Set the name of the project. Defaults to basename of the project directory.
  # For built services, concatenated with service name to form the built image name.
  project_name: container-test

  # The deployment_output_path is mounted to the Conductor container, and the 
  # `run` and `deployment` commands then write generated Ansible playbooks to it.
  # deployment_output_path: ./ansible-deployment

  # When using the k8s or openshift engines, use the following to authorize with the API.
  # Values set here will be passed to the Ansible modules. Any file paths will be mounted
  # to the conductor container, allowing the `run` command to access the API.
  #k8s_auth:
    # path to a K8s config file
    #config_file:
    # name of a context found within the config file
    #context:
    # URL for accessing the K8s API
    #host:
    # An API authentication token
    #api_key:
    # Path to a ca cert file
    #ssl_ca_cert:
    # Path to a cert file
    #cert_file:
    # Path to a key file
    #key_file:
    # boolean, indicating if SSL certs should be validated
    #verify_ssl:

  # When using the k8s or openshift engines, use the following to set the namespace.
  # If not set, the project name will be used. For openshift, the namespace maps to a project,
  # and description and display_name are supported.
  #k8s_namespace:
  #  name:
  #  description:
  #  display_name:

services: {}
  # Add your containers here, specifying the base image you want to build from.
  # To use this example, uncomment it and delete the curly braces after services key.
  # You may need to run `docker pull ubuntu:trusty` for this to work.

  # web:
  #   from: "centos:7"
  #   ports:
  #     - "80:80"
  #   command: ["/usr/bin/dumb-init", "/usr/sbin/apache2ctl", "-D", "FOREGROUND"]
  #   dev_overrides:
  #     environment:
  #       - "DEBUG=1"
registries: {}
  # Add optional registries used for deployment. For example:
  #  google:
  #    url: https://gcr.io
  #    namespace: my-cool-project-xxxxxx 

OS / ENVIRONMENT

Ansible Container, version 0.9.2
Linux, gitlab-ci, 4.4.59-92.17-default, #1 SMP Thu Apr 6 14:16:09 UTC 2017 (7bc489d), x86_64
2.7.13 (default, Jan 11 2017, 10:56:06) [GCC] /home/gitlab-runner/container/bin/python
{
  "ContainersPaused": 0, 
  "Labels": null, 
  "CgroupDriver": "cgroupfs", 
  "ContainersRunning": 1, 
  "ContainerdCommit": {
    "Expected": "422e31ce907fd9c3833a38d7b8fdd023e5a76e73", 
    "ID": ""
  }, 
  "InitBinary": "", 
  "NGoroutines": 30, 
  "Swarm": {
    "Managers": 0, 
    "ControlAvailable": false, 
    "NodeID": "", 
    "Cluster": {
      "Spec": {
        "Labels": null, 
        "TaskDefaults": {}, 
        "Orchestration": {}, 
        "EncryptionConfig": {
          "AutoLockManagers": false
        }, 
        "Raft": {
          "HeartbeatTick": 0, 
          "ElectionTick": 0
        }, 
        "CAConfig": {}, 
        "Dispatcher": {}
      }, 
      "Version": {}, 
      "ID": "", 
      "CreatedAt": "0001-01-01T00:00:00Z", 
      "UpdatedAt": "0001-01-01T00:00:00Z"
    }, 
    "Nodes": 0, 
    "Error": "", 
    "RemoteManagers": null, 
    "LocalNodeState": "inactive", 
    "NodeAddr": ""
  }, 
  "LoggingDriver": "json-file", 
  "OSType": "linux", 
  "HttpProxy": "", 
  "Runtimes": {
    "oci": {                                                                                                                                                                         [51/1680]
      "path": "/usr/bin/docker-runc"
    }, 
    "runc": {
      "path": "docker-runc"
    }
  }, 
  "DriverStatus": [
    [
      "Build Version", 
      "Btrfs v3.18.2+20150430"
    ], 
    [
      "Library Version", 
      "101"
    ]
  ], 
  "OperatingSystem": "SUSE Linux Enterprise Server 12 SP2", 
  "Containers": 16, 
  "HttpsProxy": "", 
  "BridgeNfIp6tables": true, 
  "MemTotal": 6257119232, 
  "SecurityOptions": [
    "name=apparmor"
  ], 
  "Driver": "btrfs", 
  "IndexServerAddress": "https://index.docker.io/v1/", 
  "ClusterStore": "", 
  "InitCommit": {
    "Expected": "949e6facb77383876aeff8a6944dde66b3089574", 
    "ID": "N/A"
  }, 
  "Isolation": "", 
  "SystemStatus": null, 
  "OomKillDisable": true, 
  "ClusterAdvertise": "", 
  "SystemTime": "2017-10-10T13:53:08.139044136+02:00", 
  "Name": "smash-ci", 
  "CPUSet": true, 
  "RegistryConfig": {
    "InsecureRegistryCIDRs": [
      "127.0.0.0/8"
    ], 
    "IndexConfigs": {
      "docker.io": {
        "Official": true, 
        "Name": "docker.io", 
        "Secure": true, 
        "Mirrors": []
      }
    },
    "Mirrors": []
  }, 
  "DefaultRuntime": "runc", 
  "ContainersStopped": 15, 
  "NCPU": 2, 
  "NFd": 25, 
  "Architecture": "x86_64", 
  "KernelMemory": false, 
  "CpuCfsQuota": true, 
  "Debug": false, 
  "ID": "72KW:ACAE:6PW2:3QLC:5FXM:5PRZ:ZZX2:KGLZ:S43P:QIMX:HJMA:5DW7", 
  "IPv4Forwarding": true, 
  "KernelVersion": "4.4.59-92.17-default", 
  "BridgeNfIptables": true, 
  "NoProxy": "", 
  "LiveRestoreEnabled": false, 
  "ServerVersion": "17.04.0-ce", 
  "CpuCfsPeriod": true, 
  "ExperimentalBuild": false, 
  "MemoryLimit": true, 
  "SwapLimit": false, 
  "Plugins": {
    "Volume": [
      "local"
    ], 
    "Network": [
      "bridge", 
      "host", 
      "macvlan", 
      "null", 
      "overlay"
    ], 
    "Authorization": []
  }, 
  "Images": 38, 
  "DockerRootDir": "/mnt/containers/docker", 
  "NEventsListener": 0, 
  "CPUShares": true, 
  "RuncCommit": {
    "Expected": "9c2d8d184e5da67c95d601382adf14862e4f2228", 
    "ID": "N/A"
  }
}
{
  "KernelVersion": "4.4.59-92.17-default", 
  "Arch": "amd64", 
  "BuildTime": "2017-05-30T18:21:18.905221620+00:00", 
  "ApiVersion": "1.28", 
  "Version": "17.04.0-ce", 
  "MinAPIVersion": "1.12", 
  "GitCommit": "78d1802", 
  "Os": "linux", 
  "GoVersion": "go1.7.5"
}

SUMMARY

I stumbled over a problem of ansible-container not handling an already existing mount in /run/secrets:

Using SUSE Linux Enterprise Server there are patches in place that already inject a /run/secrets folder into every running container with no way to prevent it.

It seems that this will lead to an attempt to mount the volume twice when running any ansible-container configuration resulting in an error: APIError: 500 Server Error: Internal Server Error ("linux mounts: Duplicate mount point '/run/secrets'").

I already reported a bug to SUSE, but given that some docker upstream technology (e.g. docker swarm) seems to use the same approach of mounting secrets into /run/secrets it seems viable for ansible-container to handle an already existing mount.

STEPS TO REPRODUCE

In theory reproduction under SLES just requires running any ansible-container file and it will break.

To duplicate it without SLES building the containers with explicitly mounting another directory under /run/secrets will simulate it and produce the same error.

mkdir ansible-container
cd ansible-container
ansible-container init
ansible-container --debug build --with-volumes=/tmp/:/run/secrets/

EXPECTED RESULTS

Instead of trying to mount /run/secrets twice, detect any existing mounts and just mount required secrets in sub-directories.

ACTUAL RESULTS

See above.

A debug log of running ansible-container --debug build:

https://gist.github.com/bergmannf/9905792accdb712011df7175ae4770ae

[chouseknecht]

@bergmannf

Thanks for using Ansible Container, and for taking the time to report this issue.

Ran across this while doing some 0.9.3 testing. There's definitely a conflict with the volume name /run/secrets. To get around it, we changed the name we're using to /docker/secrets, here in docker/engine.py.

If you run from source, the immediate issue should be resolved. Still need to do some testing to make sure that K8s secrets still work as expected.

[dcode]

I'd like to add this also affects when building on latest centos containers. Namely I'm using docker.io/centos/systemd. I'm running on a Fedora 27 host and /run/secrets/ now includes rhsm data.

[grantcurell]

+1 for this issue still being there on CentOS 7.

[grantcurell]

Work around: edit /usr/lib/python2.7/site-packages/container/docker/engine.py For me it was line 210, but you can do a search for `os.path.join(os.sep, 'run', 'secrets') and that's the line you want to change. As @chouseknecht mentioned, changed the word run to docker.

[j00bar]

@grantcurell and @dcode Are you using 0.9.2 or develop?

[dcode]

I tried it with 0.9.2 and the master branch. I couldn't get it to work so I tabled using ansible container for now. Hopefully @grantcurell fix works. I'll revisit it again soon.

[giovtorres]

The workaround worked for me in 0.9.2 on Fedora 27. It was line 201 for me.

[Labibme]

@grantcurell i appreciate your answer, thanks for clarification.. i had same issue and i go to file: /usr/lib/python2.7/site-packages/container/docker/engine.py and change: os.path.join(os.sep, 'run', 'secrets') ==> os.path.join(os.sep, 'docker', 'secrets')