search

test-kitchen / kitchen-docker

A Test Kitchen Driver for Docker
Apache License 2.0
462 stars 232 forks source link

Unable to map docker.sock in as volume for docker-in-docker #385

Closed NinjaSomething closed 2 years ago

NinjaSomething commented 2 years ago

:ghost: Brief Description

I am unable to map /var/run/docker.sock into the docker instance as a volume. I'm trying to do docker-in-docker.

Version

2.11.0

Environment

Pop!_OS 20.04 LTS

Scenario

I'm trying to run docker-in-docker. To do this I'm mapping /var/run/docker.sock into the kitchen-docker instance.

Steps to Reproduce

kitchen.yml

# -*- coding: utf-8 -*-
# vim: ft=yaml
---
# For help on this file's format, see https://kitchen.ci/
driver:
  name: docker
  use_sudo: false
  privileged: true
  username: root
  volume:
    - /var/run/docker.sock:/run/docker.sock

platforms:
  - name: ubuntu-18.04
    driver_config:
      run_command: /lib/systemd/systemd

suites:
  - name: default

provisioner:
  name: salt_solo
  salt_install: none
  require_chef: false
  salt_copy_filter:
    - .kitchen
    - .git
  run_salt_call: false

verifier:
  name: shell
  remote_exec: false
  command: python3 -m pytest -s -v test/
  reporter:
    - cli

Create docker instance:

$ kitchen create
-----> Starting Test Kitchen (v3.1.0)
-----> Creating <default-ubuntu-1804>...
       Sending build context to Docker daemon    895kB
       Step 1/17 : FROM ubuntu:18.04
        ---> 54919e10a95d
       Step 2/17 : RUN [ ! -f "/sbin/initctl" ] || dpkg-divert --local --rename --add /sbin/initctl                 && ln -sf /bin/true /sbin/initctl
        ---> Using cache
        ---> ab95f2f1e73e
       Step 3/17 : ENV DEBIAN_FRONTEND noninteractive
        ---> Using cache
        ---> 32f63f35d091
       Step 4/17 : ENV container docker
        ---> Using cache
        ---> 04cb6c8503cd
       Step 5/17 : RUN apt-get update
        ---> Using cache
        ---> c9fcb324b36a
       Step 6/17 : RUN apt-get install -y sudo openssh-server curl lsb-release
        ---> Using cache
        ---> d119423e696c
       Step 7/17 : RUN if ! getent passwd root; then                   useradd -d /root -m -s /bin/bash -p '*' root;                 fi
        ---> Using cache
        ---> 9be897039ab8
       Step 8/17 : RUN echo "root ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers.d/root
        ---> Using cache
        ---> 7b85951aa143
       Step 9/17 : RUN echo "Defaults !requiretty" >> /etc/sudoers.d/root
        ---> Using cache
        ---> 203124825de3
       Step 10/17 : RUN mkdir -p /root/.ssh
        ---> Using cache
        ---> ed74bcca5f48
       Step 11/17 : RUN chown -R root /root/.ssh
        ---> Using cache
        ---> 9dab53399ada
       Step 12/17 : RUN chmod 0700 /root/.ssh
        ---> Using cache
        ---> 2e86fae06aa3
       Step 13/17 : RUN touch /root/.ssh/authorized_keys
        ---> Using cache
        ---> 1dbfb112ee7e
       Step 14/17 : RUN chown root /root/.ssh/authorized_keys
        ---> Using cache
        ---> 417d5d289805
       Step 15/17 : RUN chmod 0600 /root/.ssh/authorized_keys
        ---> Using cache
        ---> e48f1c0a8f83
       Step 16/17 : RUN mkdir -p /run/sshd
        ---> Using cache
        ---> 1969651a7f7e
       Step 17/17 : RUN echo ssh-rsa\ AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOh\+nNP8wl0lAwZzsFOdPLOhPWv8fBhsswHeU15Be//SYtfFHmbNCk8dWJLQyx7DOJ/pJYPpH7KO2LyhaWAunwFcMVY1IsfcD24f7MUlIoQtPo3rKrNcm9O98xUj1n/NEuy34Tsogp6AeC9R1hiqYuX4fIv72GjC1BJSyXMSJBNCyDp5aWTlVpEAbPlrTGixofQPhL8H7XpAaVDeqz63WJUSnQrrmAnq5KvWnjdO3LefpdCNsQwvxY/laUw2bIxogxK62PDVRJCWvYfBjGKgdTflmrWcce3y31ghYDdHn/SSp6QyhDYeVrGFneiNH2lLvkEGNNkI7lSePqjyjGv0j\ kitchen_docker_key >> /root/.ssh/authorized_keys
        ---> Using cache
        ---> c230e559e99f
       Successfully built c230e559e99f
       06dd9825dd38c88455758065f37e396a12adf61e7478ae1b9f4c6289e8d62cad
       0.0.0.0:49246
       :::49246
       Waiting for SSH service on localhost:49246, retrying in 3 seconds
       [SSH] Established
       Finished creating <default-ubuntu-1804> (0m3.52s).
-----> Test Kitchen is finished. (0m3.63s)

Check to see if docker.sock was mapped into the docker instance:

$ kitchen login default-ubuntu-1804
Welcome to Ubuntu 18.04.5 LTS (GNU/Linux 5.13.0-7614-generic x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage
This system has been minimized by removing packages and content that are
not required on a system that users do not log into.

To restore this content, you can run the 'unminimize' command.
Last login: Thu Dec  2 18:43:16 2021 from 172.17.0.1

root@06dd9825dd38:~# ls -alt /var
total 56
drwxrwxrwt 1 root root  4096 Dec  2 18:47 tmp
drwxr-xr-x 1 root root  4096 Dec  2 18:47 ..
drwxr-xr-x 1 root root  4096 Nov  9 22:35 lib
drwxr-xr-x 1 root root  4096 Nov  9 22:35 log
drwxr-xr-x 1 root root  4096 Aug 27 07:18 cache
drwxr-xr-x 1 root root  4096 Aug 27 07:18 .
lrwxrwxrwx 1 root root     9 Aug 27 07:16 lock -> /run/lock
drwxrwsr-x 2 root mail  4096 Aug 27 07:16 mail
drwxr-xr-x 2 root root  4096 Aug 27 07:16 opt
lrwxrwxrwx 1 root root     4 Aug 27 07:16 run -> /run
drwxr-xr-x 2 root root  4096 Aug 27 07:16 spool
drwxr-xr-x 2 root root  4096 Apr 24  2018 backups
drwxrwsr-x 2 root staff 4096 Apr 24  2018 local

root@06dd9825dd38:~# ls /run/docker.sock
ls: cannot access '/run/docker.sock': No such file or directory

root@562d45ee9c12:~# ls /var/run/docker.sock
ls: cannot access '/var/run/docker.sock': No such file or directory

Inspect the docker instance:

[
    {
        "Id": "06dd9825dd38c88455758065f37e396a12adf61e7478ae1b9f4c6289e8d62cad",
        "Created": "2021-12-02T18:47:55.362261203Z",
        "Path": "/lib/systemd/systemd",
        "Args": [],
        "State": {
            "Status": "running",
            "Running": true,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 3823113,
            "ExitCode": 0,
            "Error": "",
            "StartedAt": "2021-12-02T18:47:55.659218767Z",
            "FinishedAt": "0001-01-01T00:00:00Z"
        },
        "Image": "sha256:c230e559e99f1fa19d04f023d6240ab7f91d40bbf45c8b43ab0560d728257e40",
        "ResolvConfPath": "/var/lib/docker/containers/06dd9825dd38c88455758065f37e396a12adf61e7478ae1b9f4c6289e8d62cad/resolv.conf",
        "HostnamePath": "/var/lib/docker/containers/06dd9825dd38c88455758065f37e396a12adf61e7478ae1b9f4c6289e8d62cad/hostname",
        "HostsPath": "/var/lib/docker/containers/06dd9825dd38c88455758065f37e396a12adf61e7478ae1b9f4c6289e8d62cad/hosts",
        "LogPath": "/var/lib/docker/containers/06dd9825dd38c88455758065f37e396a12adf61e7478ae1b9f4c6289e8d62cad/06dd9825dd38c88455758065f37e396a12adf61e7478ae1b9f4c6289e8d62cad-json.log",
        "Name": "/defaultubuntu1804-jcarlson-popos-5wojg4ev",
        "RestartCount": 0,
        "Driver": "overlay2",
        "Platform": "linux",
        "MountLabel": "",
        "ProcessLabel": "",
        "AppArmorProfile": "unconfined",
        "ExecIDs": null,
        "HostConfig": {
            "Binds": [
                "/var/run/docker.sock:/run/docker.sock"
            ],
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "json-file",
                "Config": {}
            },
            "NetworkMode": "default",
            "PortBindings": {
                "22/tcp": [
                    {
                        "HostIp": "",
                        "HostPort": ""
                    }
                ]
            },
            "RestartPolicy": {
                "Name": "no",
                "MaximumRetryCount": 0
            },
            "AutoRemove": false,
            "VolumeDriver": "",
            "VolumesFrom": null,
            "CapAdd": null,
            "CapDrop": null,
            "CgroupnsMode": "host",
            "Dns": [],
            "DnsOptions": [],
            "DnsSearch": [],
            "ExtraHosts": null,
            "GroupAdd": null,
            "IpcMode": "private",
            "Cgroup": "",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "",
            "Privileged": true,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": [
                "label=disable"
            ],
            "UTSMode": "",
            "UsernsMode": "",
            "ShmSize": 67108864,
            "Runtime": "runc",
            "ConsoleSize": [
                0,
                0
            ],
            "Isolation": "",
            "CpuShares": 0,
            "Memory": 0,
            "NanoCpus": 0,
            "CgroupParent": "",
            "BlkioWeight": 0,
            "BlkioWeightDevice": [],
            "BlkioDeviceReadBps": null,
            "BlkioDeviceWriteBps": null,
            "BlkioDeviceReadIOps": null,
            "BlkioDeviceWriteIOps": null,
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpuRealtimePeriod": 0,
            "CpuRealtimeRuntime": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": [],
            "DeviceCgroupRules": null,
            "DeviceRequests": null,
            "KernelMemory": 0,
            "KernelMemoryTCP": 0,
            "MemoryReservation": 0,
            "MemorySwap": 0,
            "MemorySwappiness": null,
            "OomKillDisable": false,
            "PidsLimit": null,
            "Ulimits": null,
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0,
            "MaskedPaths": null,
            "ReadonlyPaths": null
        },
        "GraphDriver": {
            "Data": {
                "LowerDir": "/var/lib/docker/overlay2/62f8c4605f73d887289d555cea88b527db0d152b61eecfb168c30d133b3ee1ce-init/diff:/var/lib/docker/overlay2/dc8916c53a23f4fa0bf587bacbd7e5f2c736b5d68370c8efeef14d3055dacdfb/diff:/var/lib/docker/overlay2/7936856dbc383e45d690bdd393c5b4450418c7c8caf6a967c1544ef7219b659f/diff:/var/lib/docker/overlay2/ca2e7f12c54ceb4bad9f7557e0b9e7e388fd83d36580efa537552bc7cc39be64/diff:/var/lib/docker/overlay2/8dbca5ef748a292a763de7c5ed54039ed5905919ae0e2e78d9b3e6c7a4893046/diff:/var/lib/docker/overlay2/f8951c934327d7dd3949ef93d762965a1bff4da5402dd42d597270579368ec78/diff:/var/lib/docker/overlay2/a32213854de4e1b441b9e46e9e07aab369f8522fd83db9bee2980139dee0ccd3/diff:/var/lib/docker/overlay2/1fed64185ac4ddc0d2206135adb3810b5daa54c99d7e45b3d0f6efbb28f85c1f/diff:/var/lib/docker/overlay2/2f407dc52bfe46a23be23cac8558fbce8b0bade3b1c9f0b2c6f2059f896c13d8/diff:/var/lib/docker/overlay2/49683a60cb2253c147b73ca96a8940117134fad216a84b140cf6bdf62eea436b/diff:/var/lib/docker/overlay2/3084323e443cbec44a8bcccc7199683876a43fb8af9e16b807dd56c393659deb/diff:/var/lib/docker/overlay2/4af2914e6555443089605c07a421033a2ea4d95b97335a16d1b8d5fd30cc8b41/diff:/var/lib/docker/overlay2/dffe56f40686aad1b52fb42430e5c341a7677dceb02475f2936351c89383e1fe/diff:/var/lib/docker/overlay2/662ca0896b3f17d0562ad993182107db4f6e6834ab6a66e02aebe3dddf7ee39e/diff:/var/lib/docker/overlay2/32bd0022b65989167796f12a68ba5c17bb7c76371ad400d37da956bd5f700d92/diff",
                "MergedDir": "/var/lib/docker/overlay2/62f8c4605f73d887289d555cea88b527db0d152b61eecfb168c30d133b3ee1ce/merged",
                "UpperDir": "/var/lib/docker/overlay2/62f8c4605f73d887289d555cea88b527db0d152b61eecfb168c30d133b3ee1ce/diff",
                "WorkDir": "/var/lib/docker/overlay2/62f8c4605f73d887289d555cea88b527db0d152b61eecfb168c30d133b3ee1ce/work"
            },
            "Name": "overlay2"
        },
        "Mounts": [
            {
                "Type": "bind",
                "Source": "/var/run/docker.sock",
                "Destination": "/run/docker.sock",
                "Mode": "",
                "RW": true,
                "Propagation": "rprivate"
            }
        ],
        "Config": {
            "Hostname": "06dd9825dd38",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "ExposedPorts": {
                "22/tcp": {}
            },
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "DEBIAN_FRONTEND=noninteractive",
                "container=docker"
            ],
            "Cmd": [
                "/lib/systemd/systemd"
            ],
            "Image": "c230e559e99f",
            "Volumes": null,
            "WorkingDir": "",
            "Entrypoint": null,
            "OnBuild": null,
            "Labels": {}
        },
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "9d7ed64c727693eac281f18b2d73e80fb3c173e6fbe290aa4eda89c35ec69a57",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {
                "22/tcp": [
                    {
                        "HostIp": "0.0.0.0",
                        "HostPort": "49246"
                    },
                    {
                        "HostIp": "::",
                        "HostPort": "49246"
                    }
                ]
            },
            "SandboxKey": "/var/run/docker/netns/9d7ed64c7276",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "eed1b6b880fcb434fdea4ef61d4cd98b7453aea5d8e6946d37f62592d4cb3a28",
            "Gateway": "172.17.0.1",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "172.17.0.2",
            "IPPrefixLen": 16,
            "IPv6Gateway": "",
            "MacAddress": "02:42:ac:11:00:02",
            "Networks": {
                "bridge": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": null,
                    "NetworkID": "8dc67544c99f5860dc38bc30e7e0079dbf632e868c313c6ccd4a30c29a0742a2",
                    "EndpointID": "eed1b6b880fcb434fdea4ef61d4cd98b7453aea5d8e6946d37f62592d4cb3a28",
                    "Gateway": "172.17.0.1",
                    "IPAddress": "172.17.0.2",
                    "IPPrefixLen": 16,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "02:42:ac:11:00:02",
                    "DriverOpts": null
                }
            }
        }
    }
]

Expected Result

I would expect to see docker.sock in /var/run when I log into the docker instance.

Actual Result

I do not see docker.sock in /var/run when I log into the docker instance.

NinjaSomething commented 2 years ago

I was able to determine that for some reason it's docker that isn't mounting docker.sock, or anything, into /var/run. The following docker command will create the instance but not map docker.sock as expected. 

docker -H unix:///var/run/docker.sock run -d -p 22 -v /var/run/docker.sock:/var/run/docker.sock --name defaultubuntu1804-jcarlson-popos-puw67kv4 --privileged c230e559e99f /lib/systemd/systemd shell_opts: {}

I'm trying to determine if nothing will mount into /var/run because of how the docker image is built.

NinjaSomething commented 2 years ago

I'm going to close this because I solved the problem. Apparently docker won't let you map anything into /var/run when privileged is true. The following kitchen file worked with docker-in-docker:

# -*- coding: utf-8 -*-
# vim: ft=yaml
---
# For help on this file's format, see https://kitchen.ci/
driver:
  name: docker
  use_sudo: false
  username: root
  volume:
    - /var/run/docker.sock:/run/docker.sock

transport:
  name: docker

platforms:
  - name: ubuntu-18.04
    driver_config:
      run_command: /lib/systemd/systemd

suites:
  - name: default

provisioner:
  name: salt_solo
  salt_install: none
  require_chef: false
  salt_copy_filter:
    - .kitchen
    - .git
  run_salt_call: false

verifier:
  name: shell
  remote_exec: false
  command: python3 -m pytest -s -v test/
  reporter:
    - cli