moby / moby

The Moby Project - a collaborative project for the container ecosystem to assemble container-based systems
https://mobyproject.org/
Apache License 2.0
68.73k stars 18.67k forks source link

Docker --device with different in-container path fails when userns-remap is enabled #41379

Open hawicz opened 4 years ago

hawicz commented 4 years ago

Mapping a device into a docker container with --device fails with a "no such file or directory" error.

Steps to reproduce the issue:

  1. Edit /etc/docker/daemon.json, set "userns-remap": "default"
  2. Restart docker (systemctl restart docker)
  3. Start any container with --device but use a different name for the in-container device as compared to the host device, e.g.: docker run -it --rm --device /dev/null:/dev/null2 alpine:latest /bin/sh

Describe the results you received: If failed with:

docker: Error response from daemon: OCI runtime create failed: container_linux.go:349: starting container process caused "process_linux.go:449: container init caused \"rootfs_linux.go:70: creating device nodes caused \\\"no such file or directory\\\"\"": unknown.

Describe the results you expected: When run without userns-remap enabled, the container starts normally, and the device appears at the desired path.

Additional information you deem important (e.g. issue happens only occasionally):

Using a device path that already gets mapped into the container does not product an error, but instead silently fails to map the expected device. e.g. (note the different device numbers):

docker run -it --rm --device /dev/zero:/dev/urandom alpine:latest /bin/ls -l /dev/zero /dev/urandom
crw-rw-rw-    1 nobody   nobody      1,   9 Aug 21 01:51 /dev/urandom
crw-rw-rw-    1 nobody   nobody      1,   5 Aug 21 01:51 /dev/zero

Output of docker version:

Client: Docker Engine - Community Version: 19.03.12 API version: 1.40 Go version: go1.13.10 Git commit: 48a66213fe Built: Mon Jun 22 15:45:50 2020 OS/Arch: linux/amd64 Experimental: false ... containerd: Version: 1.2.13 ... runc: Version: 1.0.0-rc10 ... docker-init: Version: 0.18.0 ...

Output of docker info:

... Operating System: Debian GNU/Linux 10 (buster) ...

Additional environment details (AWS, VirtualBox, physical, etc.):

thaJeztah commented 4 years ago

So the error comes from runc; https://github.com/opencontainers/runc/blob/master/libcontainer/rootfs_linux.go#L70-L73

    if setupDev {
        if err := createDevices(config); err != nil {
            return newSystemErrorWithCause(err, "creating device nodes")
        }

Which calls https://github.com/opencontainers/runc/blob/master/libcontainer/rootfs_linux.go#L589-L609

// Create the device nodes in the container.
func createDevices(config *configs.Config) error {
    useBindMount := system.RunningInUserNS() || config.Namespaces.Contains(configs.NEWUSER)
    oldMask := unix.Umask(0000)
    for _, node := range config.Devices {

        // The /dev/ptmx device is setup by setupPtmx()
        if utils.CleanPath(node.Path) == "/dev/ptmx" {
            continue
        }

        // containers running in a user namespace are not allowed to mknod
        // devices so we can just bind mount it from the host.
        if err := createDeviceNode(config.Rootfs, node, useBindMount); err != nil {
            unix.Umask(oldMask)
            return err
        }
    }
    unix.Umask(oldMask)
    return nil
}

Which (if user-ns is enabled) runs https://github.com/opencontainers/runc/blob/master/libcontainer/rootfs_linux.go#L632-L634

    dest := filepath.Join(rootfs, node.Path)
    if err := os.MkdirAll(filepath.Dir(dest), 0755); err != nil {
        return err
    }
    if bind {
        return bindMountDeviceNode(dest, node)
    }
func bindMountDeviceNode(dest string, node *configs.Device) error {
    f, err := os.Create(dest)
    if err != nil && !os.IsExist(err) {
        return err
    }
    if f != nil {
        f.Close()
    }
    return unix.Mount(node.Path, dest, "bind", unix.MS_BIND, "")
}
thaJeztah commented 4 years ago

/cc @kolyshkin @AkihiroSuda

t0rrant commented 3 years ago

I get the same error but when using bind mounts with docker version 19.03.14:

$ docker run --name=92a6443d-e173-4abf-8773-4b5bb31da272 --net=host --user=1000 --cap-add=SYS_ADMIN --cap-add=SYS_RESOURCE  -v=/opt/xxx/yyy/7c94aabc85d4894c:/zzz -v=/opt/xxx/yyy/zzz/www/file.sock:/.file.sock image:tag /bin/bash -c ls .
docker: Error response from daemon: OCI runtime create failed: container_linux.go:345: starting container process caused
"process_linux.go:430: container init caused \"rootfs_linux.go:58: mounting \\\"/opt/xxx/yyy/7c94aabc85d4894c\\\" to rootfs
\\\"/var/lib/docker/overlay2/<hash>/merged\\\" at \\\"/work_dir\\\" caused \\\"stat /opt/xxx/yyy/7c94aabc85d4894c: no such file or
directory\\\"\"": unknown.

Also found this issue from the kubernetes repo https://github.com/kubernetes/kubernetes/issues/96240

jfcabral commented 2 years ago

Hi, I too seem to be having this issue while setting a device with a different in-container path.

The error I get while starting the container is:

Error response from daemon: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: rootfs_linux.go:88: creating device nodes caused: no such file or directory: unknown

A sample docker-compose file is:

version: '3.8'
services:
  zigbee2mqtt:
    container_name: zigbee2mqtt
    image: koenkk/zigbee2mqtt
    restart: unless-stopped
    volumes:
      - ./data:/app/data
      - /run/udev:/run/udev:ro
    ports:
      # Frontend port
      - 8080:8080
    devices:
      # Make sure this matched your adapter location
      - /dev/ttyUSB0:/dev/ttyUSB1

If I change it from:

/dev/ttyUSB0:/dev/ttyUSB1 to /dev/ttyUSB0 it starts properly.

The most annoying thing is that the error message wasn't very clear, hence me losing a lot of time until I managed to discover this issue 😅

For reference, my specs are:

theobarkerh commented 1 month ago

Same issue when attempting to map other special character devices, i.e. /dev/gpiochip\<n> to anything other than the host's path fails:

$ gpioinfo
gpioinfo: error accessing GPIO chips: No such device

A long listing of the /dev shows gpiochip\<n-1>, which is the compose file mapped value of the host's gpiochip\<n>. None of the gpiod utilities will work with the mapped device.

Docker version 24.0.2, build 0823df7daa docker-compose version 1.29.2