Starting docker plugin with sysbox-runc as default runtime causes permission denied error

Expected result The plugin starts with no problems

Actual behaviour Docker fails to start the plugin with the following error message: Error response from daemon: OCI runtime create failed: container_linux.go:393: starting container process caused: process_linux.go:607: container init caused: standard_init_linux.go:82: validating cwd caused: rootfs_linux.go:1139: getting cwd caused: stat .: permission denied: unknown

Is it sysbox-runcs fault? I think so. I removed "default-runtime": "sysbox-runc" from /etc/docker/daemon.json and the plugin starts just fine. Adding it again will cause the plugin to not start again.

Steps to reproduce the issue

Configure sysbox-runc as the default docker runtime
Try to install and enable the latest version of my docker plugin -> will fail
Remove sysbox-runc as the default docker runtime
Try to enable my docker plugin again -> will work (note: It won't stay started but thats because the plugin is improperly configured, not because docker can't even start the container)

Software version OS: Ubuntu 20.04.3 LTS docker: 20.10.8 sysbox-runc: 0.4.0, commit: 9e55c35e249f753c7d31e987c21d4ca4a2ddacfb, built at: Tue Jul 13 19:11:44 UTC 2021

Docker plugin config.json

{
  "description": "DockerMC logging plugin",
  "documentation": "-",
  "entrypoint": ["java", "-jar", "/plugin/DMCDockerPlugin.jar"],
  "interface": {
    "socket": "dmcdriver.sock",
    "types": ["docker.logdriver/1.0", "docker.volumedriver/2.0"]
  },
  "env": [
    {
      "name": "LOG_LEVEL",
      "description": "Set log level to output for plugin logs",
      "value": "INFO",
      "settable": ["value"]
    },
    {
      "name": "DB_HOST",
      "description": "The host where the main Postgres DB is found. IP or better a hostname.",
      "value": "",
      "settable": ["value"]
    },
    {
      "name": "DB_PASSWORD",
      "description": "Password for the main Postgres DB",
      "value": "",
      "settable": ["value"]
    }
  ],
  "network": {
    "type": ""
  },
  "mounts": [
    {
      "destination": "/var/log/dmcdriver",
      "options": [
        "rbind"
      ],
      "name": "logfolder",
      "source": "/var/log",
      "settable": [
        "source"
      ],
      "type": "bind"
    }
  ]
}

Hi @derteufelqwe,

Thanks for trying Sysbox, and sorry to hear you hit this issue.

I am not too familiar with Docker plugins, so please bear with me as I learn more about it.

The error you reported is one that rarely (if ever) occurs:

validating cwd caused: rootfs_linux.go:1139: getting cwd caused: stat .: permission denied: unknown

It basically means that as the container is being setup, the container's init process is getting EPERM when doing a stat() of its current working directory. That current working directory should be the container's root filesystem jail, so it's very strange that it would hit EPERM there.

A few questions to proceed:

1) Without the Docker plugin config, are you able to use Docker+ Sysbox to launch containers?

2) Is your Ubuntu host using the shiftfs kernel module (i.e., modinfo shiftfs)?

3) Have you tried the Docker plugin by configuring Docker in userns-remap mode with the regular runc (or with sysbox-runc)?

And finally, what would be the easiest way for me to repro this on my side?

Thanks!

Hi @ctalledo,

to answer your questions:

Yes. I used docker + sysbox for months (for docker in docker) without noticing any problems. It just so happened that I wanted to start my docker plugin and noticed that error.

I think so. The output of modinfo shiftfs

filename:       /lib/modules/5.11.0-34-generic/kernel/fs/shiftfs.ko
license:        GPL v2
description:    id shifting filesystem
author:         Christian Brauner <christian.brauner@ubuntu.com>
author:         Seth Forshee <seth.forshee@canonical.com>
author:         James Bottomley
alias:          fs-shiftfs
srcversion:     D5BD8DD9C47DB27DD8E263A
depends:
retpoline:      Y
intree:         Y
name:           shiftfs
vermagic:       5.11.0-34-generic SMP mod_unload modversions
sig_id:         PKCS#7
signer:         Build time autogenerated kernel key
sig_key:        3F:78:34:4A:10:B7:65:C2:C5:FF:66:4E:89:21:35:FA:74:C9:00:B5
sig_hashalgo:   sha512

I have just now tested docker with userns-remap mode using my default user and it doesn't change anything. I can start docker containers regardless of the runtime, but can't start my docker plugin using sysbox-runc runtime. I have added the following line to my daemon.json: "userns-remap": "<my_username>"

How to reproduce the issue This should be straight forward

Download and install the latest release of sysbox-runc
Install my docker plugin with docker plugin install --grant-all-permissions derteufelqwe/dockermc-drivers:latest. This should return a wanted error like Error response from daemon: dial unix /run/docker/plugins/<hash>/dmcdriver.sock: connect: no such file or directory. This indicates that starting the plugin worked. The will crash tho because it's improperly configured.

Modify your /etc/docker/daemon.json file to use sysbox-runc as default runtime like this

...
"runtimes": {
"sysbox-runc": {
    "path": "/usr/bin/sysbox-runc"
}
},
"default-runtime": "sysbox-runc",
...

Restart the docker daemon
Try to enable the plugin with docker plugin enable derteufelqwe/dockermc-drivers:latest
This will fail with the mentioned error

Quick note: If you use sysbox-runc to start a docker-in-docker container installing the plugin inside the container works just fine.

Thanks @derteufelqwe, that's very helpful; I will try this over the weekend.

Quick note: If you use sysbox-runc to start a docker-in-docker container installing the plugin inside the container works just fine.

That's a good data point, thanks.

Hi @derteufelqwe,

I tried reproducing but don't see the validating cwd caused: rootfs_linux.go:1139: getting cwd caused: stat .: permission denied error you reported.

Here is what I did:

1) Configured Sysbox as the default runtime for Docker:

# cat /etc/docker/daemon.json                                                                                                                                                                                                                                                               
{                                                                                                                                                                                                                                                                                                                             
    "runtimes": {                                                                                                                                                                                                                                                                                                             
        "sysbox-runc": {                                                                                                                                                                                                                                                                                                      
            "path": "/usr/bin/sysbox-runc"                                                                                                                                                                                                                                                                                    
        }                                                                                                                                                                                                                                                                                                                     
    },                                                                                                                                                                                                                                                                                                                        
    "default-runtime": "sysbox-runc",                                                                                                                                                                                                                                                                                         
}

2) Installed the Docker plugin:

# docker plugin install --grant-all-permissions derteufelqwe/dockermc-drivers:latest                                                                                                                                                                                                        
latest: Pulling from derteufelqwe/dockermc-drivers                                                                                                                                                                                                                                                                            
Digest: sha256:fd9b02dbec18cda621c2c4a47e0d0787aa82db5ff4e77d0a8af1f14ade1d924f                                                                                                                                                                                                                                               
51ac8f922300: Complete                                                                                                                                                                                                                                                                                                        
Error response from daemon: dial unix /run/docker/plugins/ff9383248f954be347d6a46db6303955eaa05ec85cc6248a110e5065d448653e/dmcdriver.sock: connect: no such file or directory        

# docker plugin enable derteufelqwe/dockermc-drivers:latest                                                                                                                                                                                                                                 
Error response from daemon: dial unix /run/docker/plugins/ff9383248f954be347d6a46db6303955eaa05ec85cc6248a110e5065d448653e/dmcdriver.sock: connect: no such file or directory

At this point, I don't see any indication of the error you reported.

I can also start a Sysbox container (successfully):

# docker run --runtime=sysbox-runc -ti --rm alpine
/ #

Am I missing something in theses steps?

Hello @ctalledo,

Thanks for your quick responses. I have tried to reproduce the issue on a second ubuntu VM with no success. I have also tried to fix the issue on my main VM with kernel and software upgrades etc. Still no success. I have also tried to break the installation on my second VM but that didn't work either. It looks like this is an extreme edge case and because I have no clue how to reproduce the issue, I will just create a new VM and use that from now on.

Thanks for you help.

Thanks @derteufelqwe; If you hit it again don't hesitate to re-open the issue, happy to help you resolve it.

And thanks for giving Sysbox a shot, hope you are finding it useful!

nestybox / sysbox

Starting docker plugin with sysbox-runc as default runtime causes permission denied error #400