replicate / cog

Containers for machine learning
https://cog.run
Apache License 2.0
8.08k stars 562 forks source link

docker: Error response from daemon: invalid mount config for type "bind": stat /my/current/folder: permission denied. #261

Closed annahung31 closed 3 years ago

annahung31 commented 3 years ago

Hi, Thanks for the cool work!

I'm following the tutorial here and working on Ubuntu as a non-root user. But when I run cog run python, I encountered an error:

Running 'python' in Docker with the current directory mounted as a volume...
docker: Error response from daemon: invalid mount config for type "bind": stat  /my/current/folder: permission denied.

It seems related to the mount setting of docker, but I'm not familiar with the docker setting. How can I deal with the error? Where can I find the Dockerfile that cog created and modify the setting? Thanks!


The full building log:

Building Docker image from environment in cog.yaml...
[+] Building 3.8s (12/12) FINISHED                                                                                                                                                                                             
 => [internal] load build definition from Dockerfile                                                                                                                                                                      0.0s
 => => transferring dockerfile: 489B                                                                                                                                                                                      0.0s
 => [internal] load .dockerignore                                                                                                                                                                                         0.0s
 => => transferring context: 2B                                                                                                                                                                                           0.0s
 => resolve image config for docker.io/docker/dockerfile:1.2                                                                                                                                                              1.8s
 => CACHED docker-image://docker.io/docker/dockerfile:1.2@sha256:e2a8561e419ab1ba6b2fe6cbdf49fd92b95912df1cf7d313c3e2230a333fdbcc                                                                                         0.0s
 => [internal] load metadata for docker.io/library/python:3.8                                                                                                                                                             1.3s
 => [internal] load build context                                                                                                                                                                                         0.0s
 => => transferring context: 13.61kB                                                                                                                                                                                      0.0s
 => [stage-0 1/4] FROM docker.io/library/python:3.8@sha256:c2842aababbe377f9c76f172d9eb39487e23f306b2f29f020f3f6654cb0876e9                                                                                               0.0s
 => CACHED [stage-0 2/4] COPY .cog/tmp/cog-0.0.1.dev-py3-none-any.whl /tmp/cog-0.0.1.dev-py3-none-any.whl                                                                                                                 0.0s
 => CACHED [stage-0 3/4] RUN --mount=type=cache,target=/root/.cache/pip pip install /tmp/cog-0.0.1.dev-py3-none-any.whl                                                                                                   0.0s
 => CACHED [stage-0 4/4] WORKDIR /src                                                                                                                                                                                     0.0s
 => exporting to image                                                                                                                                                                                                    0.0s
 => => exporting layers                                                                                                                                                                                                   0.0s
 => => writing image sha256:ea7f49cf567ab25ac6b8d6c8a6abb3d6bb6c1738b4a2c08ff19dd75536e42811                                                                                                                              0.0s
 => => naming to docker.io/library/cog-cogtest-base                                                                                                                                                                       0.0s
 => exporting cache                                                                                                                                                                                                       0.0s
 => => preparing build cache for export                                                                                                                                                                                   0.0s

Running 'python' in Docker with the current directory mounted as a volume...
docker: Error response from daemon: invalid mount config for type "bind": stat /home/annahung/189/my_project/playground/cog_test: permission denied.
See 'docker run --help'.
ⅹ exit status 126
andreasjansson commented 3 years ago

Hi @annahung31!

That's a weird error, I haven't seen that before. I'm guessing the user that the Docker daemon runs under doesn't have permission to read from your cog_test/ directory.

You could try to chmod -R 777 cog_test from the playground/ directory. But I'm not sure that will work.

Is this a cloud instance or a university machine in some sort of managed cluster?

annahung31 commented 3 years ago

Thanks for the reply! unfortunately changing the permission doesn't solve the error.

It's a machine in the cluster managed by the lab I belong to. We(me and my colleagues) maintain the machine by ourselves, but only one system administrator for each machine, so most of us are non-root users.

Just a note, I also tried on adding myself into the docker group like the Answer of this page, but it still doesn't work.

andreasjansson commented 3 years ago

Fixed! 🥳

The issue was that the folder was part of an sshfs mount. user_allow_other was disabled which meant that Docker didn't have permission to access it.

Solution: https://serverfault.com/a/947184

We should try to catch errors like this and point to solutions.

annahung31 commented 3 years ago

Thanks Andreas!

zeke commented 2 years ago

@allcontributors please add @annahung31 for bugs

allcontributors[bot] commented 2 years ago

@zeke

I've put up a pull request to add @annahung31! :tada:

cloneofsimo commented 1 year ago

Sharing my experience here : I just had similar issue troubleshooting with weight download with cog run for https://github.com/replicate/cog-stable-diffusion.

 => => exporting layers                                                                                                                                                                                                                                                        0.0s
 => => writing image sha256:4cb11762fd809c053c4c72975c48a7f0df2f032f58d9cca9566176ff3a304d4c                                                                                                                                                                                   0.0s
 => => naming to docker.io/library/cog-lorarep-base                                                                                                                                                                                                                            0.0s
 => exporting cache                                                                                                                                                                                                                                                            0.0s
 => => preparing build cache for export                                                                                                                                                                                                                                        0.0s

Running 'script/download-weights' in Docker with the current directory mounted as a volume...
[FATAL tini (7)] exec script/download-weights failed: Permission denied
ⅹ exit status 126

Solution was to give 777 permission to all the scripts on my projects so docker daemons can run them, just like @andreasjansson suggested above.