make design fails: 'permission denied' accessing file inside docker container

[growly]

In my caravel_user_project-derived project I cannot run make user_proj_example or even my own design through make bfg_mux_test. Both fail with a 'permission denied' error accessing a path in the docker container:

$ make bfg_mux_test
make -C openlane bfg_mux_test
make[1]: Entering directory '/home/aryap/src/bfg_mux_test_gf180mcu/openlane'
# bfg_mux_test
mkdir -p ./bfg_mux_test/runs/22_12_04_22_13
rm -rf ./bfg_mux_test/runs/bfg_mux_test
ln -s $(realpath ./bfg_mux_test/runs/22_12_04_22_13) ./bfg_mux_test/runs/bfg_mux_test
docker run -it -v $(realpath /home/aryap/src/bfg_mux_test_gf180mcu/..):$(realpath /home/aryap/src/bfg_mux_test_gf180mcu/..) -v /home/aryap/src/bfg_mux_test_gf180mcu/dependencies/pdks:/home/aryap/src/bfg_mux_test_gf180mcu/dependencies/pdks -v /home/aryap/src/bfg_mux_test_gf180mcu/caravel:/home/aryap/src/bfg_mux_test_gf180mcu/caravel -v /home/aryap/src/bfg_mux_test_gf180mcu/dependencies/openlane_src:/openlane -v /home/aryap/src/bfg_mux_test_gf180mcu/mgmt_core_wrapper:/home/aryap/src/bfg_mux_test_gf180mcu/mgmt_core_wrapper -e PDK_ROOT=/home/aryap/src/bfg_mux_test_gf180mcu/dependencies/pdks -e PDK=gf180mcuC -e MISMATCHES_OK=1 -e CARAVEL_ROOT=/home/aryap/src/bfg_mux_test_gf180mcu/caravel -e OPENLANE_RUN_TAG=22_12_04_22_13 -e MCW_ROOT=/home/aryap/src/bfg_mux_test_gf180mcu/mgmt_core_wrapper -u 1002:1002 \
        efabless/openlane:2022.11.29 sh -c "flow.tcl -design $(realpath ./bfg_mux_test) -save_path $(realpath ..) -save -tag 22_12_04_22_13 -overwrite -ignore_mismatches"
OpenLane 235fa7a4a2872e779588919c58fc4fa32568e075
All rights reserved. (c) 2020-2022 Efabless Corporation and contributors.
Available under the Apache License, version 2.0. See the LICENSE file for more details.

[INFO]: Using configuration in '../home/aryap/src/bfg_mux_test_gf180mcu/openlane/bfg_mux_test/config.tcl'...
[ERROR]: Traceback (most recent call last):
  File "/openlane/scripts/config/tcl.py", line 490, in <module>
    cli()
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/openlane/scripts/config/tcl.py", line 411, in extract_process_info
    with open(output, "w") as f:
PermissionError: [Errno 13] Permission denied: '/home/aryap/src/bfg_mux_test_gf180mcu/openlane/bfg_mux_test/runs/22_12_04_22_13/config_in.tcl'
make[1]: *** [Makefile:73: bfg_mux_test] Error 255
make[1]: Leaving directory '/home/aryap/src/bfg_mux_test_gf180mcu/openlane'
make: *** [Makefile:108: bfg_mux_test] Error 2

I can to run docker successfully without root, e.g.

$ docker run hello-world

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
    (amd64)
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID:
 https://hub.docker.com/

For more examples and ideas, visit:
 https://docs.docker.com/get-started/

I can also run OpenLane through the usual OpenLane flow (i.e. `make test) successfully.

[growly]

Within the container the volumes seem to be mounted as root:

ll /home/aryap/src/bfg_mux_test_gf180mcu/openlane/bfg_mux_test/runs
total 8.0K
drwxr-xr-x 2 root root 4.0K Dec  5 06:30 22_12_04_22_30/
lrwxrwxrwx 1 root root   79 Dec  5 06:30 bfg_mux_test -> /home/aryap/src/bfg_mux_test_gf180mcu/openlane/bfg_mux_test/runs/22_12_04_22_30/

If I invoke docker without the -u $UID:$UID flag I can access this path:

$ touch /home/aryap/src/bfg_mux_test_gf180mcu/openlane/bfg_mux_test/runs/22_12_04_22_30/test
touch: cannot touch ‘/home/aryap/src/bfg_mux_test_gf180mcu/openlane/bfg_mux_test/runs/22_12_04_22_30/test’: Permission denied

vs

$ docker run -it -v $(realpath /home/aryap/src/bfg_mux_test_gf180mcu/..):$(realpath /home/aryap/src/bfg_mux_test_gf180mcu/..) -v /home/aryap/src/bfg_mux_test_gf180mcu/dependencies/pdks:/home/aryap/src/bfg_mux_test_gf180mcu/dependencies/pdks -v /home/aryap/src/bfg_mux_test_gf180mcu/caravel:/home/aryap/src/bfg_mux_test_gf180mcu/caravel -v /home/aryap/src/bfg_mux_test_gf180mcu/dependencies/openlane_src:/openlane -v /home/aryap/src/bfg_mux_test_gf180mcu/mgmt_core_wrapper:/home/aryap/src/bfg_mux_test_gf180mcu/mgmt_core_wrapper -e PDK_ROOT=/home/aryap/src/bfg_mux_test_gf180mcu/dependencies/pdks -e PDK=gf180mcuC -e MISMATCHES_OK=1 -e CARAVEL_ROOT=/home/aryap/src/bfg_mux_test_gf180mcu/caravel -e OPENLANE_RUN_TAG=22_12_04_22_13 -e MCW_ROOT=/home/aryap/src/bfg_mux_test_gf180mcu/mgmt_core_wrapper          efabless/openlane:2022.11.29 bash
OpenLane Container (235fa7a):/openlane$ touch /home/aryap/src/bfg_mux_test_gf180mcu/openlane/bfg_mux_test/runs/22_12_04_22_30/test

[growly]

In the Makefile openlane/Makefile, commenting out the -u flag to docker seems to fix the problem:

$ git diff
diff --git a/openlane/Makefile b/openlane/Makefile
index e1e116f..3d74d15 100644
--- a/openlane/Makefile
+++ b/openlane/Makefile
@@ -53,8 +53,8 @@ docker_startup_mode = $(shell test -t 0 && echo "-it" || echo "--rm" )
 docker_run = \
        docker run $(docker_startup_mode) \
        $(docker_mounts) \
-       $(docker_env) \
-       -u $(shell id -u $(USER)):$(shell id -g $(USER))
+       $(docker_env) #\
+       #-u $(shell id -u $(USER)):$(shell id -g $(USER))

 list:
        @echo $(designs)

[growly]

I found another machine where this works as intended. On that machine, this mount: -v $(realpath /home/aryap/src/bfg_mux_test_gf180mcu/..):$(realpath /home/aryap/src/bfg_mux_test_gf180mcu/..) mounts as the given UID and GID. On my machine where this doesn't work, it mounts as root.

Both are running docker version 20.10.21.

[kareefardi]

what is the output of command id in all permutations (i.e. outside the container, inside the container with and without -u flag)

[growly]

host environment:

uid=1002(aryap) gid=1002(aryap) groups=1002(aryap),24(cdrom),25(floppy),27(sudo),29(audio),30(dip),44(video),46(plugdev),109(netdev),113(bluetooth),118(scanner),1004(docker)

inside container with -u 1002:1002 flag:

uid=1002 gid=1002 groups=1002

without -u flag:

uid=0(root) gid=0(root) groups=0(root)

[kareefardi]

It is interesting that with -u id output non root uid and gid while the volumes are mounted as root. Is this behavior consistent over different run directories?. Perhaps for some reason a run directory owned by root was created (for example due to runnig as a sudo make). Can you tree cleaning the runs directory of bfg_mux_test. Then try again while keeping -u and see if the same error persist.

[growly]

I haven't touched sudo, I used the same steps to build caravel_user_project on this machine as on my other machine (namely make setup).

The mounted directory has my user's permissions (not root) in the host environment, so it's strange that it gets mounted as root. Other volumes are also consistently mounted as root on the problematic machine. I actually think this is a problem with docker or how it's configured, since I've seen this issue reported a bunch now without a clear cause.

Perhaps the only thing to do here is to make a note of this problem so that others can figure it out faster. Or add something like a permission test to the build scripts to determine if the -u flag is necessary.

[josuah]

A similar issue was encountered elsewhere on Slack, for which the discussion is available here, and features various debugging steps, and screenshots/copy-paste of logs with every debugging steps: https://web.open-source-silicon.dev/t/12321953/hello-i-was-trying-to-use-caravel-template-to-start-using-op#823bdd1f-99a2-442f-ade1-268faaf739df

The proposed workaround did work, although not a long-term solution maybe... https://github.com/efabless/caravel_user_project/issues/221#issuecomment-1336838988

Thanks for submitting that bug, it helped while searching similar issues.