Volume error on a clean launch with the `docker-compose_dev-sif.erda.dk_full.yml on Rocky 9,3 due to `ro` flag on the `vgrid_files_readonly` bind mount

[rasmunk]

When trying to create the migrid container via the docker-compose_dev-sif.erda.dk_full.yml configuration, the docker runtime seems to not like that the readonly flag bind,ro is set on the vgrid_files_readonly bind volume. https://github.com/ucphhpc/docker-migrid-ucphhpc/blob/85e882ca2351cf473c56914cad98257845d1936d/docker-compose_dev-sif.erda.dk_full.yml#L390

System details: OS: Rocky 9,3 Docker version:

Client: Docker Engine - Community
 Version:           24.0.7
 API version:       1.43
 Go version:        go1.20.10
 Git commit:        afdd53b
 Built:             Thu Oct 26 09:09:13 2023
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          24.0.7
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.10
  Git commit:       311b9ff
  Built:            Thu Oct 26 09:07:45 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.26
  GitCommit:        3dd1e886e55dd695541fdcd67420c2888645a495
 runc:
  Version:          1.1.10
  GitCommit:        v1.1.10-0-g18a0cb0
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Upon launching, docker throws the following error:

Error response from daemon: failed to mount local volume: mount ./storage/vgrid_files_readonly:/var/lib/docker/volumes/docker-migrid_vgrid_files_readonly/_data, flags: 0x1001: no such file or directory

However, you can get it to work, if launch the docker-migrid stack without the ro flag on the vgrid_files_readonly volume and the subsequently readd it after the stack has been taken down again.

Steps to mitigate:

1. remove the ro flag on vgrid_files_readonly
2. launch the stack via make up
3. stop the stack via make down
4. readd the ro flag on the vgrid_files_readonly
5. relaunch the stack with make up

So far I have only tested it on Rocky 9,3. Therefore I can't say if this error also occurs on other distributions or versions.

[jonasbardino]

Sounds like an issue with the specific dev-sif storage/ setup to me. Probably something @Martin-Rehr has more insight into.

[Martin-Rehr]

The problem is RO binds to 'state' because of the 'chown -R $USER:$USER ${MIG_ROOT}/state' command in docker-entry.sh

My suggestion is that we move the 'chown -R $USER:$USER ${MIG_ROOT}/state' out of docker-entry.sh and into the build environment.

[rasmunk]

Would it be possible to have chown in both places? I'm just contemplating a scenario where you fix some issue inside ${MIG_ROOT}/state files on the host with a user that has a different UID, and then subsequently relaunch the docker-migrid stack, which would then have permission issues inside ${MIG_ROOT}/state

[jonasbardino]

I have a strong suspicion that the chown was in fact moved to docker-entry.sh because it didn't work correctly to do during build. Probably hinges on the volume vs plain dir discussion.

[jonasbardino]

Hmmm, maybe I'm mixing it up with other adjustments going into docker-entry for that reason. I can trace the introduction of that chown -R all the way back to 8b414873057cb5b48f053024e42ef45cd06913f4 several years ago.

[Martin-Rehr]

This fixes the issue: https://github.com/ucphhpc/docker-migrid-ucphhpc/commit/3180d514a4299f1e12d37889568a917635400904 https://github.com/ucphhpc/docker-migrid-ucphhpc/commit/3180d514a4299f1e12d37889568a917635400904

On 23 Jan 2024, at 09.20, Jonas Bardino @.***> wrote:

Hmmm, maybe I'm mixing it up with other adjustments going into docker-entry for that reason. I can trace the introduction of that chown -R all the way back to rev 8b414873057cb5b48f053024e42ef45cd06913f4 several years ago.

— Reply to this email directly, view it on GitHub https://github.com/ucphhpc/docker-migrid-ucphhpc/issues/1#issuecomment-1905524791, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACVL5PKM55TXGZDV6O4ISRDYP5XDLAVCNFSM6AAAAABBQHPNZ6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMBVGUZDINZZGE. You are receiving this because you were mentioned.