mount state directory to keep cache when building inside containers

[msvechla]

Hi,

Thanks for developing this awesome tool!

I am trying to run builds inside a container based CI pipeline. To speed up the build process, keeping the cache folder across builds would be nice.

I tried mounting a directory from the host to persist the cache, however the build then fails bind mounting my state volume to /tmp/buildkit-mount525951409

docker run -it --privileged --workdir /home/user/src -v $(pwd):/home/user/src:ro -v /tmp/img:/tmp/state  r.j3ss.co/img build -s /tmp/state -t test .
Building docker.io/library/test:latest
Setting up the rootfs... this may take a bit.
[+] Building 0.0s (2/2) FINISHED
 => [internal] load .dockerignore                                                                                                           0.0s
 => => transferring context: 2B                                                                                                             0.0s
 => [internal] load Dockerfile                                                                                                              0.0s
 => => transferring dockerfile: 444B                                                                                                        0.0s
failed to solve: failed to read dockerfile: failed to mount /tmp/buildkit-mount525951409: [{Type:bind Source:/tmp/state/runc/native/snapshots/snapshots/2 Options:[rbind ro]}]: operation not permitted

Do you have any hints on how I can accomplish mounting the cache from the host, when building inside a container?

Thanks for your help!

[jessfraz]

make the directory not in /tmp :)

On Wed, Sep 26, 2018 at 4:09 PM msvechla notifications@github.com wrote:

Hi,

Thanks for developing this awesome tool!

I am trying to run builds inside a container based CI pipeline. To speed up the build process, keeping the cache folder across builds would be nice.

I tried mounting a directory from the host to persist the cache, however the build then fails bind mounting my state volume to /tmp/buildkit-mount525951409

docker run -it --privileged --workdir /home/user/src -v $(pwd):/home/user/src:ro -v /tmp/img:/tmp/state r.j3ss.co/img build -s /tmp/state -t test . Building docker.io/library/test:latest Setting up the rootfs... this may take a bit. [+] Building 0.0s (2/2) FINISHED => [internal] load .dockerignore 0.0s => => transferring context: 2B 0.0s => [internal] load Dockerfile 0.0s => => transferring dockerfile: 444B 0.0s failed to solve: failed to read dockerfile: failed to mount /tmp/buildkit-mount525951409: [{Type:bind Source:/tmp/state/runc/native/snapshots/snapshots/2 Options:[rbind ro]}]: operation not permitted

Do you have any hints on how I can accomplish mounting the cache from the host, when building inside a container?

Thanks for your help!

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/genuinetools/img/issues/161, or mute the thread https://github.com/notifications/unsubscribe-auth/ABYNbGU1FgccFiuPK78w8ox7TQfCSMmKks5ufAkXgaJpZM4W7pI4 .

--

Jessie Frazelle 4096R / D4C4 DD60 0D66 F65A 8EFC 511E 18F3 685C 0022 BFF3 pgp.mit.edu http://pgp.mit.edu/pks/lookup?op=get&search=0x18F3685C0022BFF3

[msvechla]

I also tried it with different directories like ${PWD}/state and mounting it to different directories inside the container (e.g. /state or /home/user/state. However I had still no luck. I am probably missing something obvious here...

docker run -it --privileged --workdir /home/user/src -v $(pwd):/home/user/src:ro -v ${PWD}/state:/state  r.j3ss.co/img build -s /state -t test .
Building docker.io/library/test:latest
Setting up the rootfs... this may take a bit.
[+] Building 0.0s (2/2) FINISHED
 => [internal] load .dockerignore                                                                                                           0.0s
 => => transferring context: 2B                                                                                                             0.0s
 => [internal] load Dockerfile                                                                                                              0.0s
 => => transferring dockerfile: 444B                                                                                                        0.0s
failed to solve: failed to read dockerfile: failed to mount /tmp/buildkit-mount538742665: [{Type:bind Source:/state/runc/native/snapshots/snapshots/2 Options:[rbind ro]}]: operation not permitted

[jessfraz]

oh its because that image runs as a different user who does not have access to the dir, i should either at an entrypoint to chown the dir before running or you can runas root in the container On Wed, Sep 26, 2018 at 4:27 PM msvechla notifications@github.com wrote:

I also tried it with different directories like ${PWD}/state and mounting it to different directories inside the container (e.g. /state or /home/user/state. However I had still no luck. I am probably missing something obvious here...

docker run -it --privileged --workdir /home/user/src -v $(pwd):/home/user/src:ro -v ${PWD}/state:/state r.j3ss.co/img build -s /state -t test . Building docker.io/library/test:latest Setting up the rootfs... this may take a bit. [+] Building 0.0s (2/2) FINISHED => [internal] load .dockerignore 0.0s => => transferring context: 2B 0.0s => [internal] load Dockerfile 0.0s => => transferring dockerfile: 444B 0.0s failed to solve: failed to read dockerfile: failed to mount /tmp/buildkit-mount538742665: [{Type:bind Source:/state/runc/native/snapshots/snapshots/2 Options:[rbind ro]}]: operation not permitted

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

--

Jessie Frazelle 4096R / D4C4 DD60 0D66 F65A 8EFC 511E 18F3 685C 0022 BFF3 pgp.mit.edu

[msvechla]

Thats also what I thought, however I already gave ownership to uid 1000 on the host, which maps correctly inside the container:

host:

 ls -l / |grep state
drwxrwxrwx   2 1000  wheel    64 Sep 27 11:18 state

inside container:

docker run -it --privileged --entrypoint sh -v /state:/state  r.j3ss.co/img
/ $ ls -l / |grep state
drwxrwxrwx    2 user     user            64 Sep 27 09:18 state

build:

docker run -it --privileged --workdir /home/user/src -v $(pwd):/home/user/src:ro -v /state:/state  r.j3ss.co/img build -s /state -t test .
Building docker.io/library/test:latest
Setting up the rootfs... this may take a bit.
[+] Building 0.1s (2/2) FINISHED
 => [internal] load .dockerignore                                                                                                                           0.0s
 => => transferring context: 2B                                                                                                                             0.0s
 => [internal] load Dockerfile                                                                                                                              0.0s
 => => transferring dockerfile: 444B                                                                                                                        0.0s
failed to solve: failed to read dockerfile: failed to mount /tmp/buildkit-mount047803625: [{Type:bind Source:/state/runc/native/snapshots/snapshots/2 Options:[rbind ro]}]: operation not permitted 

[jessfraz]

Hmm I tested locally with an entry point to chown and that worked for me

On Thu, Sep 27, 2018 at 02:23 msvechla notifications@github.com wrote:

Thats also what I thought, however I already gave ownership to uid 1000 on the host, which maps correctly inside the container:

host:

ls -l / |grep state drwxrwxrwx 2 1000 wheel 64 Sep 27 11:18 state

inside container:

docker run -it --privileged --entrypoint sh -v /state:/state r.j3ss.co/img / $ ls -l / |grep state drwxrwxrwx 2 user user 64 Sep 27 09:18 state

build:

docker run -it --privileged --workdir /home/user/src -v $(pwd):/home/user/src:ro -v /state:/state r.j3ss.co/img build -s /state -t test . Building docker.io/library/test:latest Setting up the rootfs... this may take a bit. [+] Building 0.1s (2/2) FINISHED => [internal] load .dockerignore 0.0s => => transferring context: 2B 0.0s => [internal] load Dockerfile 0.0s => => transferring dockerfile: 444B 0.0s failed to solve: failed to read dockerfile: failed to mount /tmp/buildkit-mount047803625: [{Type:bind Source:/state/runc/native/snapshots/snapshots/2 Options:[rbind ro]}]: operation not permitted

— You are receiving this because you commented.

Reply to this email directly, view it on GitHub https://github.com/genuinetools/img/issues/161#issuecomment-425021897, or mute the thread https://github.com/notifications/unsubscribe-auth/ABYNbNy0O2667y1WviW8oyKhODpvYGx3ks5ufJkEgaJpZM4W7pI4 .

--

Jessie Frazelle 4096R / D4C4 DD60 0D66 F65A 8EFC 511E 18F3 685C 0022 BFF3 pgp.mit.edu http://pgp.mit.edu/pks/lookup?op=get&search=0x18F3685C0022BFF3

[msvechla]

Finally got another chance to debug this today. Looks like its an issue with the operating system. My tests were done on mac osx 10.13.6 with docker 18.06.1-ce, build e68fc7a and I could not get it to work there.

However on our linux based cloud environment everything worked as expected. Will close this for now. Thanks for your help!