Cache permissions

[pwaller]

Hi! I'm excited about the new builder frontend. I have been dreaming of this sort of functionality for years now! Great to see it happening, looks like there are some very neat features in here :tada:.

My Dockerfile looks something like this:

# syntax=docker/dockerfile:experimental
USER nobody:nogroup

COPY go.mod go.sum .
RUN --mount=type=cache,target=/home/nobody/.cache/go-build \
  go mod download

But it fails with permission denied:

#17 0.658 go: disabling cache (/home/nobody/.cache/go-build) due to initialization failure: mkdir /home/nobody/.cache/go-build/00: permission denied
#17 0.658 go: cannot use modules with build cache disabled

Is there a way to specify the owner of the cache? One of my aims is to avoid using root as early as I can during the build, and to do the build as nobody. So I use root to install a few distro packages, then drop permissions to build the go code.

I wasn't sure whether this is the right place to file this issue. Please let me know if I should have filed it in the moby repository.

[pwaller]

I tried to work around it with something like this - in other words, making the directories exist in the cache and giving them correct permissions before dropping:

# syntax=docker/dockerfile:experimental

USER root

RUN --mount=type=cache,id=go,target=/home/nobody/.cache/ \
    --mount=type=cache,id=go,target=/home/nobody/go/ \
    mkdir -p /home/nobody/.cache/go-build \
              /home/nobody/go/pkg \
 && chown nobody:nogroup \
      /home/nobody/.cache/go-build \
      /home/nobody/go/pkg

USER nobody:nogroup

COPY --chown=nobody go.mod go.sum .
RUN --mount=type=cache,id=go,target=/home/nobody/.cache/ \
    --mount=type=cache,id=go,target=/home/nobody/go/ \
    go mod download

... However, this doesn't work. It works the first time, but then if I do docker builder prune, the cache directories goes away, but the pre-nobody stage is 'cached', so when go mod download is run, the cache directories which are meant to have the appropriate owner don't exist anymore.

[tonistiigi]

The workaround that should work should be something like:

FROM _ AS cachebaseforgo
RUN mkdir -p go-build && chmod 0777 go-build

FROM 

RUN --mount=type=cahce,target=/home/nobody/.cache/,from=cachebaseforgo

You can also use source= for a specific submount of the cache.

For this specific case, I would argue though that there isn't much reason to change user before that command. The main reason to avoid root is to avoid accidental accesses to specific files with security properties in host, but with build any mounts from host are disabled anyway. root user does not have a meaning for process execution permissions in here. For the file access, a better solution is also to map the user with user namespace.

@AkihiroSuda Seems that only secrets and ssh support overriding the permissions in LLB. Maybe we should allow it for cache type as well as the workaround isn't that simple. Although it would be conflicting with from then.

[pwaller]

Ah, thanks for the hints about source= and from=. Those don't appear to be documented on the cache type, by the way: https://github.com/moby/buildkit/blob/594f95bc1d15c3218bdbf4635993b4ecda3e3cc9/frontend/dockerfile/docs/experimental.md#run---mounttypecache

They are mentioned on the bind type but I didn't automatically infer that they would therefore work elsewhere.

[tonistiigi]

added more docs in #728

[pwaller]

@tonistiigi minor issues with this:

FROM _ AS cachebaseforgo
RUN mkdir -p go-build && chmod 0777 go-build

1. I get rpc error: code = Unknown desc = failed to create LLB definition: failed to parse stage name "_": invalid reference format

2. If I use scratch, there is no shell to run mkdir and chown with.

3. If I use alpine or some other base, the resulting cache has lots of other stuff in it.

[tonistiigi]

The _ was just an example to be replaced with any image. If you use alpine you can use src to only use subdir as a cache mount or you can make an extra stage from scratch and copy the subdir to it.

I'm open to suggestions if there are ideas for modifications that make this flow simpler.

[arncore]

For this specific case, I would argue though that there isn't much reason to change user before that command. The main reason to avoid root is to avoid accidental accesses to specific files with security properties in host, but with build any mounts from host are disabled anyway. root user does not have a meaning for process execution permissions in here. For the file access, a better solution is also to map the user with user namespace.

@tonistiigi How hard would it be if mount=type=cache took uid and gid like ssh and secret mounts? My use case for a different user is that the yocto project build tool bitbake does not allow root builds so I have to make another user. Because of that I am used to just making a build user for every dockerfile I make to avoid headaches. It would be nice if all these new mount types had feature parity. Sometimes it's not just about security as we already know potentially dangerous commands are stripped out.

[pwaller]

Oops, I've hit this again, on a fresh project starting from scratch in a completely different context. Would be nice to have a simple workaround such as being able to set uid/gid as @arn95 suggests.

[ghost]

https://github.com/moby/buildkit/blob/master/frontend/dockerfile/docs/experimental.md I just tried setting the uid and gid with the cache mount type, and no luck, I'm assuming this feature is not supported yet?

[arncore]

Make sure you’re on master. The uid/gid for cache mounts commit hasn’t been merged into the versioned/release branches.

@tonistiigi Thank you!

[ghost]

Thanks for the information @arn95

[aquark]

This is fixed with # syntax=docker/dockerfile:1.1.3-experimental but that syntax is apparently not available for arm64 (it works on amd64).