Buildx does not allow oci-layout --context to be readonly.

thesayyn commented 4 weeks ago

Contributing guidelines

[x] I've read the contributing guidelines and wholeheartedly agree

I've found a bug and checked that ...

[x] ... the documentation does not mention anything about my problem
[x] ... there are no open or closed issues that are related to my problem

Description

Currently when attaching an oci-layout context to buildx, it wants to create some lock/ingest files within the target directory even though those files are not used in an meaningful way in this instance as far as in can see.

Here it wants to create index.json.lock if the digest is not provided

https://github.com/docker/buildx/blob/746eadd16e461ac1a961bb22e7fea8f1d05da88f/build/opt.go#L509

And here if you provide a tag.

https://github.com/docker/buildx/blob/746eadd16e461ac1a961bb22e7fea8f1d05da88f/build/opt.go#L514

Context:

I am trying to build a Bazel ruleset to allow running buildx under Bazel, and to attach contexts to buildx so it can be copied into the resulting container, however, as a accidental write prevention measure, Bazel marks artifacts from other build rules as readonly which is where buildx fails because it assumes the stores to be writeable even though it does not need them to be writable in this instance.

Expected behaviour

I'd expect the oci-layout store to not create these ingestion/lock files eagerly.

Actual behaviour

oci-layout store creates ingest/lock files eagerly preventing the directories to be marked readonly.

Buildx version

v0.17.1

Docker info

No response

Builders list

NAME/NODE         DRIVER/ENDPOINT             STATUS  PLATFORMS
builder-docker    docker-container                    
  builder-docker0 unix:///var/run/docker.sock running linux/amd64, linux/amd64/v2, linux/amd64/v3, linux/amd64/v4, linux/386
default *         docker                              
  default         default                     running linux/amd64, linux/386

Configuration

FROM ghcr.io/kamilkisiela/graphql-hive/apollo-router:f73977ea2219fb16a77602fd7e7c79069f1447fa

SHELL ["/bin/bash", "-c"]

RUN apt-get -y update
RUN apt-get install --no-install-recommends -y jq
RUN apt-get clean
RUN rm -rf /var/lib/apt/lists/*

Build logs

buildx build . --file services/applications/apollo-router/Dockerfile --builder builder-docker '--output=type=oci,tar=false,dest=bazel-out/k8-fastbuild/bin/services/applications/apollo-router/build' '--build-context=ghcr.io/kamilkisiela/graphql-hive/apollo-router:f73977ea2219fb16a77602fd7e7c79069f1447fa=oci-layout://bazel-out/k8-fastbuild/bin/external/rules_oci~~oci~o_apollo-router_linux_amd64/layout@sha256:7554a215f5d770c12bc2bf3486b48430ee867209a9e1ae146c62121eb6f93d35')

ERROR: invalid store at bazel-out/k8-fastbuild/bin/external/rules_oci~~oci~o_apollo-router_linux_amd64/layout: mkdir bazel-out/k8-fastbuild/bin/external/rules_oci~~oci~o_apollo-router_linux_amd64/layout/ingest: permission denied

Additional info

FWIW, i filed this issue in containerd repo https://github.com/containerd/containerd/issues/10885 as one of the offending library seems to be coming from there.

tonistiigi commented 4 weeks ago

I can reproduce the problem with --build-context directory when it is mounted readonly. But the error is

ERROR: oci-layout reference "oci-layout://out2" could not be resolved: could not lock<dir>/index.json.lock: open <dir>/index.json.lock: read-only file system

Because it tries to create a file lock. Maybe you are on a system that doesn't support file locks, so you don't get that error. If I suppress error with file locks and don't have ingest directory in my build context then I do get the error with that path as well. If you don't get an issue with file locks, then the workaround is to just make sure that an empty ingest directory already exists in the build-context directory.

thesayyn commented 4 weeks ago

then the workaround is to just make sure that an empty ingest directory already exists in the build-context directory.

i see that could be useful as a workaround, makes sense to create that folder when creating the oci-layout, i'll look into that but it would be great if this was fixed here.

Maybe you are on a system that doesn't support file locks

oh the reason i don't get the index.json.lock issue is because i use a digest (to workaround) so that codepath is never hit. right here https://github.com/docker/buildx/blob/746eadd16e461ac1a961bb22e7fea8f1d05da88f/build/opt.go#L508

tonistiigi commented 4 weeks ago

PRs: https://github.com/moby/buildkit/pull/5452 https://github.com/containerd/containerd/pull/10894

thesayyn commented 3 weeks ago

Thank you @tonistiigi, i can confirm that creating ingest and index.json.lock works as a workaround.

tonistiigi commented 3 weeks ago

The lock side of this has been fixed in buildx v0.18.0 , the "ingest" side will need to wait for containerd libs update.

thompson-shaun commented 2 days ago

backport to containerd 1.7.x can be tracked at https://github.com/containerd/containerd/pull/10913

thompson-shaun commented 2 days ago

Looks like containerd v1.7 should have a backport and any versions beyond would have the fixes relaated to this time. Closing this issue. @thesayyn LMK if we need to re-open this

docker / buildx