search

moby / buildkit

concurrent, cache-efficient, and Dockerfile-agnostic builder toolkit
https://github.com/moby/moby/issues/34227
Apache License 2.0
8.02k stars 1.12k forks source link

[v0.13 regression] buildkit/runc-native/snapshots/snapshots/10/var/cache/apt/archives/partial: permission denied #4789

Open reimgun opened 5 months ago

reimgun commented 5 months ago

buildkit version tried: 0.13.0 and 0.13.1

Error Message:

#9 [internal] load build context
#9 transferring context: 179B 0.0s done
#9 ERROR: error from sender: open /home/jenkins/workspace/prod-images-0/at_ubi9_minimal_filebeat/.local/share/buildkit/runc-native/snapshots/snapshots/10/var/cache/apt/archives/partial: permission denied
------
 > [internal] load build context:
------
error: failed to solve: error from sender: open /home/jenkins/workspace/prod-images-0/at_ubi9_minimal_filebeat/.local/share/buildkit/runc-native/snapshots/snapshots/10/var/cache/apt/archives/partial: permission denied

with Version 0.12.5 all is working perfectly fine.

AkihiroSuda commented 5 months ago

Please provide host OS, kernel version, filesystem, and a (minimal and yet self-contained) reproducer

reimgun commented 5 months ago

@AkihiroSuda container information where buildkit is installed: OS: NAME="Red Hat Enterprise Linux" VERSION="9.3 (Plow)" bash-5.1$ uname -a Linux jenkins-6d54b58dc5-2lxtg-gh-ubi-mc3kx 5.10.205-195.807.amzn2.x86_64 #1 SMP Tue Jan 16 18:28:59 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

container is running in kubernetes Version 1.27

we run buildkit now more then 1,5 years in that setup.

always worked well until version 0.13.0 and 0.13.1

AkihiroSuda commented 5 months ago

Please provide a reproducer

reimgun commented 5 months ago

buildctl-daemonless.sh build --frontend dockerfile.v0 --local context=/home/jenkins/workspace/prod-images-0/at_ubi9_minimal_filebeat --local dockerfile=/home/jenkins/workspace/prod-images-0/at_ubi9_minimal_filebeat --opt filename=Dockerfile --metadata-file metadata.json --output 'type=image,"name=****.azurecr.io/base-images-untested/at_ubi9_minimal_filebeat:8.12.2.3,*****.azurecr.io/base-images-untested/at_ubi9_minimal_filebeat:latest",push=true'

reimgun commented 2 months ago

@AkihiroSuda any update ? i have tested it also with 0.14.0 - it's the same ....

it happens also for other images:

#6 [internal] load build context
#6 transferring context: 129B 0.8s done
#6 ERROR: error from sender: open /home/jenkins/workspace/prod-images-0/at_ubi8_dotnet_radarlive_settingsmanager/build_images/.tar-scan: permission denied
------
 > [internal] load build context:
------
error: failed to solve: error from sender: open /home/jenkins/workspace/prod-images-0/at_ubi8_dotnet_radarlive_settingsmanager/build_images/.tar-scan: permission denied

could it be that it hast problems with .dirs like .local or .tar-scan ?

reimgun commented 2 months ago

@AkihiroSuda

here also the debug logs:

error: failed to solve: error from sender: open /home/jenkins/workspace/prod-images-0/at_ubi9_minimal_filebeat/.local/share/buildkit/runc-overlayfs/snapshots/snapshots/10/work/work: permission denied
7148 v0.14.1 buildkitd --oci-worker-no-process-sandbox --addr=unix:///run/user/1000/buildkit/buildkitd.sock
github.com/tonistiigi/fsutil.(*receiver).run.func2
    /src/vendor/github.com/tonistiigi/fsutil/receive.go:211
golang.org/x/sync/errgroup.(*Group).Go.func1
    /src/vendor/golang.org/x/sync/errgroup/errgroup.go:75
runtime.goexit
    /usr/local/go/src/runtime/asm_amd64.s:1650

7148 v0.14.1 buildkitd --oci-worker-no-process-sandbox --addr=unix:///run/user/1000/buildkit/buildkitd.sock
github.com/moby/buildkit/session/filesync.recvDiffCopy
    /src/session/filesync/diffcopy.go:108
github.com/moby/buildkit/session/filesync.FSSync
    /src/session/filesync/filesync.go:235
github.com/moby/buildkit/source/local.(*localSourceHandler).snapshot
    /src/source/local/source.go:264
github.com/moby/buildkit/source/local.(*localSourceHandler).Snapshot
    /src/source/local/source.go:153
github.com/moby/buildkit/solver/llbsolver/ops.(*SourceOp).Exec
    /src/solver/llbsolver/ops/source.go:108
github.com/moby/buildkit/solver.(*sharedOp).Exec.func2
    /src/solver/jobs.go:1100
github.com/moby/buildkit/util/flightcontrol.(*call[...]).run
    /src/util/flightcontrol/flightcontrol.go:122
sync.(*Once).doSlow
    /usr/local/go/src/sync/once.go:74
sync.(*Once).Do
    /usr/local/go/src/sync/once.go:65
runtime.goexit
    /usr/local/go/src/runtime/asm_amd64.s:1650

7148 v0.14.1 buildkitd --oci-worker-no-process-sandbox --addr=unix:///run/user/1000/buildkit/buildkitd.sock
github.com/moby/buildkit/solver.(*edge).execOp
    /src/solver/edge.go:979
github.com/moby/buildkit/solver/internal/pipe.NewWithFunction.func2
    /src/solver/internal/pipe/pipe.go:82
runtime.goexit
    /usr/local/go/src/runtime/asm_amd64.s:1650

7192 v0.14.1 buildctl --addr=unix:///run/user/1000/buildkit/buildkitd.sock --debug build --frontend dockerfile.v0 --local context=/home/jenkins/workspace/prod-images-0/at_ubi9_minimal_filebeat --local dockerfile=/home/jenkins/workspace/prod-images-0/at_ubi9_minimal_filebeat --opt filename=Dockerfile --metadata-file metadata.json --output type=image,"name=itmpproductshared.azurecr.io/base-images-untested/at_ubi9_minimal_filebeat:8.14.2.2,itmpproductshared.azurecr.io/base-images-untested/at_ubi9_minimal_filebeat:latest",push=true --opt build-arg:HTTP_PROXY=http://proxy.regional.devops-services.ec1.aws.aztec.cloud.allianz:8080/ --opt build-arg:http_proxy=http://proxy.regional.devops-services.ec1.aws.aztec.cloud.allianz:8080/ --opt build-arg:HTTPS_PROXY=http://proxy.regional.devops-services.ec1.aws.aztec.cloud.allianz:8080/ --opt build-arg:https_proxy=http://proxy.regional.devops-services.ec1.aws.aztec.cloud.allianz:8080/ --opt build-arg:NO_PROXY=localhost,127.0.0.0/8,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,44.0.0.0/8,.allianz,.aeat.allianz.at,.svc,.cluster.local,jmp.allianz.net,cmp.allianz.net,github.developer.allianz.io --opt build-arg:no_proxy=localhost,127.0.0.0/8,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,44.0.0.0/8,.allianz,.aeat.allianz.at,.svc,.cluster.local,jmp.allianz.net,cmp.allianz.net,github.developer.allianz.io --opt build-arg:PROXY_HOST=proxy.regional.devops-services.ec1.aws.aztec.cloud.allianz --opt build-arg:PROXY_PORT=8080 --opt build-arg:NONPROXYHOSTS=localhost|127.*|10.*|172.16.0.0/12|192.168.*|44.*|*.allianz|*.aeat.allianz.at|*.svc|*.cluster.local|jmp.allianz.net|cmp.allianz.net|github.developer.allianz.io
google.golang.org/grpc.(*ClientConn).Invoke
    /src/vendor/google.golang.org/grpc/call.go:35
github.com/moby/buildkit/api/services/control.(*controlClient).Solve
    /src/api/services/control/control.pb.go:2234
github.com/moby/buildkit/client.(*Client).solve.func2
    /src/client/solve.go:274
golang.org/x/sync/errgroup.(*Group).Go.func1
    /src/vendor/golang.org/x/sync/errgroup/errgroup.go:75
runtime.goexit
    /usr/local/go/src/runtime/asm_amd64.s:1650

7192 v0.14.1 buildctl --addr=unix:///run/user/1000/buildkit/buildkitd.sock --debug build --frontend dockerfile.v0 --local context=/home/jenkins/workspace/prod-images-0/at_ubi9_minimal_filebeat --local dockerfile=/home/jenkins/workspace/prod-images-0/at_ubi9_minimal_filebeat --opt filename=Dockerfile --metadata-file metadata.json --output type=image,"name=itmpproductshared.azurecr.io/base-images-untested/at_ubi9_minimal_filebeat:8.14.2.2,itmpproductshared.azurecr.io/base-images-untested/at_ubi9_minimal_filebeat:latest",push=true --opt build-arg:HTTP_PROXY=http://proxy.regional.devops-services.ec1.aws.aztec.cloud.allianz:8080/ --opt build-arg:http_proxy=http://proxy.regional.devops-services.ec1.aws.aztec.cloud.allianz:8080/ --opt build-arg:HTTPS_PROXY=http://proxy.regional.devops-services.ec1.aws.aztec.cloud.allianz:8080/ --opt build-arg:https_proxy=http://proxy.regional.devops-services.ec1.aws.aztec.cloud.allianz:8080/ --opt build-arg:NO_PROXY=localhost,127.0.0.0/8,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,44.0.0.0/8,.allianz,.aeat.allianz.at,.svc,.cluster.local,jmp.allianz.net,cmp.allianz.net,github.developer.allianz.io --opt build-arg:no_proxy=localhost,127.0.0.0/8,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,44.0.0.0/8,.allianz,.aeat.allianz.at,.svc,.cluster.local,jmp.allianz.net,cmp.allianz.net,github.developer.allianz.io --opt build-arg:PROXY_HOST=proxy.regional.devops-services.ec1.aws.aztec.cloud.allianz --opt build-arg:PROXY_PORT=8080 --opt build-arg:NONPROXYHOSTS=localhost|127.*|10.*|172.16.0.0/12|192.168.*|44.*|*.allianz|*.aeat.allianz.at|*.svc|*.cluster.local|jmp.allianz.net|cmp.allianz.net|github.developer.allianz.io
github.com/moby/buildkit/client.(*Client).solve.func2
    /src/client/solve.go:290
golang.org/x/sync/errgroup.(*Group).Go.func1
    /src/vendor/golang.org/x/sync/errgroup/errgroup.go:75
runtime.goexit
    /usr/local/go/src/runtime/asm_amd64.s:1650