search

moby / buildkit

concurrent, cache-efficient, and Dockerfile-agnostic builder toolkit
https://github.com/moby/moby/issues/34227
Apache License 2.0
7.85k stars 1.09k forks source link

exporting cache to harbor error #2479

Open weiwendi opened 2 years ago

weiwendi commented 2 years ago

buildkit version

buildctl github.com/moby/buildkit v0.9.2
buildkitd github.com/moby/buildkit v0.9.2 a14b4e097ae1dc7514c5febd6d75f742a166ea75

command

buildctl --addr tcp://buildkitd:1234 build --frontend=dockerfile.v0 \
             --local context=. --local dockerfile=. \
             --output type=image,name=hub.aiops.red/common/builder:v4,push=true \
             --export-cache type=registry,mode=max,ref=hub.aiops.red/common/builder:cache \
             --import-cache type=registry,ref=hub.aiops.red/common/builder:cache

errors

 => ERROR exporting cache                                                                                                                                                                            15.8s
 => => preparing build cache for export                                                                                                                                                              13.8s
 => => writing layer sha256:0f7fd9f8d114013b2d0a5b94d7d708fd7f5e76ba47862f755376b3e9bf3a99c3                                                                                                          0.1s
 => => writing layer sha256:116209eb403c9e3d691d48996430b189c9955b2c2784ed94124fcb70973fbbe6                                                                                                          0.0s
 => => writing layer sha256:35a4f197768941ef308d981a94f6d06fb77b9f2ba89dc04d2daf8292ee297315                                                                                                          0.0s
 => => writing layer sha256:38b1453721cb272cf531587daa8f79cc38641a9750dc269d7f2f08feb80f6602                                                                                                          0.0s
 => => writing layer sha256:3d938c97ec6a17ff0d82b6318f14ac4fd32f421eb25945afd3ecbb4df21a8b12                                                                                                          0.1s
 => => writing layer sha256:6ac12068f30ddb70a6c7973db73dfbad98097b209f571d6b33e445f12b53a4c0                                                                                                          1.5s
 => => writing layer sha256:74e27dc593d49a6d728dfe36976cb1469e076fbf3611e501fd030308cd212a80                                                                                                          0.0s
 => => writing layer sha256:780391780e2083631e4e53928ce31e5a28e6e3480b8cdedb02aa4a9f732c7dcf                                                                                                          0.0s
 => => writing layer sha256:90fe46dd819953eb995f9cc9c326130abe9dd0b3993a998e12c01d0218a0b831                                                                                                          0.0s
 => => writing layer sha256:a1d0c75327776413fa0db9ed3adcdbadedc95a662eb1d360dad82bb913f8a1d1                                                                                                          0.0s
 => => writing layer sha256:bbc37f14aded2d49bfac62dfa404755c9f1cadfee2b35933e4906f0054782888                                                                                                          0.0s
 => => writing layer sha256:d00a973d5fda7185a48f71764a6d081156770a44774a9404948ad7b16f193b6a                                                                                                          0.1s
 => => writing layer sha256:e6105bb37144c9a18dd4950f7388e19fdf3855626520789f9a562e83ea91fd18                                                                                                          0.0s
 => => writing config sha256:3dc4daec3225a15d14963ee4fb072083e6af7863729539e72246c9cc63beecad                                                                                                         0.1s
 => => writing manifest sha256:1920fe7cf510f24c9a473dc77db9dbf7673f3a3d2276a53b3a434a6e399f9955                                                                                                       0.1s
 => [auth] common/builder:pull,push token for hub.aiops.red                                                                                                                                    0.0s
------
 > exporting cache:
------
error: failed to solve: error writing manifest blob: failed commit on ref "sha256:1920fe7cf510f24c9a473dc77db9dbf7673f3a3d2276a53b3a434a6e399f9955": unexpected status: 404 Not Found
dardonkov commented 2 years ago

@weiwendi Hello, did you manage to solve this, what was the problem?

l4d2boomer commented 2 years ago

@weiwendi How did you solve the problem?

crazy-max commented 1 year ago

Was also able to repro on demo.goharbor.io: https://github.com/crazy-max/docker-build-push-action/actions/runs/4074732012/jobs/7020226741#step:7:415

#42 writing layer sha256:fb35763663536953b1c62525e2d3d34772e1d1a08b6cf8e2a22185e5f532ccf3 done
#42 writing config sha256:59f681c07b641ee8a546872b52d475be2ae111ded5c060b71e5088f3b231428e
#42 writing config sha256:59f681c07b641ee8a546872b52d475be2ae111ded5c060b71e5088f3b231428e 0.3s done
#42 writing manifest sha256:6a7bcd7517f1447682d422fc13f0ef3022bd4839e7a62d5d90361a5309c15c65
#42 preparing build cache for export 2.0s done
#42 writing manifest sha256:6a7bcd7517f1447682d422fc13f0ef3022bd4839e7a62d5d90361a5309c15c65 0.2s done
#42 ERROR: error writing manifest blob: failed commit on ref "sha256:6a7bcd7517f1447682d422fc13f0ef3022bd4839e7a62d5d90361a5309c15c65": unexpected status: 404 Not Found

Relevant BuildKit logs: https://github.com/crazy-max/docker-build-push-action/actions/runs/4074732012/jobs/7020226741#step:17:708

   time="2023-02-02T13:04:16Z" level=debug msg="do request" request.header.content-type=application/vnd.oci.image.index.v1+json request.header.user-agent=buildkit/v0.11 request.method=PUT spanID=3b4de2307190b3c1 traceID=173eca867cace87aeee153f0c2f2ef43 url="https://demo.goharbor.io/v2/build-push-action/test-docker-action/manifests/cache"
  time="2023-02-02T13:04:17Z" level=debug msg="fetch response received" response.header.connection=keep-alive response.header.content-length=174 response.header.content-type="application/json; charset=utf-8" response.header.date="Thu, 02 Feb 2023 13:04:17 GMT" response.header.server=nginx response.header.set-cookie="sid=8aa40db2ce950035f960341cd53ab06c; Path=/; HttpOnly" response.header.x-request-id=65e629b4-62bd-4370-b1b9-c325316d2655 response.status="404 Not Found" spanID=3b4de2307190b3c1 traceID=173eca867cace87aeee153f0c2f2ef43 url="https://demo.goharbor.io/v2/build-push-action/test-docker-action/manifests/cache"
  time="2023-02-02T13:04:17Z" level=debug msg="unexpected response" body="{\"errors\":[{\"code\":\"NOT_FOUND\",\"message\":\"artifact build-push-action/test-docker-action@sha256:0269c10e600f3a375f36ddabdbd264ce9503a455f0d0969ce8a00f24eaecc032 not found\"}]}\n" resp="&{404 Not Found 404 HTTP/1.1 1 1 map[Connection:[keep-alive] Content-Length:[174] Content-Type:[application/json; charset=utf-8] Date:[Thu, 02 Feb 2023 13:04:17 GMT] Server:[nginx] Set-Cookie:[sid=8aa40db2ce950035f960341cd53ab06c; Path=/; HttpOnly] X-Request-Id:[65e629b4-62bd-4370-b1b9-c325316d2655]] {0xc0025ff300} 174 [] false false map[] 0xc004e17500 0xc0004ff810}" spanID=3b4de2307190b3c1 traceID=173eca867cace87aeee153f0c2f2ef43
  time="2023-02-02T13:04:17Z" level=error msg="/moby.buildkit.v1.Control/Solve returned error: rpc error: code = Unknown desc = error writing manifest blob: failed commit on ref \"sha256:6a7bcd7517f1447682d422fc13f0ef3022bd4839e7a62d5d90361a5309c15c65\": unexpected status: 404 Not Found"
  error writing manifest blob: failed commit on ref "sha256:6a7bcd7517f1447682d422fc13f0ef3022bd4839e7a62d5d90361a5309c15c65": unexpected status: 404 Not Found
  1 v0.11.2 buildkitd --debug
  github.com/moby/buildkit/cache/remotecache.(*contentCacheExporter).Finalize
    /src/cache/remotecache/export.go:140
  github.com/moby/buildkit/solver/llbsolver.runCacheExporters.func1.1.1
    /src/solver/llbsolver/solver.go:605
  github.com/moby/buildkit/solver/llbsolver.inBuilderContext.func1
    /src/solver/llbsolver/solver.go:913
  github.com/moby/buildkit/solver.(*Job).InContext
    /src/solver/jobs.go:611
  github.com/moby/buildkit/solver/llbsolver.inBuilderContext
    /src/solver/llbsolver/solver.go:909
  github.com/moby/buildkit/solver/llbsolver.runCacheExporters.func1.1
    /src/solver/llbsolver/solver.go:586
  golang.org/x/sync/errgroup.(*Group).Go.func1
    /src/vendor/golang.org/x/sync/errgroup/errgroup.go:75
  runtime.goexit
    /usr/local/go/src/runtime/asm_amd64.s:1594

  1 v0.11.2 buildkitd --debug
  main.unaryInterceptor.func1
    /src/cmd/buildkitd/main.go:576
  github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1
    /src/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:25
  github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1
    /src/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:34
  github.com/moby/buildkit/api/services/control._Control_Solve_Handler
    /src/api/services/control/control.pb.go:2440
  google.golang.org/grpc.(*Server).processUnaryRPC
    /src/vendor/google.golang.org/grpc/server.go:1340
  google.golang.org/grpc.(*Server).handleStream
    /src/vendor/google.golang.org/grpc/server.go:1713
  google.golang.org/grpc.(*Server).serveStreams.func1.2
    /src/vendor/google.golang.org/grpc/server.go:965
  runtime.goexit
    /usr/local/go/src/runtime/asm_amd64.s:1594
OlivierKessler01 commented 11 months ago

Yup, seems like Harbor doesn't like it