Failed to push image: failed commit on ref

ArnaudPalgen commented 1 year ago

Contributing guidelines

[X] I've read the contributing guidelines and wholeheartedly agree

I've found a bug and checked that ...

[ ] ... the documentation does not mention anything about my problem
[ ] ... there are no open or closed issues that are related to my problem

Description

I'm trying to build and upload a docker image to a private repository.

No problem when I export the image locally on my machine.

I have seen several issues of this kind, but none of them solved the problem. Maybe someone has a list of things to do for this kind of issue ?

Expected behaviour

The image is sent to the repository.

Actual behaviour

I have the following error:

ERROR: failed to solve: failed to push REDACTED/python-docker:1.0.0: failed commit on ref "index-sha256:3be21126cdfd0a02270874f65545c8c631b0fb17005ad10b7cd4c871110a21d7": unexpected status: 500 Server Error

Buildx version

github.com/docker/buildx v0.10.0 876462897612d36679153c3414f7689626251501

Docker info

No response

Builders list

NAME/NODE           DRIVER/ENDPOINT             STATUS  BUILDKIT PLATFORMS
my-builder *  docker-container
  my-builder0 unix:///var/run/docker.sock running v0.11.3  linux/arm64, linux/amd64, linux/amd64/v2, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/mips64le, linux/mips64, linux/arm/v7, linux/arm/v6
default             docker
  default           default                     running 20.10.22 linux/arm64, linux/amd64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6
desktop-linux       docker
  desktop-linux     desktop-linux               running 20.10.22 linux/arm64, linux/amd64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6

Configuration

To be sure that the bugs does not come from my Dockerfile, I tested with an example Dockerfile that's comes from https://docs.docker.com/language/python/build-images/

FROM python:3.8-slim-buster

WORKDIR /app

COPY requirements.txt requirements.txt
RUN pip3 install -r requirements.txt

COPY . .

CMD [ "python3", "-m" , "flask", "run", "--host=0.0.0.0"]

The following command is used:

docker buildx build -t REDACTED/python-docker:1.0.0 -t REDACTED/python-docker:latest --platform linux/arm64 --push .

Logs

time="2023-02-27T09:05:16Z" level=debug msg="unexpected response" body="{\"errors\":[{\"code\":\"UNKNOWN\",\"message\":\"unknown\",\"detail\":null}]}"
time="2023-02-27T09:05:16Z" level=error msg="/moby.buildkit.v1.Control/Solve returned error: rpc error: code = Unknown desc = failed to push REDACTED/python-docker:1.0.0: failed commit on ref \"index-sha256:e4c40d1fbf086be3c817d8a4a4e6cd66e9c5f6ea140eb58e9b02a384d987c6aa\": unexpected status: 500 Server Error"
failed to push REDACTED/python-docker:1.0.0: failed commit on ref "index-sha256:e4c40d1fbf086be3c817d8a4a4e6cd66e9c5f6ea140eb58e9b02a384d987c6aa": unexpected status: 500 Server Error
1 v0.11.3 buildkitd --debug
github.com/moby/buildkit/exporter/containerimage.(*imageExporterInstance).Export
    /src/exporter/containerimage/export.go:310
github.com/moby/buildkit/solver/llbsolver.(*Solver).Solve.func7
    /src/solver/llbsolver/solver.go:542
github.com/moby/buildkit/solver/llbsolver.inBuilderContext.func1
    /src/solver/llbsolver/solver.go:913
github.com/moby/buildkit/solver.(*Job).InContext
    /src/solver/jobs.go:611
github.com/moby/buildkit/solver/llbsolver.inBuilderContext
    /src/solver/llbsolver/solver.go:909
github.com/moby/buildkit/solver/llbsolver.(*Solver).Solve
    /src/solver/llbsolver/solver.go:541
github.com/moby/buildkit/control.(*Controller).Solve
    /src/control/control.go:424
github.com/moby/buildkit/api/services/control._Control_Solve_Handler.func1
    /src/api/services/control/control.pb.go:2438
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1
    /src/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:25
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.UnaryServerInterceptor.func1
    /src/vendor/go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc/interceptor.go:325
main.unaryInterceptor.func1
    /src/cmd/buildkitd/main.go:572
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1
    /src/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:25
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1
    /src/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:34
github.com/moby/buildkit/api/services/control._Control_Solve_Handler
    /src/api/services/control/control.pb.go:2440
google.golang.org/grpc.(*Server).processUnaryRPC
    /src/vendor/google.golang.org/grpc/server.go:1340
google.golang.org/grpc.(*Server).handleStream
    /src/vendor/google.golang.org/grpc/server.go:1713
google.golang.org/grpc.(*Server).serveStreams.func1.2
    /src/vendor/google.golang.org/grpc/server.go:965
runtime.goexit
    /usr/local/go/src/runtime/asm_arm64.s:1172

1 v0.11.3 buildkitd --debug
main.unaryInterceptor.func1
    /src/cmd/buildkitd/main.go:576
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1
    /src/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:25
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1
    /src/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:34
github.com/moby/buildkit/api/services/control._Control_Solve_Handler
    /src/api/services/control/control.pb.go:2440
google.golang.org/grpc.(*Server).processUnaryRPC
    /src/vendor/google.golang.org/grpc/server.go:1340
google.golang.org/grpc.(*Server).handleStream
    /src/vendor/google.golang.org/grpc/server.go:1713
google.golang.org/grpc.(*Server).serveStreams.func1.2
    /src/vendor/google.golang.org/grpc/server.go:965
runtime.goexit
    /usr/local/go/src/runtime/asm_arm64.s:1172

Additional info

I have tried the following things (not in order):

docker buildx prune
Remove the created builder and create a new one
Stop Docker, remove the .docker directory and restart docker
Use Kubernetes builder
Try different target architecture
Push to another repository (Gitlab). It also fails

jedevc commented 1 year ago

What registry are you using? The response you get:

time="2023-02-27T09:05:16Z" level=debug msg="unexpected response" body="{\"errors\":[{\"code\":\"UNKNOWN\",\"message\":\"unknown\",\"detail\":null}]}"

This looks very strange - getting a 500 response in an internal registry seems like an external problem - I don't suppose you have logs from your registry that you can share?

ArnaudPalgen commented 1 year ago

What registry are you using? The response you get:
time="2023-02-27T09:05:16Z" level=debug msg="unexpected response" body="{\"errors\":[{\"code\":\"UNKNOWN\",\"message\":\"unknown\",\"detail\":null}]}"
This looks very strange - getting a 500 response in an internal registry seems like an external problem - I don't suppose you have logs from your registry that you can share?

The problem is not with my repository. I tried on Gitlab and I have the same problem except that Gitlab gives me a 404 error. When I use docker build, I have no problem.

jedevc commented 1 year ago

Out of curiosity, does setting --provenance=false succeed?

ArnaudPalgen commented 1 year ago

Out of curiosity, does setting --provenance=false succeed?

Yes ! But why ?

jedevc commented 1 year ago

I suspect it's something to do with an older registry that doesn't support OCI images (the standard that the industry is moving to, instead of docker-specific media types). With BuildKit v0.11, we produce attestations, which automatically require using OCI media types.

Asked again - what's the software powering your registry? From your logs I'm guessing harbor? If so, that seems similar to https://github.com/docker/build-push-action/issues/780#issuecomment-1408361775, my suggested solution for that was:

What version of harbor registry are you using? I suspect it's likely you're using version 1? Harbor v2 has been out since 2020, and was one of the first registries to support the OCI image and distribution standards. I'd recommend an upgrade to the latest version - this should fix your issue without needing to set provenance: false.

Does pushing to GitLab with --provenance=false work? If it still doesn't, that's a separate issue. I've not seen any users struggling with pushing to GitLab, 404s are a not-found error, maybe a typo or something in the registry name?

ArnaudPalgen commented 1 year ago

Sorry, the repository is Nexus OSS 3.30.1-01

Note that a few months ago, I was able to push multi-architecture images built with buildx without any problem.

reza-sadrinia commented 1 year ago

I have the same problem And I am trying to use the nexus registry, but I am facing the same error And finally I had to use docker private registry https://docs.docker.com/registry/

crazy-max commented 11 months ago

Sorry, the repository is Nexus OSS 3.30.1-01

Note that a few months ago, I was able to push multi-architecture images built with buildx without any problem.

Nexus supports OCI index spec since 3.47.1, see https://github.com/docker/build-push-action/pull/800 and our e2e tests: https://github.com/docker/build-push-action/actions/runs/6453977865/job/17518599615#step:11:393

Like @jedevc said, you can disable provenance if you registry does not support OCI index.

snakeek commented 3 months ago

Out of curiosity, does setting --provenance=false succeed?

Work for me , thanks

docker / buildx