Failing manifest upload from Github Action docker/build-push-action@v6

[hikkoiri]

Hi,

first of all: great project. I love it!

Right now I want to use this solution to store docker images which are directly assembled as part of a Github action pipeline. That is what I am using, nothing special and close to the example provided by docker.

jobs:
  CD:
    needs: [CI, Export]
    runs-on: [ubuntu-latest]
    name: Continuous delivery (Create and push shippable container image)
    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Set up QEMU
        uses: docker/setup-qemu-action@v3

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3

      - name: Login to registry
        uses: docker/login-action@v3
        with:
          registry: ${{ env.CONTAINER_REGISTRY }}
          username: ${{ secrets.CONTAINER_REGISTRY_USER }}
          password: ${{ secrets.CONTAINER_REGISTRY_PASSWORD }}

      - name: Build and push
        uses: docker/build-push-action@v6
        with:
          context: .
          push: true
          tags: ${{ env.CONTAINER_REGISTRY }}/${{ github.event.repository.name }}:${{ needs.Export.outputs.image_tag }}

I expected, that the build and push work out of the box, unfortunately I am faced by the following error message after the successful image build:

#16 [auth] sharing credentials for <redacted_endpoint>.<worker_subdomain>.workers.dev
#16 DONE 0.0s
#17 exporting to image
#17 exporting layers 0.0s done
#17 exporting manifest sha256:687c114eec0986641d418738f076f41173a5a6186ed890bafc18c85a2ff9ad4d done
#17 exporting config sha256:cd316114510a02721b9875402918cb60dd4fd76d962321b85f8754150620a8de done
#17 exporting attestation manifest sha256:146fc1d0d2a68c7f22eb73cd32b42cc6fa9376b81173eada519f6487662192e6 done
#17 exporting manifest list sha256:93c1778a31bd7a9f7dd29568974158581ec96afad7f99fd1aa29d80d1fc3778b done
#17 pushing layers
#17 pushing layers 3.9s done
#17 pushing manifest for <redacted_endpoint>.<worker_subdomain>.workers.dev/web:v1.1.1-SNAPSHOT@sha256:93c1778a31bd7a9f7dd29568974158581ec96afad7f99fd1aa29d80d1fc3778b
#17 12.84 error: failed commit on ref "index-sha256:93c1778a31bd7a9f7dd29568974158581ec96afad7f99fd1aa29d80d1fc3778b": unexpected status from PUT request to https://<redacted_endpoint>.<worker_subdomain>.workers.dev/v2/web/manifests/v1.1.1-SNAPSHOT: 500 Internal Server Error
#17 12.84 retrying in 1s

The action retries to upload the manifest a couple of times, before failing ultimately.

I was checking the worker logs and found following information (a bit redacted):

{
  "outcome": "ok",
  "exceptions": [],
  "logs": [
    {
      "message": [
        "Error putting manifest:",
        "zod error: - invalid_type: Required: config\n\t- invalid_type: Required: layers"
      ],
      "level": "error"
    }
  ],
  "event": {
    "response": {
      "status": 500
    }
  }
}

As far as I understand this is schema validation issue: https://github.com/cloudflare/serverless-registry/blob/main/src/manifest.ts

Unfortunately this is the point where I am not knowledgable enough to further understand what is causing the issue: Whether this originates from a sloppy implementation of the Github Action provided by Docker or if it is a compatibility issue with your API. I would love to see this combo work together and I assume that more people will use the same setup in the future.

Do you have any idea on how to fix this?

Already thank you very much. I appreciate your support.

Best regards

[gabivlj]

Thank you for submitting the issue. Could you check if https://github.com/cloudflare/serverless-registry/pull/54 fixes it for you? Cheers!

[hikkoiri]

Hi, thanks for the ultra fast support. Unfortunately another error occurs now:

#17 pushing layers 2.6s done
#17 pushing manifest for <redacted_endpoint>.<worker_subdomain>.workers.dev/web:v1.1.1-SNAPSHOT@sha256:1716b3d67b1a001fdd1fc41800734b8df3856c1c6b23e35ee8e7a94245b79959
#17 pushing manifest for <redacted_endpoint>.<worker_subdomain>.workers.dev/web:v1.1.1-SNAPSHOT@sha256:1716b3d67b1a001fdd1fc41800734b8df3856c1c6b23e35ee8e7a94245b79959 1.3s done
#17 ERROR: failed to push <redacted_endpoint>.<worker_subdomain>.workers.dev/web:v1.1.1-SNAPSHOT: failed commit on ref "manifest-sha256:ad7c5fe9fa9db4cbb8e22de4c5e2473b512cbcf2d782be741bc1deb6acc814ff": unexpected status from PUT request to https://<redacted_endpoint>.<worker_subdomain>.workers.dev/v2/web/manifests/sha256:ad7c5fe9fa9db4cbb8e22de4c5e2473b512cbcf2d782be741bc1deb6acc814ff: 400 Bad Request

Logs from the worker:

{
  "truncated": false,
  "outcome": "ok",
  "exceptions": [],
  "logs": [
    {
      "message": [
        "Digest sha256:43c4264eed91be63b206e17d93e75256a6097070ce643c5e8f0379998b44f170,sha256:5b19511a843df5d68c62b357426dd4e99e48fbeb9c085260de375065b969561f,sha256:652d69a25e853e561388e4ea6f55072df1747066277ef8310aff10d601150385,sha256:51676974aef5e1f3c046f2d40fa8e10d03a4c37e962e00f46bcfb5af242e81ad,sha256:bb16f69e8876d046e20b50c0873ac84b46e7b60926bbcc72a32765ad981cc732,sha256:6fb07faa0055e50ddac110c0d0b6286235e9bd9c0d4de00f0dcb5860dd5833a6,sha256:c298c5a0cd21956f1dec93f16c6968b7b009b43f22add9e78d18273bb91661f5,sha256:0c02f601d0eed2923ae2087212c9c0753846732b22db5f2088ec0daf62387e12,sha256:07f6d20ee159c897bf9fce0462d52f39c7ab530be246f3818bb30cc688f804ee doesn't exist"
      ],
      "level": "error",
    }
  ],
  "event": {
    "request": {
      "url": "https://<redacted_endpoint>.<worker_subdomain>.workers.dev/v2/web/manifests/sha256:REDACTED",
      "method": "PUT",
    },
    "response": {
      "status": 400
    }
  }
}

I cross-checked some of the digests and was able to find files with the same name in the blob directory.

[gabivlj]

Sorry, it was a very silly mistake :). I pushed another commit. Can you try again?

[hikkoiri]

The 500 is back.

Github Runner:

Error: buildx failed with: ERROR: failed to solve: failed to push<redacted_endpoint>.<worker_subdomain>.workers.dev/web:v1.1.1-SNAPSHOT: failed commit on ref "index-sha256:3bc21a00b7ae21f2b883067a22274dd700590d01dce0ce4dfc39ff25020096e2": unexpected status from PUT request to https:/<redacted_endpoint>.<worker_subdomain>.workers.dev/v2/web/manifests/v1.1.1-SNAPSHOT: 500 Internal Server Error

Worker logs:

 "logs": [
    {
      "message": [
        "Error putting manifest:",
        "zod error: - invalid_union: Invalid input: "
      ],
      "level": "error"
    }
  ],

[gabivlj]

Probably it's a multiarch manifest. Can you try again? :pray:

[hikkoiri]

Yes, but the error message still stays the same:

"zod error: - invalid_union: Invalid input: "

[gabivlj]

Hello! Good news is that I was able to repro and fix the issue. For the record, this is what I did to repro locally:

1. Have a Dockerfile, anything works really.

   FROM ubuntu:latest

2. docker buildx it with multiarch options.

   
   # Create insecure builder so we can push to localhost (this works in linux, maybe you have to do more in macos)
   docker buildx create --name ci-builder-2 --driver-opt network=host --use --buildkitd-flags '--allow-insecure-entitlement security.insecure' --use

Build and push

docker buildx build --push \
--driver-opt network=host --platform linux/arm64/v8,linux/amd64 --tag localhost:9999/multi:latest .

With the new changes, this works now. Let me know if you can confirm that it also works in the github runner.

[hikkoiri]

Hell yeah, it works now. Thank you so much for your support @gabivlj ! Kudos to you.