Blob Upload Invalid When Pushing to Local Distribution Registry

[thebeanogamer]

Environment:

• Jib version: 3.4.3
• Build tool: Gradle 8.9
• OS: Fedora Linux 40

Description of the issue:

When pushing to a local instance of https://github.com/distribution/distribution, the image will build correctly but fail to push to the registry with the below exception.

com.google.cloud.tools.jib.plugins.common.BuildStepsExecutionException: Tried to push BLOB for localhost:5000/example with digest sha256:f72f7ee7417d7dc1d181843bc32c8c5d47822e3ae0ac08aa7c72ac09055040c2 but failed because: other: blob upload invalid

Occasionally after retrying the push will appear to succeed, but clients will get stuck when they try and pull it so I don't think it's actually succeeded.

Expected behaviour:

The container should push to the registry and be pullable by OCI clients.

Steps to reproduce:

A repo with this ready can be found on https://github.com/thebeanogamer/jib-issue-poc.

1. Start a local container registry (hash is pinned for reproducibility, but I've seen this with many versions)

   docker run -d -p 5000:5000 --name registry docker.io/library/registry:2.8.3@sha256:12120425f07de11a1b899e418d4b0ea174c8d4d572d45bdb640f93bc7ca06a3d

2. Create an empty Java project using Gradle and add the below build script
3. Try to build and push the container

   ./gradlew jib

jib-gradle-plugin Configuration:

jib {
    container {
        mainClass = "com.example.Main"
        format = ImageFormat.OCI
    }
    setAllowInsecureRegistries(true)
    to {
        image = "localhost:5000/example:latest"
    }
}

Log output:

Registry and Gradle logs are on https://gist.github.com/thebeanogamer/b58cf7a4d9d3c86c2adb0b43dddfcb7d.

upload resumed at wrong offest on the registry side is interesting, but I don't know what to read into it. It looks quite similar to https://github.com/containers/buildah/issues/1133, but the registry fix there was merged long ago.

Additional Information:

We've only seen this issue on a few machines, despite them having the same OS packages and output from ./gradlew buildEnvironment.

Sometimes deleting ~/.cache/google-cloud-tools-java ~/.config/google-cloud-tools-java ~/.gradle .gradle build causes a build to succeed, but that may be coincidence.

Without the format = ImageFormat.OCI this setup hangs until it throws BrokenPipe.

[thebeanogamer]

It looks quite similar to https://github.com/containers/buildah/issues/1133, but the registry fix there was merged long ago.

This may not be entirely true. https://github.com/distribution/distribution/pull/3519 is linked as the fix there, and although it was merged in 2022, it hasn't hit any of the 2.X releases. It's also not the driver that distribution will use by default:

storage:
    filesystem:
        rootdirectory: /var/lib/registry

However, if I upgrade the distribution container to 3.0.0-beta.1 then this issue goes away. I'll search through the changelog to see if I can find the culprit.

I'm not going to close this yet as I'm not sure I've found the actual issue, and as help for future travellers.

[thebeanogamer]

Even weirder, the issues pushed to 3.0.0-beta.1 are pullable by Docker but not by Podman/Skopeo... I am honestly not even sure if this is a Jib issue at this point or if my computer is just haunted.

[blakeli0]

@thebeanogamer Did you find more information? We are closing this issue as it does not seem to be a Jib issue per if I upgrade the distribution container to 3.0.0-beta.1 then this issue goes away. Feel free to re-open it if needed.