project-zot / zot

zot - A scale-out production-ready vendor-neutral OCI-native container image/artifact registry (purely based on OCI Distribution Specification)
https://zotregistry.dev
Apache License 2.0
876 stars 93 forks source link

[Bug]: InvalidPart: All non-trailing parts must have the same length #2142

Open pablo-salad opened 9 months ago

pablo-salad commented 9 months ago

zot version

ghcr.io/project-zot/zot-minimal-linux-amd64:v2.0.0

Describe the bug

Pushing an image to Zot configured with remote store (Cloudflare R2) is reporting InvalidPart: All non-trailing parts must have the same length when is trying to complete a multipart upload.

To reproduce

We prepared a Docker compose to reproduce this error. Just clone this repository. You'll find a Zot image with the remote configuration and an Ubuntu image with Skopeo installed to push the image.

Configuration To setup the images, you'll need to have a Cloudflare account, create an R2 bucket and create a R2 API Token to access the bucket. After this, open compose.yaml and fill these fields:

To run the test, execute docker compose up in the folder where the repository was cloned.

Error After a few minutes, usually uploading a large layer, you will see InvalidPart: All non-trailing parts must have the same length error in the console.

Expected behavior

Able to push any image without errors using Zot with remote storage.

Screenshots

image

Additional context

The current Zot configuration uses remote storage, in our case, Cloudflare R2 which has S3 API compatibility. The size of the images uploaded to the registry have an average of 5GB. We had some luck pushing small images, less than 100MB, changing chunk size in the config but no success with larger ones.

peusebiu commented 9 months ago

Hello @pablo-salad

Thank you for trying zot and for the feedback.

This seems to be a known issue in the library we are using for s3. We are waiting for a fix on their side.

Here is the issue: https://github.com/distribution/distribution/issues/3873 Here is the PR which is fixing it: https://github.com/distribution/distribution/pull/3940

rchincha commented 9 months ago

https://github.com/opencontainers/distribution-spec/issues/485

^ also this.