Reuploading a deleted image fails with a 404 error

Expected behavior and actual behavior:

When deleting an image, and then reuploading exactly the same image, harbor fails with the following error:

DEBU[0057] PUT https://harbor/v2/ci/myimage/manifests/12y7wy47kpjz4zzjljfl8pn3zhcgzbli
DEBU[0057] Error uploading manifest 12y7wy47kpjz4zzjljfl8pn3zhcgzbli to harbor/ci/myimage: unknown: http status code: 404, body: {"errors":[{"code":"MANIFEST_UNKNOWN","message":"manifest unknown","detail":{"Name":"ci/myimage","Revision":"sha256:1217a73a782bdf510904350c700a817f0f7fd24f359b0e2b076bbf025c56bb09"}}]}
 while writing manifest "{\"schemaVersion\":2,\"mediaType\":\"application/vnd.docker.distribution.manifest.v2+json\",\"config\":{\"mediaType\":\"application/vnd.docker.container.image.v1+json\",\"size\":9423,\"digest\":\"sha256:15ff9633ccfe0683ac45caa90a406a042448d473fc1fd5e9dccf0f7fa86c3b1f\"},\"layers\":[{\"mediaType\":\"application/vnd.docker.image.rootfs.diff.tar.gzip\",\"size\":3706184842,\"digest\":\"sha256:d9de1611b9961feedf3d0917c306c962104777e8f88c8c303e2f102798feb795\"}]}"

We expect the image to be uploaded without issues.

Steps to reproduce the problem:

NOTE: I'm not sure this reproducer works - I made a reproducer for the underlying issue in distribution over at #4269

# Upload an image. I used skopeo for this
skopeo --insecure-policy copy --dest-tls-verify=false "docker://alpine:3.19.1" "docker://harbor/ci/alpine:3.19.1" --multi-arch all

# We can verify that the push worked by downloading its manifest. So far so good.
curl -H 'Accept:application/vnd.docker.distribution.manifest.v2+json' http://harbor/v2/ci/alpine/manifests/3.19.1

# Next up, let's delete the manifest. This will delete from the registry, and run its garbage-collection
curl -XDELETE http://harbor/v2/ci/alpine/manifests/3.19.1

# And finally, let's try pushing the same image again. This will fail with the MANIFEST_UNKNOWN error.
skopeo --insecure-policy copy --dest-tls-verify=false "docker://alpine:3.19.1" "docker://harbor/ci/alpine:3.19.1" --multi-arch all

Versions:

harbor version: v2.10.0-6abb4eab
docker engine version: N/A
docker-compose version: N/A

Additional context:

This seems to stem from an upstream bug in distribution's garbage collector, which does not clean up the redis cache, leading to an incoherent cache and all kind of niceness. See upstream bug @ https://github.com/distribution/distribution/issues/4269

goharbor / harbor

Reuploading a deleted image fails with a 404 error #19988