Replicate delete operation cause error

[likunbyl]

I deployed two set of harbor 1.3.0 in my kubernetes cluster, and replicate repositories from one to another.

I used the following command to delete over 100 tags of one repository:

for tag in `cat tag-file`; do
curl -u admin:Harbor12345 -X DELETE ui-1:8080/api/repositories/ai/moop/tags/$tag
done

And then, some replications failed, with logs like this:

2018-02-09T10:18:10+08:00 [INFO] initialization completed: repository: ai/moop, tags: [20171228-1046], destination URL: https://registry2.prod.com, insecure: true, destination user: admin
2018-02-09T10:18:11+08:00 [ERROR] [delete.go:119]: an error occurred while deleting repository ai/moop:20171228-1046 on https://registry2.prod.com with user admin: 500 {"errors":[{"code":"UNKNOWN","message":"unknown error","detail":{"Path":"/docker/registry/v2/repositories/ai/moop/_manifests/tags/20171222-1532/current/link","DriverName":"filesystem"}}]}

It seems that when we are deleting the tag 20171228-1046, it tries to delete 20171222-1532 , what's happened here ? The log about deleting tag 20171222-1532:

2018-02-09T10:18:02+08:00 [INFO] initialization completed: repository: ai/moop, tags: [20171222-1532], destination URL: https://registry2.prod.com, insecure: true, destination user: admin
2018-02-09T10:18:12+08:00 [INFO] repository ai/moop:20171222-1532 on https://registry2.prod.com has been deleted

I noticed that the time period is overlapped.

How can I handle this? Is that ok to just delete the tags from the _manifests directory of the second server ?

likun

[reasonerjt]

Are these tags pointing to the same image?

As for the 500 it looks like due to https://github.com/vmware/harbor/issues/4214

[likunbyl]

I don't know if these tags has the same digest, but they were deleted successfully at local server, these errors were happened at replication time.

[ywk253100]

It's a dup of #4214. Closing this one.

[likunbyl]

Why it's a dup of #4214 ? What I mean by 'deleted successfully at local server' is that these images are not the same image. @ywk253100

[ywk253100]

The image can only be deleted by digest. When you try to delete 20171228-1046, Harbor's backend gets the digest of 20171228-1046 first, and then calls API of the registry to delete the digest. In registry, all tags refer to this digest will be deleted. So you saw 20171222-1532 in the log.

There is a bug in the registry when trying to delete the same images concurrently which is mentioned in #4214.

[likunbyl]

If 20171222-1532 and 20171228-1046 had the same digest, it would prompt error when I deleted these images at local registry server. Maybe they were deleted not concurrently at local registry server, while concurrently at replicate time. But these images are built through these steps: rsync code from somewhere, and then docker build it, not through docker tag, so there's very little possibiliby for these images have the same digest.