search

neuro-inc / platform-registry-api

Other
1 stars 1 forks source link

Image is not deleted from the catalog in onprem #477

Open YevheniiSemendiak opened 3 years ago

YevheniiSemendiak commented 3 years ago

When you delete the last tag of image, the image itself should not be displayed in neuro image ls. This holds only for cluster deployments, where do not use managed image registry.

@anayden you told we have enabled garbage collector, but I didn't found where is it. Could you elaborate, please?

github-actions[bot] commented 2 years ago

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 14 days

github-actions[bot] commented 2 years ago

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 14 days

YevheniiSemendiak commented 2 years ago

Trying to reproduce, pushed image:test-garbage-collector:v1 image to onprem-poc cluster and removed it, awaiting the GC to process (does it once in a day).

image image
YevheniiSemendiak commented 2 years ago

The issue is still relevant:

image

@zubenkoivan, could you please check the garbage collector?

zubenkoivan commented 2 years ago

Hey @YevheniiSemendiak . This is not related to garbage collection. This feature is not supported by private docker registry instances. Please see the issue below. There is a PR that might solve it once merged.

Issue: https://github.com/distribution/distribution/issues/2314

PR: https://github.com/distribution/distribution/pull/3195

anayden commented 2 years ago

Cronjob was setup by Alin.

image image

BUT, looking at our chat, I see neither of us verified the files are actually deleted. So it might be, the job is running, but is not doing anyting.

YevheniiSemendiak commented 2 years ago

From the issues and PR Ivan mentioned, its seen that the blobs are actually deleted by the garbage-collect command, but the repositories without manifests (https://github.com/distribution/distribution/pull/3195). So, we are blocked by the uplink.

bigtfromaz commented 2 years ago

I am prepping to develop a basic open-source package primarily to list and remove tags and images when a repository admin no longer wants them available to the repository's consumers. I ran into this issue while prototyping my intended HTTP conversations in Postman.

In my findings we see that after an image is deleted from the repository using the Docker-Content-Digest, the repository is still listed in _catalog. The problem is the HTTP DELETE conversation corrupted the registry datastore.

When a repository is in this state, and some user or script issues a docker pull imagename command using the default tag, or any tag, they receive this message:

   `Error response from daemon: manifest for myhost.net/hello-world-old:latest not found: 
     manifest unknown: manifest unknown`

This message indicates a corrupt registry datastore because the response should be NOT FOUND.

When a utility like this API comes in from the side and manipulates the database, it should never corrupt the database. In my opinion, remediating this should be a high priority.