vsoch / django-oci

Open Containers distribution spec module for Django (under development)
https://vsoch.github.io/django-oci/
Other
16 stars 1 forks source link

Anticipated bug: different repositories with blobs with the same digest #8

Open vsoch opened 4 years ago

vsoch commented 4 years ago

I anticipate a bug for the case when different repositories share images with the same blobs. Let's test and make sure this doesn't happen.

ad-m commented 4 years ago

The reuse of blobs by different images is the basis of image deduplication. To remove blobs for this reason, Docker Registry uses a garbage collector to remove blobs, which requires a long "stop-the-world" (make registry read-only). As far as I know, I have not analyzed very deeply, the limitation of such a limitation is the implementation of the OCI registry that can perform GC online.

How does this implementation intend to approach identifying layers that are no longer used?

vsoch commented 4 years ago

The actual query wouldn't be hard, it would just find blobs not linked to any image manifest. But currently, it makes no effort to check or cleanup, since it's a module intended for someone else to add to their Django project, I suspect they might want to have control over some cleanup strategy, however it might be done. The current storage is alongside the registry on the filesystem, but it's setup to allow for implementation of different storage backends, meaning you could imagine cloud storage, or something like Minio, and you can imagine the cleanup strategy would also vary based on that. So my thinking is that I'd first want to finish the module to add a few basic default (frontend interface) views, and then implement more storage backends. When those are done, I'm going to implement an example registry that uses it, and there I'm thinking it would be good to include some examples of cleanup. I might implement it on the level of the example, or if appropriate, could add it back here. So if you can provide what you have in mind that would give me a good start.