Closed shoneefd closed 1 month ago
@shoneefd thanks for reporting this. How exactly are you using zot in your setup?
@rchincha We have two separate zot servers in each of our development and production environments. One is the primary; we use it as an OCI registry for images needed by containerized workloads running on, in the case of the production environment, thousands or more machines. It is this primary server to which, up to now, all of our direct OCI operations have been directed, and thus all of these malformed blobs are on that server. The second server is a mirror, which we keep synchronized to the primary server by way of the sync plugin, which does not retrieve these malformed blobs. (As a result, at least in the development environment, the primary server currently has almost 100GB more data in its storage than our mirror, which should be in sync.)
Originally, the mirror was only intended as a storage backup of the contents of the primary server. However, as traffic has scaled up we have decided to start using both servers as endpoints for the registry. This makes it doubly important that GC cleanup happens properly, as otherwise this junk data will not only pile up, it will bring our servers even farther out of sync, making it more difficult to tell which inconsistencies are caused by junk data and which are cause for more concern.
@shoneefd https://github.com/project-zot/zot/pull/2599 has been merged, pls. go ahead and build zot from main and try. This will be included in the next release.
If you confirm this is fixed, pls close the issue.
@shoneefd closing this issue since the PR is now merged and we haven't heard anything back.
Pls. re-open if you disagree.
zot version
2.0.4
Describe the bug
We have a registry server which has several storage blobs in a malformed state with no live blobs and large files trapped permanently in the
.uploads
subdirectory. Here is an example of the observed directory structure:In total, eleven blocks of this malformed type are taking up nearly 100 GB of junk space. We have enabled both the GC and the scrub plugin, and neither appears capable of clearing out these blocks. Here are the relevant parts of our
config.json
:To reproduce
We suspect this issue occurs when a blob upload in-progress is canceled, but have not yet reproduced
Expected behavior
The GC should clean up these blocks.
Screenshots
No response
Additional context
No response