search

distribution / distribution

The toolkit to pack, ship, store, and deliver container content
https://distribution.github.io/distribution
Apache License 2.0
8.96k stars 2.49k forks source link

Zero byte file (layer) returned after GC when running with InMemory cache #2966

Open irbull opened 5 years ago

irbull commented 5 years ago

With Docker Registry version v2.6.2-14-ga66a4c3, after deleting a repository and running the GC, the REST endpoints still returned valid responses for manifests within that repository. However, when we tried to fetch layers, the endpoint returned a 0-byte file.

Because the manifest was successfully returned, docker push did not upload new layers, however, when other services (k8s in our case) tried to pull these images they would fail. This left the registry in a corrupt state with no easy path forward to fix it.

To fix this we removed blobdescriptor: inmemory from our configuration file.

glance- commented 4 years ago

I've looked into this to, and it looks like there's no way to force registry into dropping its blobdescriptor cache.

I'd suggest a simple signal is hooked up so that for example SIGHUP drops the inmemory blobdescriptor cache. As far as I've read the code the inMemoryBlobDescriptorCacheProvider has a Clear() function, so it shouldn't be that hard to plumb up.