Manifest 'deleted' but still in volume

[lvthillo]

I'm using a registry container version 2.4 on Docker 1.12.0 I push an image in my registry. After that I try to find the revision of that image (sha356:xxx) based on its tag to delete the image.

curl -k -v --silent -H "Accept: application/vnd.docker.distribution.manifest.v2+json" -X GET http://localhost:5000/v2/myproj/busybox/manifests/latest 2>&1 | grep Docker-Content-Digest | awk '{print ($3)}'
sha256:1359608115b94599e5641638bac5aef1ddfaa79bb96057ebf41ebc8d33acf8a7

Now I delete it:

curl -k -v --silent -X DELETE http://localhost:5000/v2/myproj/busybox/manifests/sha256:1359608115b94599e5641638bac5aef1ddfaa79bb96057ebf41ebc8d33acf8a7
* About to connect() to localhost port 5000 (#0)
*   Trying ::1...
* Connected to localhost (::1) port 5000 (#0)
> DELETE /v2/myproj/busybox/manifests/sha256:1359608115b94599e5641638bac5aef1ddfaa79bb96057ebf41ebc8d33acf8a7 HTTP/1.1
> User-Agent: curl/7.29.0
> Host: localhost:5000
> Accept: */*
>
< HTTP/1.1 202 Accepted
< Docker-Distribution-Api-Version: registry/2.0
< X-Content-Type-Options: nosniff
< Date: Tue, 06 Sep 2016 15:57:36 GMT
< Content-Length: 0
< Content-Type: text/plain; charset=utf-8
<
* Connection #0 to host localhost left intact

It's accepted but when I'm going to check inside my docker volume: I still see: cd /var/lib/docker/volumes/registry-volume/_data/docker/registry/v2/repositories/myproj/busybox/_manifests/revisions/sha256/1359608115b94599e5641638bac5aef1ddfaa79bb96057ebf41ebc8d33acf8a7/

I'm able to run the GC.:

docker exec -i aadbac436c85 bin/registry garbage-collect  /etc/docker/registry/config.yml
time="2016-09-06T16:17:39Z" level=info msg="Deleting blob: /docker/registry/v2/blobs/sha256/13/1359608115b94599e5641638bac5aef1ddfaa79bb96057ebf41ebc8d33acf8a7" go.version=go1.6.3 instance.id=2d3367f3-9471-4450-92ef-71dcc62b4994
time="2016-09-06T16:17:39Z" level=info msg="Deleting blob: /docker/registry/v2/blobs/sha256/2b/2b8fd9751c4c0f5dd266fcae00707e67a2545ef34f9a29354585f93dac906749" go.version=go1.6.3 instance.id=2d3367f3-9471-4450-92ef-71dcc62b4994
time="2016-09-06T16:17:39Z" level=info msg="Deleting blob: /docker/registry/v2/blobs/sha256/8d/8ddc19f16526912237dd8af81971d5e4dd0587907234be2b83e249518d5b673f" go.version=go1.6.3 instance.id=2d3367f3-9471-4450-92ef-71dcc62b4994

That seems to work fine: I went from 672k to 12k. But the problem is that the revision is still in my container. It isn't deleted. I'm not able to delete it again (404 not found). I'm also not able to run the GC again.

Second issue is repushing.:

[root@master1 registry]# docker push localhost:5000/myproj/busybox:latest
The push refers to a repository [localhost:5000/myproj/busybox]
8ac8bfaff55a: Layer already exists
latest: digest: sha256:1359608115b94599e5641638bac5aef1ddfaa79bb96057ebf41ebc8d33acf8a7 size: 527
[root@master1 registry]# cd /var/lib/docker/volumes/registry-volume/_data/docker/registry/
[root@master1 registry]# du -sh
24K

Only 24k used what isn't possbile. Deleting the 'new push' isn't possible too.

[lvthillo]

Restarting makes it possbile to repush images. The GC does not seem to clean up my blobs which are still growing..

[mortensteenrasmussen]

Hi @lorenzvth7

This repo belongs to the old and deprecated v1 repo. The new repo is at https://github.com/docker/distribution

And I believe there's already an issue about what you mention as the second issue. https://github.com/docker/distribution/issues/2094

You should restart your registry after doing a GC, and then you should be okay. You also need to put the registry into read-only mode when doing GC - if someone pushes while GC is running, bad things can happen. I'd personally advise you stop the registry, run GC, and then start it.

Edit: If this is no longer relevant, please close the issue :)