Should images with different tags have different digests?

[sergeyfd]

Hello,

How digests for manifests are generated? I tagged the same image with three different tags and pushed them into 2.4.1 registry. Two of then have identical digests while third one is different. Is it supposed behavior? I know that in the older versions of registry digests were different for all tags, but I am not sure at what version this behavior changed.

-bash-4.2$ curl http://localhost:5000/v2/dubros/test/tags/list
{"name":"dubros/test","tags":["latest","t1","t2"]}
-bash-4.2$ curl --header "Accept: application/vnd.docker.distribution.manifest.v2+json" -I -X HEAD localhost:5000/v2/dubros/test/manifests/latest 2>/dev/null | grep Docker-Content-Digest | awk '{print $2}'
sha256:0a88770e824e5b7bcd15eec6da32f22e6d4e84c1fef9a5cdc5bbafd8917f9c31
-bash-4.2$ curl --header "Accept: application/vnd.docker.distribution.manifest.v2+json" -I -X HEAD localhost:5000/v2/dubros/test/manifests/t1 2>/dev/null | grep Docker-Content-Digest | awk '{print $2}'
sha256:2e1a1a37978fae1d0fa6aff2aada9a512d5c811eea9d5a174b8610744db6b103
-bash-4.2$ curl --header "Accept: application/vnd.docker.distribution.manifest.v2+json" -I -X HEAD localhost:5000/v2/dubros/test/manifests/t2 2>/dev/null | grep Docker-Content-Digest | awk '{print $2}'
sha256:2e1a1a37978fae1d0fa6aff2aada9a512d5c811eea9d5a174b8610744db6b103

-bash-4.2$ docker images
REPOSITORY                                                    TAG                 IMAGE ID            CREATED             SIZE
localhost:5000/dubros/test                                    latest              1feba5a478fe        9 months ago        204.9 MB
localhost:5000/dubros/test                                    t1                  1feba5a478fe        9 months ago        204.9 MB
localhost:5000/dubros/test                                    t2                  1feba5a478fe        9 months ago        204.9 MB
qa.registry.docker.site.gs.com/gs_cloud/httpd                 latest              1feba5a478fe        9 months ago        204.9 MB

[RichardScothern]

hi @sergeyfd they should have the same digest. I'm almost certain you are seeing #1771 (fix on its way), but if you can confirm I'll close this issue in favour of that.

[sergeyfd]

I am not sure that this is the same issue because I push same image, but with different tags. I am not even sure that this is an issue with the registry and not docker.

Having same digests for images with different tags is a bit dangerous, IMHO. One could want to delete an image with a particular tag so he'd obtain a digest for that image and would unexpectedly delete other images that are tagged differently but share the same digest.

BTW it looks like, images tagged and pushed by Docker 1.7 always get different digests.

[dmcgowan]

@sergeyfd it would be a docker issue to push different manifest digests since the registry does not alter the manifest on push. Can you confirm that is the case by looking at the Digest: .... output on push. If you are seeing this from the same engine doing a push, then there is some issue there. If you are pushing from different docker engines, then the manifest digest is not guaranteed even for the same image id.

[sergeyfd]

Yes, I can confirm that in Docker 1.11 digests are the same and in Docker 1.7 they are different. Shall I open an issue for Docker Engine?

[dmcgowan]

@sergeyfd digests being different in 1.7 is a known issue, in 1.10 we switched to content addressable storage and added cross repository push to mitigate this. If you want consistent digests on push you need 1.10 or newer.

[sergeyfd]

I'd rather have different digests for different tags, because in my view it's dangerous to have them the same for the reasons above.

As far as I understand manifests include tags so technically they are different, though they might include exactly the same set of layers.

[dmcgowan]

I understand what you are saying about the danger but we need to fix this by having better ways to cleanup images. I agree just always removing an image by digest can be risky and we should find a better solution for it.

As for the manifests, we have released schema 2 of the manifest which does strongly enforce that an image and a tag are separate, and therefore multiple tags can point to a single digest. This is now the default with pushed images.

[sergeyfd]

I would be absolutely satisfied if you had an API call to delete a tag :-) But unfortunately you do not. Are there any plans to introduce such call since tags and images are different entities?

[holgerreif]

@dmcgowan writes:

As for the manifests, we have released schema 2 of the manifest which does strongly enforce that an image and a tag are separate, and therefore multiple tags can point to a single digest.

Are you refering to the manifest v2-2 spec? Actually, there is no notion of a tag within that manifest.

So the relationship between tag and image is only maintaned within the storage data structures within the registry? A quick test seems to support this:

docker push localhost:5000myrepo/myimage:x
docker exec -ti registry bash
cd /var/lib/registry/docker/registry/v2/repositories/myrepo/myimage/_manifests/tags
cp -a x y
exit
docker pull localhost:5000myrepo/myimage:y

I would be absolutely satisfied if you had an API call to delete a tag

+1

I strongly support the differentiation between removing an image and removing a tag.

[SathishSuresh-opex]

Hey guys , any updates on this thread ? I am also facing similar issue, I have an image with tags - latest,dev and prod. When I try to delete dev tag from registry, even the latest and prod tags are getting deleted.

[sudo-bmitch]

Digests for manifests are calculated by hashing the raw manifest contents (e.g. running the curl output through sha256sum) and the manifest itself doesn't contain the tag reference. Instead the tag is a pointer to the manifest digest. Currently the only API call for deleting is to delete the entire image manifest, which results in all tags also being deleted. There been discussion on improving the tag APIs in OCI distribution-spec. Until something like that makes it into the spec and then implemented in registries, I'm hacking around this in regclient by pushing a temporary unique manifest to the tag and then delete that unique manifest.