Digests on Dockerhub and those fetched by docker pull do not match

[roy-work]

Problem description

The data on what SHAs an image on Dockerhub is on doesn't seem to match what I get from docker pull. For example, take ubuntu:bionic. If I look up that tag on Dockerhub, the lastest digest for amd64 is sha256:134c7fe821b9d359490cd009ce7ca322453f4f2d018623f849e580a89a685e5d. But if I run docker pull ubuntu:bionic, I get a different digest:

» docker pull ubuntu:bionic
bionic: Pulling from library/ubuntu
Digest: sha256:6e9f67fa63b0323e9a1e587fd71c561ba48a034504fb804fd26fd8800039835d
Status: Image is up to date for ubuntu:bionic

(And this digest doesn't match any architecture, not just amd64. I checked the others just in case it was for some odd reason pulling an x86 image.)

Why do these not match?

[roy-work]

Apparently, the name you use to fetch an image is embedded in the manifest.json inside the image, and this data comes into account when computing the digest.

I still don't know why you would do such a thing, nor do I know why the website, on the page for the tag itself, shows you the digest for fetching it as if you hadn't somehow specified a tag?

[manishtomar]

@roy-work The digest displayed by the CLI is the digest of manifest list that composes of multiple manifests each representing an image for a particular platform. The digests shown in Hub are digests of each of these platform images. The CLI actually downloads the 134c7... if your machine is amd64/linux but displays the digest of the top-level manifest list. Ideally it should display the amd64/linux to avoid such confusion. Or Hub UI can display the digest of the top-level manifest list.

[mms-gianni]

Is there a reliable way to check if the local running image is the same or actual one offered by docker hub?

[zxkane]

Pulling image with library prefix still get a digest not matching the manifest,

docker pull library/golang:1.11.1
1.11.1: Pulling from library/golang
Digest: sha256:63ec0e29aeba39c0fe2fc6551c9ca7fa16ddf95394d77ccee75bc7062526a96c
Status: Image is up to date for golang:1.11.1
docker.io/library/golang:1.11.1

Get the digest from manifest,

curl -H "Accept: application/vnd.docker.distribution.manifest.list.v2+json"  --header "Authorization: Bearer $token"  "https://registry-1.docker.io/v2/library/golang/manifests/1.11.1"
{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
   "manifests": [
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 1796,
         "digest": "sha256:df8db4a3a7dee9782e0f1bdcc9d676bc2de0dc1d2dc2952d9b9b3718445b1455",
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 1795,
         "digest": "sha256:faf19ab885f9d0779bbbd615eee610c5aedfdd203ac5fceefbd5fc8afd73a2ac",
         "platform": {
            "architecture": "arm",
            "os": "linux",
            "variant": "v7"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 1795,
         "digest": "sha256:311e6ce0d12e300c6529a64cc3ad14692fe413d864082bfd81346eee949ced35",
         "platform": {
            "architecture": "arm64",
            "os": "linux",
            "variant": "v8"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 1796,
         "digest": "sha256:7318908975c42670e6ebecb58ecb1729642ca331aa57904888eb1a0c7d5949b7",
         "platform": {
            "architecture": "386",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 1794,
         "digest": "sha256:5947b86ec0492e7bf3d017daf1243472a4e78fbca3a9e57d3313a4367c1d26a2",
         "platform": {
            "architecture": "ppc64le",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 1796,
         "digest": "sha256:da7bb1aa2efe46ffaa33b4da932e8038101fb3d93aa7350de9aa29f4fffb9318",
         "platform": {
            "architecture": "s390x",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 3246,
         "digest": "sha256:57e9ef57989766bf736f28c746e30a892a503a81a2053affc1dff185c7be5cbc",
         "platform": {
            "architecture": "amd64",
            "os": "windows",
            "os.version": "10.0.14393.2485"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 3245,
         "digest": "sha256:662c92894844017d3953f530bb179af71fbf2210bc33c053254b7c8bafcc6e4a",
         "platform": {
            "architecture": "amd64",
            "os": "windows",
            "os.version": "10.0.16299.665"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 3244,
         "digest": "sha256:c251086c392ed43eb8e1277412de4ab674efd72cb47d93da18539be70639f210",
         "platform": {
            "architecture": "amd64",
            "os": "windows",
            "os.version": "10.0.17134.285"
         }
      }
   ]
}

For image golang:1.11.1 on arch linux and x86_64 we got different digests. It makes us confused to maintain a mirror of some images.

[thaJeztah]

Some discussion on https://github.com/moby/moby/issues/40636#issuecomment-595985775 as well (w.r.t. the presentation of the manifest digest for multi-arch images)

[Smyl3]

So, as I see there is no such way to verify and match a downloaded image is the same as on the docker hub despite there are everything have tons of unique id-s. (I wanted to check which image is download in reality related to arches.) I tried to figure this out, and checked all of id-s, digests, sha-s but nothing matched. I checked all id-s on hub and inspected images with CLI and looked on portainer locally too, but nothing. Ridiculous.

[pcworld]

Actually, passing -v to curl in the API call proposed in https://github.com/docker/hub-feedback/issues/1925#issuecomment-638036979 returns a digest in the Docker-Content-Digest and Etag HTTP headers that is equal to the one output by docker images --digests and docker image inspect imagename:tag --format '{{json .RepoDigests}}' See also https://stackoverflow.com/a/64309017/1543768

[kamkudla]

Is this actually still a thing? Hard to believe it.

[adwhit]

Seems especially pertinent when running an M1 Mac. I'm trying to verify that docker is pulling the AArch64 not AMD64 image. This would seem the obvious way to check but nope.

[arsulegai]

Bummer! I thought SHA256 could help me justify what's happening in my case. It's actually a thing.

[vimtor]

I share @adwhit pain

[b0r1sp]

Can sombody of the docker-team please fix that issue in a comprehensible way? Not being able to easily compare digests, to answer questions like "What version did I just download?" and "Which version of an image is located on my server?" is a missing thing. To use digests for that, is IMHO the way to reliably achieve that. Please advise if there's another intended way, thank you!

[ingshtrom]

I believe https://github.com/docker/hub-feedback/issues/2043#issuecomment-974235357 responds to the confusion here of which digest is where, as there are multiple which are involved in Docker Images.

In addition, these recommendations might help explain what we are doing going forward to help with this problem.

[andrewchen5678]

Came into this issue and want to move everything to github container registry instead because it provides the sha256 digest for all platforms on their website which is the same as the one from docker pull I can easily use to compare and pin my image to that multiplatform digest instead of having to pull the image manually first.

The "sha256:da52ff461520afd42284adfea5fb7c6c932d7900622a6e3612a3fab1ef3de7a4" here from github works for both amd64 and arm64 and it is the same as the one I get after docker pull, which docker hub won't provide on its web portal. Instead, it only provides one for amd64 only and another for arm64 only.

[thaJeztah]

If you're looking for this, also upvote https://github.com/docker/roadmap/issues/262 to help getting work on this prioritised internally

[zim32]

So there is now way currently to figure out which acual tag I've installed? I just want to backup my data and figure out which image I've installed so versions match exactly

[axel-h]

Looks like this could be helpful: https://github.com/ryandaniels/docker-script-find-latest-image-tag

[loynoir]

@zim32 https://github.com/safe-waters/docker-lock

Just a prevention.

With safe-waters/docker-lock, all image X:latest go X:latest@hash, seems very reproducible.

---

But no idea how to use hash get it's related versions, or said tags.

For example, given image X version 1.2.3 was hash A.

Does any one know how can I got 1.2.3 using hash A?

Thanks.

[max-allan]

Can sombody of the docker-team please fix that issue in a comprehensible way? Not being able to easily compare digests, to answer questions like "What version did I just download?" and "Which version of an image is located on my server?" is a missing thing. To use digests for that, is IMHO the way to reliably achieve that. Please advise if there's another intended way, thank you!

I think the answer is to use skopeo copy --preserve-digests instead of docker to get your containers. Seems weak that docker don't provide this option.

But if you've used docker pull already, there is no way to compare with docker hub.

[thaJeztah]

@max-allan it's a presentation issue in the Docker Hub UI; see https://github.com/docker/hub-feedback/issues/1925#issuecomment-1004890419

If you're looking for this, also upvote docker/roadmap#262 to help getting work on this prioritised internally

And https://github.com/docker/roadmap/issues/262#issuecomment-1311883372 (just copied my comment from the issue in moby/moby)

[jjm2473]

WTF of this shit! If they don't match, why you show them? Even it can not be used to verify tag.

[loynoir]

FYI, my workaround to automate get hash from tag

https://github.com/google/go-containerregistry/blob/main/cmd/crane/doc/crane_digest.md

[x-yuri]

It doesn't match because for multi-arch images docker pull shows the manifest list digest, not the manifest digest. To be more precise, docker pull shows the digest of what it gets when it requests a manifest. In case of multi-arch images it gets a manifest list.

To find out which image you're running you need to determine the manifest list digest that was stored locally (RepoDigest). This way:

$ docker images --digests | less
REPOSITORY  TAG     DIGEST                                                                    IMAGE ID       CREATED         SIZE
...
alpine      3.18.2  sha256:82d1e9d7ed48a7523bdebc18cf6290bdb97b82302a8a9c27d4fe885949ea94d1   c1aabb73d233   12 days ago     7.33MB
...

Or using docker inspect:

$ docker inspect c1aabb73d233 -f '{{json .RepoDigests}}'
["alpine@sha256:82d1e9d7ed48a7523bdebc18cf6290bdb97b82302a8a9c27d4fe885949ea94d1"]

Then get the list of tags:

$ docker run --rm regclient/regctl tag ls alpine | less
...
3.18.0
3.18.2
...
latest

Then query the manifest list digest on Docker Hub until you find the one that matches the RepoDigest (82d1e9d7ed48):

$ docker run --rm regclient/regctl image digest alpine:3.18.0
sha256:02bb6f428431fbc2809c5d1b41eab5a68350194fb508869a33cb1af4444c9b11

$ docker run --rm regclient/regctl image digest alpine:3.18.2
sha256:82d1e9d7ed48a7523bdebc18cf6290bdb97b82302a8a9c27d4fe885949ea94d1

But I guess images can be retagged on Docker Hub. As such, the image you're running... the tag may no longer point to your image.

More on it here.

[brenc]

This is so confusing. Here is what worked for me:

$ docker buildx imagetools inspect savonet/liquidsoap:rolling-release-2.2.x
Name:      docker.io/savonet/liquidsoap:rolling-release-2.2.x
MediaType: application/vnd.docker.distribution.manifest.list.v2+json
Digest:    sha256:defe450609e84a4ee3393c4dd0f365f19ff3e385153c9b2d14192348f3b1e9b7

Manifests: 
  Name:      docker.io/savonet/liquidsoap:rolling-release-2.2.x@sha256:d0a3d00f1c95d62eb2a57b81f94f5b657c82ec35f9668622af55cb49d6cbff99
  MediaType: application/vnd.docker.distribution.manifest.v2+json
  Platform:  linux/amd64

  Name:      docker.io/savonet/liquidsoap:rolling-release-2.2.x@sha256:ea729e24f9ed8fa0d9f5e33ce84ad8192c0322194f2960f57e6672a588d412b7
  MediaType: application/vnd.docker.distribution.manifest.v2+json
  Platform:  linux/arm64

The above shas for each arch match what's on Docker Hub.

https://stackoverflow.com/a/72340552

[MingTaLee]

This is so confusing. Here is what worked for me:

$ docker buildx imagetools inspect savonet/liquidsoap:rolling-release-2.2.x
Name:      docker.io/savonet/liquidsoap:rolling-release-2.2.x
MediaType: application/vnd.docker.distribution.manifest.list.v2+json
Digest:    sha256:defe450609e84a4ee3393c4dd0f365f19ff3e385153c9b2d14192348f3b1e9b7

The above shas for each arch match what's on Docker Hub.

https://stackoverflow.com/a/72340552

Of course it matches... because it pulls info from the registry (docker hub), not the local image you have.

You can run: docker buildx imagetools inspect --help

and will see: Usage: docker buildx imagetools inspect [OPTIONS] NAME

Show details of an image in the registry

And I agree with almost everyone here that it's frustrating not to able to check whether the local image matches the info on docker hub.

[albgen]

bump

[SK-StYleZ]

bump

[tyzbit]

Continually bumping will only result in this issue being locked to collaborators only, it will not increase visibility or prioritize this any higher than it already is.

(I'm not affiliated with this project)

[lukoboi]

Closing this as I've just added the digest on hub 🎉 The index digest will be displayed when present now.

[bickford]

The Index Digests would be much more useful if they were searchable on Docker Hub.

It's great progress that the Index Digests are now listed on Docker Hub when present, but the whole point of this was to be able to easily match local Docker SHA digests with Docker Hub digests to confirm they are correct. It works great if you are checking a known version - for example if you are fetching a 'latest' tag image and then checking it matches immediately. However, trying to locate older Index Digests is very hard on the Docker Hub as they are not searchable in any way I can locate. Someone correct me if I'm wrong.

Example: I'm running an older version of 'deluan/navidrome' from just 4 months ago, fetched with 'latest' tag initially, it contains no other tags on my local Docker installation (checked with 'docker inspect'). I know there's been at least a dozen updates to the image since I last updated my container as Diun has been sending me emails to notify updates, so I want to check if there were any major changes between my version and the current 'latest' version on Docker, or if I can simply install over the top to update.

I can get the local Docker digest via:

docker images --digests | grep navi
deluan/navidrome           latest          sha256:ea426987cc4a148eb6a9fcea60ab44cc0dca4d7f0f1e12a4b48f5937454da4d9   d8156521ce64   3 months ago    173MB

But there is no way for me to then search for this digest on the Docker Hub. So I instead click through each version and check the Index Digest in each version until I find it.. actually 22 versions back at 'deluan/navidrome:0.51.1'. That's only 4 months. I can imagine for a container someone has not updated for a year or more they would have to do a lot more digging. This should be a really simple task that takes a few seconds, but instead it feels like I'm rifling through a filing cabinet looking up dewey decimal symbols.

Please add the ability to search for Index Digests, it will save a lot of people a lot of time.

[thaJeztah]

@bickford that's quite orthogonal to this ticket, and not sure if that's easy to support. That said, it looks like there's some existing feature requests on the roadmap issue tracker;

• https://github.com/docker/roadmap/issues/185
• https://github.com/docker/roadmap/issues/663
• slightly related; https://github.com/docker/roadmap/issues/580