registry-image pulls image with wrong SHA digest

[mreiche]

Describe the bug

I have some registry-image resources, pulling from private Artifactory.

build-meta-data

  - name: build-meta-data
    type: registry-image
    source:
      repository: ((docker_hub_mirror))/swce/metadata-resource
      tag: release-v0.0.3

bitbucket-pull-request

  - name: bitbucket-pull-request
    type: registry-image
    source:
      repository: ((docker_hub_mirror))/platform/concourse-bitbucket-pullrequest-resource
      tag: sha256:b98a6d635d50d7e0fc64ba4184a23c8493db4b4e995af45da6940c116ab0b121

Both are failing, because of the following reasons:

selected worker: concourse-worker-0
fetching artifactory.local/docker/swce/metadata-resource@sha256:1f8c0fd22ad21032f9464a86c51d97c0228bdd8be44fd10616ea16e331220a4b
ERRO[0000] download failed: get image: GET artifactory.local/v2/docker/swce/metadata-resource/manifests/sha256:1f8c0fd22ad21032f9464a86c51d97c0228bdd8be44fd10616ea16e331220a4b: MANIFEST_UNKNOWN: The named manifest is not known to the registry.; map[manifest:swce/metadata-resource] 

 selected worker: concourse-worker-0
fetching artifactory.local/docker/platform/concourse-bitbucket-pullrequest-resource@sha256:1f8c0fd22ad21032f9464a86c51d97c0228bdd8be44fd10616ea16e331220a4b
ERRO[0000] download failed: get image: GET https://artifactory.local/v2/docker/platform/concourse-bitbucket-pullrequest-resource/manifests/sha256:1f8c0fd22ad21032f9464a86c51d97c0228bdd8be44fd10616ea16e331220a4b: MANIFEST_UNKNOWN: The named manifest is not known to the registry.; map[manifest:sha256:1f8c0fd22ad21032f9464a86c51d97c0228bdd8be44fd10616ea16e331220a4b] 

Both resources pulling 1f8c0fd22ad21032f9464a86c51d97c0228bdd8be44fd10616ea16e331220a4b, no matter which tag or SHA was given.

I took a look into Artifactory's access log:

[ACCEPTED DOWNLOAD] docker-hub-cache:swce/metadata-resource/release-v0.0.3/manifest.json  for client : xxx / 

I checked the manifest.json for the SHA, because I was thinking that Artifactory has a bug resolving tags. But then I found more logs like that in the request log, that shows that Artifactory get requested for the same resource hash for several other images:

a.b.c.d|xxx|HEAD|/api/docker/docker/v2/platform/docker-buildx-resource/manifests/sha256:1f8c0fd22ad21032f9464a86c51d97c0228bdd8be44fd10616ea16e331220a4b|404|-1|0|2|go-containerregistry/v0.12.1
a.b.c.d|xxx|HEAD|/api/docker/docker/v2/troykinsella/concourse-artifactory-resource/manifests/sha256:1f8c0fd22ad21032f9464a86c51d97c0228bdd8be44fd10616ea16e331220a4b|404|-1|0|256|go-containerregistry/v0.12.1
a.b.c.d|xxx|HEAD|/api/docker/docker/v2/swce/metadata-resource/manifests/sha256:1f8c0fd22ad21032f9464a86c51d97c0228bdd8be44fd10616ea16e331220a4b|404|-1|0|460|go-containerregistry/v0.12.1
a.b.c.d|xxx|HEAD|/api/docker/docker/v2/platform/concourse-bitbucket-pullrequest-resource/manifests/sha256:1f8c0fd22ad21032f9464a86c51d97c0228bdd8be44fd10616ea16e331220a4b|404|-1|0|2|go-containerregistry/v0.12.1

Where does 1f8c0fd22ad21032f9464a86c51d97c0228bdd8be44fd10616ea16e331220a4b comes from? I do not find any image with this SHA in the Artifactory nor do i find it in the resource_caches table of Concourse:

select *  from resource_caches where metadata ilike '%1f8c0fd22ad21032f9464a86c51d97c0228bdd8be44fd10616ea16e331220a4b%';

Is there a bug somewhere between image-registry and or go-containerregistry?

I also tried to get rid of caches by deleting Worker PVCs and re-creating the Pipeline completely.

Reproduction steps

I dont know if there are steps to reproduce.

Expected behavior

image-registry should pull the right image.

Additional context

No response

[mreiche]

We tried to debug registry-image by adding

debug: true

since then, the bug is gone for 24h now.

[mreiche]

We switch back to docker-image.

[kalensk]

@mreiche I'm late to the party, but I happened across your issue. I am wondering if its related to this current issue https://github.com/concourse/registry-image-resource/issues/356 I just filed where using images built with newer versions of docker fail to push using registry-image concourse resource. The details looks fairly similar.

It's probably too late for you to dig into the details or reproduce, but thought I'd mention it.