Problem fetching the latest image in ECR

[sp-ludovic-ivain]

Hi all, I’m actually using this resource type to get a image from ecr to use in a PUT step, but it is actually failing with this error in this PUT step:

waiting for docker to come up...
Pulling  xxxxxxxxxxxxx.dkr.ecr.us-east-1.amazonaws.com/my-image-name@sha256:xxxx01234567890987654322...
Error response from daemon: manifest for  xxxxxxxxxxxxx.dkr.ecr.us-east-1.amazonaws.com/my-image-name@sha256:xxxx01234567890987654322 not found

Pulling  xxxxxxxxxxxxx.dkr.ecr.us-east-1.amazonaws.com/my-image-name@sha256:xxxx01234567890987654322 (attempt 2 of 3)...
Error response from daemon: manifest for  xxxxxxxxxxxxx.dkr.ecr.us-east-1.amazonaws.com/my-image-name@sha256:xxxx01234567890987654322 not found

Pulling  xxxxxxxxxxxxx.dkr.ecr.us-east-1.amazonaws.com/my-image-name@sha256:xxxx01234567890987654322 (attempt 3 of 3)...
Error response from daemon: manifest for  xxxxxxxxxxxxx.dkr.ecr.us-east-1.amazonaws.com/my-image-name@sha256:xxxx01234567890987654322 not found

Failed to pull image  xxxxxxxxxxxxx.dkr.ecr.us-east-1.amazonaws.com/my-image-name@sha256:xxxx01234567890987654322.
fetching the image did not produce a volume

the resource type configuration :

- name: notifier
    type: docker-image
    source:
      repository: xxxxxxxxxxxxx.dkr.ecr.us-east-1.amazonaws.com/my-image-name

It seems that it’s trying to retrieve an image that doesn’t actually exist.

The sha256 didn’t correspond to the latest version of this image. (the custom resource is configured to get the latest by default, ad even if use the tag parameter, it doesn't change nothing).

I’ve tried to do a manuel check of this resource to try to refresh it, but it fails with the same.

Tryed to delete the workers thinking about a local cache into them, but unsuccessful.

Any other kind of cache in the Postgresql database ? Any idea ?

Thanks a lot

[ari-becker]

We ran into the same issue, so it's not just you.

We worked around it by moving to registry-image-resource, now that it has ECR support.

[xtremerui]

What is the version looks like of your resource? If the sha comes from a resource check then it makes sence to pull using that sha.

[ari-becker]

@xtremerui The sha does come from a resource check, but we use tag in the source for readability / maintainability. Anyway, a mismatched hash isn't the underlying issue here. We ran into this issue twice - once while updating the tag of a resource, and once while moving a pipeline as-is from a Concourse 4.2.2 instance (where it was working just fine) to a Concourse 5.8.1 instance (where we ran into the issue).

[analytically]

When upgrading from 6.2.0 to 6.3.0 neither docker-image nor registry-image works. Not a good situation.

[analytically]

@jamieklassen

[jamieklassen]

@analytically can you help to find a reproducible? In this example and in concourse/concourse#5817 ECR was being used, but in concourse/concourse#5818 it was Docker Hub. Resource versions certainly are stored in the database, but I'm stuck thinking about where these nonexistent SHAs are coming from. Just staring at the code in this repo doesn't yield much so I'd love to be able to find a real-world example.