[ECR] [bug]: pull through cache rule related cache repository's immutability not working as expected

[peteher]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Tell us about your request During ECR's pull through cache rule->creation template->immutable cache repository creation and image pull testing I found that immutability configured on the cache repo (set by creation template) does not work as expected.

Which service(s) is this request for? ECR

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard? Documentation states:

"Turning on image tag immutability for repositories using a pull through cache rule will prevent Amazon ECR from updating images using the same tag."

I'm using pull through cache rule and creation template and upon my first pull of an image through the cache, the related cache private repository is added as the creation template specifies: immutable - this is fine.

I upload an image (image A) with a tag (tag X) to a public repo (AWS ECR or Docker hub) and pull it. Image A tag X gets into the ECR private cache just fine.

Now swap the image for the tag in the upstream repository: I upload another image (image B) with same tag (tag X) to the public repo. Since the public repo is mutable, this works fine.

I expect subsequent pulls via the cache repo (which is immutable) would always provide me the original image A.

This works till the ~24 hrs cache expiration

"Amazon ECR checks the upstream registry to see if there is a new version of the image and will attempt to update your private registry at least once every 24 hours"

Once the 24 hrs expired, subsequent pulls via the cache repository providing the newer image B for tag X - so immutability does not work.

Interestingly AWS console UI and AWS CLI shows otherwise: with immutability enabled on the cache repo, the original image A shows still for the tag X. But actual pull on the image via the cache by a client shows the client is provided with the newer image B for tag X.

In other words: UI/awscli shows as it was indended to work, but actual image pull by a client show differently.

Note: you have to be patient and wait for ~24 hrs with your client's pull to confirm the issue, since cache repo's force update is just a feature request at this point: https://github.com/aws/containers-roadmap/issues/2254

Are you currently working around this issue? No suitable workaround

[kattmang]

AWS ECR engineer here: yes, this is certainly a bug that's a side effect of how we serve images directly from upstream under certain conditions (like this one). We should not return to you an image when the 24 cache window expires from the upstream that we would never sync to an ECR repository. We'll work on this and post an update when we have one.

[peteher]

Hi @kattmang

I'm just checking in to learn if you have this item showing up in your dev pipeline as an issue to fix yet - also if you have some estimation about a possible completion/release date for a fix?

Thanks in advance for all the efforts!