search

Azure / acr

Azure Container Registry samples, troubleshooting tips and references
https://aka.ms/acr
Other
162 stars 106 forks source link

Cached images do not update. #765

Open davidkarlsen opened 6 days ago

davidkarlsen commented 6 days ago

Describe the bug Images to not re-pull tags that are updated. Additionally - if one deletes a a whole registry path that was cached, one will get a not found when later pulling (i.e. ACR not re-pull from upstream)

To Reproduce Steps to reproduce the behavior:

  1. Have a rule like Screenshot 2024-06-29 at 13 17 45
  2. observe images not being updated

ghcr.io/davidkarlsen/utils-image main 3a641dbbef6f 22 hours ago 599MB MYREG.azurecr.io/ghcr.io/davidkarlsen/utils-image main 9c88223a0707 11 days ago 575MB

Expected behavior Should update tags. Should be able to re-pull images that were previously cached but have been deleted.

MYREG.azurecr.io/ghcr.io/davidkarlsen/flyway-operator-catalog:alpha not found: manifest unknown: manifest tagged by "alpha" is not found but docker pull ghcr.io/davidkarlsen/flyway-operator-catalog:alpha will work just fine

Screenshots If applicable, add screenshots to help explain your problem.

Any relevant environment information

Additional context Add any other context about the problem here.

If any information is a concern to post here, you can create a support ticket or send an email to acrsup@microsoft.com.

luisdlp commented 5 days ago

Thanks for reporting the bug. @davidkarlsen can you try to delete and recreate the cache rule? Are you still able to repro? We recently deployed in our cache backend that could have caused this issue. If this still repros, please open a support ticket or send us email with your registry name so that we can investigate further.

davidkarlsen commented 5 days ago

@luisdlp for that I have to drop the existing repos:

"details":[{"code":"TargetRepositoryPrefixMatchesExistingRegistries","message":"Conflict: There are one or more target repositories in the registry which match the target repository prefix 'ghcr.io/*'. Cache rules cannot be created for existing repositories in registry. For more information, please visit https://aka.ms/acr/cache"}]}]}]}}

I'll try that and see.

davidkarlsen commented 5 days ago

@luisdlp

this works:

  1. deleting the cache rule
  2. and all existing repos created by it
  3. and re-creating cache-rule

makes it work again.

But surely a fix is needed in ACR to avoid that?

luisdlp commented 4 days ago

@davidkarlsen We had a regression on a recent deployment which we deployed a fix for last week. Unfortunately, you might need to re-create other rules if you see the same behavior. We apologize for this inconvenience.

davidkarlsen commented 4 days ago

More stuff, seems like a UI-glitch:

If you pull:

docker pull --platform linux/amd64 MYREG/docker.io/library/ubuntu:noble

it will actually download, but it won't show when you list repositories in the azure portal. Others that are non-library ones, for instance docker.io/bitnami/sealed-secrets-controller will be listed...

luisdlp commented 3 days ago

@davidkarlsen are you using anonymous pulls from DockerHub? Starting on 6/30, DH started enforcing normal rate limiting rates for Azure customers. We highly recommend getting a free or paid subscription so that you have a higher rate limit. Caching does an initial import of the image into the ACR and then pulls it from there. If the import fail (likely due to throttling) the image will be pulled directly from DH. Since the import is not succeeding, you are not able to see the image cached in the registry. Can you try with authenticated pull?