az acr repository show-tags is not always returning all tags

C302 commented 9 months ago

Describe the bug

On firing a deployment pipeline, the pipeline will first check if the container-image of with that specific version actually is valid (if i exists in the registry).

Initially this worked fine. But when we fire up this deployment pipeline from the pipeline that actually is producing this container-version we have issues, this because the check says the tag is not present in the container registry.

First we taught that the ACR needed some time to process that newly created tag, we we added a dynamic delay between ending the publish-pipeline and deploy-pipeline. That dynamic delay si going to check with "az acr repository show-tags" if that tag is retrievable, if it has been found = fire up the deployment-pipeline.

But to our surprise, not all executions of "az acr repository show-tags" are giving back the same result, mainly the last added tag is sometimes present and sometimes not.

we're executing "az acr repository show-tags" with difference service principles, could there be an issue

Related command

az acr repository show-tags

Errors

no error message

Issue script & Debug output

no consistent return value when executing "az acr repository show-tags"

Expected behavior

return the correct tag-collection

Environment Summary

publish pipeline is using Azure CLI 2.55.0 deployment pipeline is using Azure CLI 2.56.0

Additional context

No response

yonzhan commented 9 months ago

Thank you for opening this issue, we will look into it.

Wwwsylvia commented 9 months ago

Hi @C302 , does the registry have geo-replication? Would you like to open a support case so that we can better troubleshoot the issue?

C302 commented 9 months ago

Hi @Wwwsylvia, a support ticket was opened (2402050050003898) and receive following info => https://learn.microsoft.com/en-us/azure/container-registry/container-registry-geo-replication#considerations-for-using-a-geo-replicated-registry

This means that this behavior is "by design", we did some tests and confirmed our suspicion.

It took around 10~15 seconds before an WEST-EU tagged image becomes available in the EAST-US region. because we're using a mix of self-hosted runners/ GH hosted runners/on-prem runners our results differ sometimes (especially because GH hosted runners are runnning ... unknown region)

Some points for improvement:

MS implements an ACR pull-through functionalty (if ACR receives an info request, it first checks if it's in sync with other ACR's)
AZ CLI providers a parameter to define to what region the request is send
GH provides a way to define in what region you want a GH self hosted runner to spin up

Azure / azure-cli