az acr login with private endpoint goes via public route

[r3-jerrysteele]

Describe the bug

Command Name az acr login

Errors:

Error response from daemon: Get https://myregistryname.azurecr.io/v2/: denied: client with IP '<my public IP>' is not allowed access. Refer https://aka.ms/acr/firewall to grant access
Login failed.

To Reproduce:

Steps to reproduce the behavior. Note that argument values have been redacted, as they may contain sensitive information.

• Connect to Azure VPN
• az acr login -n myregistryname --subscription mysubscription

Expected Behavior

Login to the ACR's private endpoint via the connected VPN

Environment Summary

Linux-4.4.0-19041-Microsoft-x86_64-with-debian-bullseye-sid
Python 3.6.10
Installer: DEB

azure-cli 2.19.1

Using WSL1

Additional Context

I am logged into a VPN (Azure Virtual Network Gateway), I have made the necessary adjustments to the /etc/resolv.conf so that the ACR resolves to the Private Link IP:

Server:         10.x.x.x.
Address:        10x.x.x#53

Non-authoritative answer:
myregistryname.azurecr.io canonical name = myregistryname.privatelink.azurecr.io.
Name:   myregistryname.privatelink.azurecr.io
Address: 10.x.x.x

The ACR is configured not to allow public access, but has a private endpoint configured which is known to work

I can curl the URL of the ACR and get a 200 response (as it goes via the private address), it seems that the issue is with azure-cli itself.

ip route 10.x.x.x also shows that the route goes via the VPN interface

[ghost]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @toddysm, @yugangw-MSFT.

Issue Details

---

## Describe the bug **Command Name** `az acr login` **Errors:** ``` Error response from daemon: Get https://myregistryname.azurecr.io/v2/: denied: client with IP '' is not allowed access. Refer https://aka.ms/acr/firewall to grant access Login failed. ``` ## To Reproduce: Steps to reproduce the behavior. Note that argument values have been redacted, as they may contain sensitive information. - Connect to Azure VPN - `az acr login -n myregistryname --subscription mysubscription` ## Expected Behavior Login to the ACR's private endpoint via the connected VPN ## Environment Summary ``` Linux-4.4.0-19041-Microsoft-x86_64-with-debian-bullseye-sid Python 3.6.10 Installer: DEB azure-cli 2.19.1 Using WSL1 ``` ## Additional Context I am logged into a VPN (Azure Virtual Network Gateway), I have made the necessary adjustments to the `/etc/resolv.conf` so that the ACR resolves to the Private Link IP: ``` Server: 10.x.x.x. Address: 10x.x.x#53 Non-authoritative answer: myregistryname.azurecr.io canonical name = myregistryname.privatelink.azurecr.io. Name: myregistryname.privatelink.azurecr.io Address: 10.x.x.x ``` The ACR is configured not to allow public access, but has a private endpoint configured which is known to work I can `curl` the URL of the ACR and get a 200 response (as it goes via the private address), it seems that the issue is with `azure-cli` itself. `ip route 10.x.x.x` also shows that the route goes via the VPN interface

Author:	r3-jerrysteele
Assignees:	-
Labels:	`Container Registry`, `Service Attention`, `needs-triage`, `question`
Milestone:	-

[yungezz]

route to appropriate team

[yugangw-msft]

@r3-jerrysteele, sorry for missing this. The best way to diagnose this is to verify the DNS setting. If not resolving to the private ip, then the error is expected. Doc is here:

https://docs.microsoft.com/en-us/azure/container-registry/container-registry-private-link#validate-private-link-connection.

Please let us know what you found out

[fieldp]

Hi. I have the exact same issue. Not sure if it's a CLI issue or user error, so I posted it on the Microsoft Q&A forum. https://docs.microsoft.com/en-us/answers/questions/632408/trying-to-access-an-azure-container-registry-with.html

[LahariChidura]

I am facing the same issue. The ACR DNS resolves to private IP address in the vnet -

however, az acr login throws this error -

[fieldp]

Good to know not just me then. I gave up. I had a back and forth with MS support but just could not solve it. If anyone has a new ideas I would be happy try :-)

On May 16, 2022, fieldp @.***> wrote:

I am facing the same issue.  The ACR DNS resolves to private IP address in the vnet -  

however, az acr login throws this error -  

  — Reply to this email directly, view it on GitHub https://github.com/Azure/azure-cli/issues/17137#issuecomment- 1127842531, or unsubscribe https://github.com/notifications/unsubscribe- auth/AVIA2WUDOTQPC4MBITOADXTVKJVGHANCNFSM4YIZXCPQ. You are receiving this because you commented.Message ID: <Azure/azure- @.***>

[mdheerendra]

I have this same issue, I've created a Azure Resourse Manager Service Connection and While using Azure CLI command az acr build --image reponame:imagename --registry acrname --file Dockerfile . with self hosted agent, I'm getting the same error.

[ttasharski73]

I found that using docker login / docker build correctly uses the private endpoint.

I created a token on the ACR with a PW

Using this to login as example echo $ACR_PW| docker login -u $ACR_USER --password-stdin registryUrl

This to build as example docker build -t registryUrl/imagename --file DockerFile .

[lordisp]

I'm running private hosted devops agents. Login works but building the image using az acr build get still somehow routed over the public route:

az acr login --name myreg.azurecr.io
az acr build --registry myreg.azurecr.io --image xxx:yyy .

========================== Starting Command Output ===========================
/usr/bin/bash /agent/_work/_temp/771f368c-362a-4c11-8df9-b49ece019f58.sh
WARNING: The login server endpoint suffix '.azurecr.io' is automatically omitted.
Login Succeeded
WARNING: The login server endpoint suffix '.azurecr.io' is automatically omitted.
WARNING: Packing source code into tar to upload...
WARNING: Excluding '.git' based on default ignore rules
WARNING: Excluding '.gitignore' based on default ignore rules
WARNING: Uploading archived source code from '/tmp/build_archive_c4dc880b67914c5d9e0fdabd3ca87293.tar.gz'...
WARNING: Sending context (854.722 KiB) to registry: myreg...
WARNING: Queued a build with ID: cb4
WARNING: Waiting for an agent...
2022/07/06 22:12:39 Downloading source code...
2022/07/06 22:12:40 Finished downloading source code
2022/07/06 22:12:41 Using acb_vol_055ad1ce-aff6-4195-bf76-f619d26a7abf as the home volume
2022/07/06 22:12:41 Setting up Docker configuration...
2022/07/06 22:12:41 Successfully set up Docker configuration
2022/07/06 22:12:41 Logging in to registry: myreg.azurecr.io

failed to login, ran out of retries: failed to set docker credentials: Error response from daemon: Get "[https://acrlhggixiservicesp.azurecr.io/v2/"](https://myreg.azurecr.io/v2/%22): denied: client with IP '20.50.200.13' is not allowed access. Refer https://aka.ms/acr/firewall to grant access.
: exit status 1

The only way to get this working is building the image using docker build, tag it to the acr and push it:

az login --identity
az acr login --name myreg.azurecr.io
docker build -t foo:v1.0 .
docker tag foo:v1.0 myreg.azurecr.io/foo:v1.0
docker push myreg.azurecr.io/foo:v1.0

I believe this is a bug in az acr build cli

[vf-doctor]

Just want to add that I am also experiencing the same issue. Doing a nslookup or dig on the acr endpoint resolves to the private IP address but az acr build routes via the public endpoint.

And because I am running the agent in an azure container instance I have had problems getting docker to run inside the container. So the workaround noted above is not as easy for me to deploy.

[thedheerendra]

I had found with a Microsoft Case that you need to whitelist those 2,3 public IPs on ACR(with private endpoints) which will appear with that error. Those IPs are somehow useful for build.

[vf-doctor]

@thedheerendra Unfortunately, when you turn off public access entirely you no longer are able to whitelist any public IPs. The az acr build generates ACR tasks which use dedicated compute nodes to perform the container build. These virtual nodes exist outside the virtual network the build agent is running in. That seems to be the crux of the issue.

[eiji102030]

I have the same issue. Are there any updates?

[sbussetti]

~confirmed this issue on macos and ubuntu operating systems today~ az acr login seems to work for me via private endpoint. It's az acr build that I have this issue with. It seems they have worked around this issue this way: https://github.com/Azure/acr/issues/603

[Wompipomp]

We have the same issue. az acr build gets routed via public route :(

[sbussetti]

@Wompipomp see https://github.com/Azure/acr/issues/603

[flavian-anselmo]

Any updates on this ?

[toddysm]

Changing the assignment to @northtyphoon and @terencet-dev for next steps.

[rajeshkaremane]

have same issue while running az cli on build agent and acr over private link

Work around az acr login --name registryname cd $(Build.SourcesDirectory) docker buildx build -t registryname.azurecr.io/reponame:$(Build.BuildId) --platform=linux/arm64 . docker push registryname.azurecr.io/reponame:$(Build.BuildId) if ($LastExitCode -ne 0) { Write-Host “##vso[task.complete result=Failed;]DONE” exit 1 }

[AurimasNav]

In my case it is az acr run --cmd "acr purge.." that is failing to work via private endpoint, probably the same root cause as for build.

[Yeseh]

have same issue while running az cli on build agent and acr over private link

Work around az acr login --name registryname cd $(Build.SourcesDirectory) docker buildx build -t registryname.azurecr.io/reponame:$(Build.BuildId) --platform=linux/arm64 . docker push registryname.azurecr.io/reponame:$(Build.BuildId) if ($LastExitCode -ne 0) { Write-Host “##vso[task.complete result=Failed;]DONE” exit 1 }

This problem seems to still be present. This workaround did the trick for us.

[fortunkam]

Did this ever get resolved? I am seeing the same exact problem as the original post. nslookup resolves the correct vnet address of the ACR, az acr login tries to work via the public address (and recommends adding my IP to the acr firewall)

[ldvy]

I have the same problem. The xxx.azurecr.io correctly resolves to xxx.privatelink.azurecr.io and goes through the internal endpoint to the ACR. However, upon running az acr login it seems to go over public internet. Ridiculous that this hasn't been fixed for this long.