Pushing images to Azure Container Registry extremely slow

[TurtleBeach]

Is this a BUG REPORT or FEATURE REQUEST?: Bug Report What happened?: Multiple attempts to push ~700MB image (nanoserver + powershell + java SE) take over 2 hours then fail with "unauthorized: authentication required" What did you expect to happen?: The image to be placed in container registry (image based only on nanoserver succeeded) How do you reproduce it (as minimally and precisely as possible)?: Build image locally with Docker, confirm it runs with minimal (echo) action. Logged in (az acr login). Also tried changing registry from Basic to Standard, which did not help. Anything else we need to know?: tried with Docker Push … from W10 command line and from Powershell (same command) Environment (if applicable to the issue):

• Azure CLI/Powershell/SDK version CLI 2.0.63
• Docker version 18.09.2
• OS version Windows 10 Pro 1809 17763.437
• Datetime (UTC) when the issue occurred ~21:00 UTC April 28
• Registry and image names base image from MS mcr.microsoft.com/powershell:6.2.0-nanoserver-1809_kb4480116_amd64 email sent to acrsup@microsoft.com If any information is a concern to post here, you can create a support ticket or send email to acrsup@microsoft.com.

[bramkl]

Also getting time out errors again today, ACR in West Europe, running docker-compose push from Azure Devops

@sajayantony can you reopen this thread?

[bramkl]

@sajayantony thanks for reopening.

It worked at the first attempt today. Does this mean something has been fixed? It does seem related to Azure DevOps? I'm a bit worried, because our production deployment depends on this.

Cheers

[sajayantony]

This is now mitigated, we are monitoring this and we have a plan to address West Europe first which is one of our most highly loaded region. We identified the reason we missed detecting this performance degradation and that is being addressed. Besides the targeted mitigation, I'll continue sharing details on the work that will be rolling out in Jan - https://github.com/Azure/acr/issues/322

[lpfeup]

I am also getting impacted by this and unable to deploy. docker-compose push timing out as https://github.com/Azure/acr/issues/214#issuecomment-516791610.

[jikuma]

@lpfeup Can you try again. We faced a latency issue in west Europe and its mitigated. I will post an RCA after an investigation.

[lpfeup]

@jikuma I confirm it worked now

[rantoniuk]

This is still happening for me (ACR in WestEurope, pushing from PL, 200MB image takes around 5 minutes - local network is definitely not an issue here, as well geo-replication shouldn't be needed in this case AFAIU.

[Kalmis]

@jikuma @sajayantony Still impacted by this yesterday and today.

[sajayantony]

The first phase of this has been rolled out and we are closely monitoring West Europe.

[rantoniuk]

Thanks for update @sajayantony. Just ran a test, pushing a 241MB layer from PL takes 4m.

[laurilehmijoki]

Thank you @sajayantony for working on this issue. However, we are still experiencing it: yesterday it took around 10 minutes for our CircleCI build to push a 80 MB Docker image to Azure westeurope ACR.

[rerime]

We are facing same problem. Constantly retrying while pushing image to ACR. Region West Europe.

[sajayantony]

The team is starting a second phase of this work and will be adding more capacity this week to West Europe. We are actively monitoring and rebalancing loads.

[chpspiir]

Seeing the same thing in North Europe this morning. Quick calculations show a consistent 500KB/s when pushing over the last hour. This is a fresh registry, so I am not sure if it is related to that. SKU is standard.

[mrvarmazyar]

Hey there,

We're facing the slowness issues in push section to ACR on Us West region. Any update about any sort of solution?

Thanks

[sajayantony]

Azure Container Registry now has increased capacity and also repartitioned compute to handle growing load in multiple regions. We are also monitoring latency actively. As a majority of this work has been deployed I would like to close this thread. For immediate help - please do open a support ticket from the Azure Portal. /cc @mnltejaswini

[yahya99223]

Funny thing that this thread has been closed 9 days ago, and I got this issue just now.

[vprudnikov]

The same issue for me. It usually takes 1-2 minutes to push hundreds of KB (not even MB) of code. And I see a lot of the following messages in my Job's logs:

76e5599e09ca: Preparing
61869b6bddfa: Preparing
5216338b40a7: Preparing
1f9b9bc05081: Layer already exists
76e5599e09ca: Layer already exists
61869b6bddfa: Waiting
5216338b40a7: Waiting
61869b6bddfa: Layer already exists
5216338b40a7: Layer already exists
2c226cc1f74d: Layer already exists
561808fa72a8: Layer already exists
0507a19651dc: Layer already exists
17130: digest: sha256:9e6cce00f231d5714d2c41abf924f9df12d3083d3df90109a4bfca4b866fbb89 size: 1789
7540222ad8dd: Preparing
209863904d0d: Preparing
650f6b4912cb: Preparing
1f9b9bc05081: Preparing
76e5599e09ca: Preparing
61869b6bddfa: Preparing
5216338b40a7: Preparing
1f9b9bc05081: Layer already exists
76e5599e09ca: Layer already exists
61869b6bddfa: Layer already exists
5216338b40a7: Layer already exists
209863904d0d: Layer already exists
7540222ad8dd: Layer already exists
650f6b4912cb: Layer already exists
17139: digest: sha256:fad65e7f4040246e3bd32e9799a183571907ecccaaa24eb0dc4b58315cc40fac size: 1789
3649d4cd4fdd: Preparing
f2589bac2f02: Preparing
4f63cbe372f7: Preparing
1f9b9bc05081: Preparing
76e5599e09ca: Preparing
61869b6bddfa: Preparing
5216338b40a7: Preparing
61869b6bddfa: Waiting
5216338b40a7: Waiting
1f9b9bc05081: Layer already exists
61869b6bddfa: Layer already exists
5216338b40a7: Layer already exists
76e5599e09ca: Layer already exists
4f63cbe372f7: Layer already exists
f2589bac2f02: Layer already exists
3649d4cd4fdd: Layer already exists
17143: digest: sha256:c67314424b549b9c1977d01ca833e7293479ea581cc2dcc1c454d25be426a35b size: 1789

[adamaiken89]

It seems the issue happens again. I have a 1GB image for CI and after 10 mins our CI didn't download the whole image.

[laurilehmijoki]

@sajayantony I'm experiencing slow download times from ACR to CircleCI.

Are there any known workarounds for the slow network connection between ACR and CirleCI?

[jageshmaharjan]

Even at the same region, the process of image push is slow. Not always, but at intermittent interval of time. Not sure, if this is bug or issue with my network connectivity.

[flieks]

for me it always takes a half hour to publish the update

[brechtvhb]

Pushing to ACR is painfully slow today (EU West).

[toddysm]

This issue is now locked.

If you're experiencing a performance problem with your Azure container registry, such as delays pushing images, see the following topics:

• Troubleshoot registry performance
• Check the health of an Azure container registry

If your issue isn't resolved, please open a support ticket.