Transient EOF issues pulling images

[phlax]

Problem description

We pull images fairly frequently from dockerhub as part of our CI (https://github.com/envoyproxy/envoy)

We have been seeing EOF errors pulling the images with increasing frequency

An example failure is here - but this seems to be happening ~daily on different images in different situations

Step 1/3 : FROM node:alpine@sha256:b3ca07adf425d043e180464aac97cb4f7a566651f77f4ecb87b10c10788644bb
docker.io/library/node:alpine@sha256:b3ca07adf425d043e180464aac97cb4f7a566651f77f4ecb87b10c10788644bb: Pulling from library/node
Get "[https://registry-1.docker.io/v2/library/node/manifests/sha256:59e776513bcb08c151d8ed4162b83bbecf1d5c629b61e3c2b4c1cae33b0fbdc3"](https://registry-1.docker.io/v2/library/node/manifests/sha256:59e776513bcb08c151d8ed4162b83bbecf1d5c629b61e3c2b4c1cae33b0fbdc3%22): EOF

the issues are transient and dont seem to happen more than once for any given incident

Debug Information

Browser name and version:

URL:

Some recent examples

• https://dev.azure.com/cncf/envoy/_build/results?buildId=113806&view=logs&j=f1a49366-51c0-540f-0294-f73c2bb311a9&t=7fd036c9-61ec-5b59-3645-e90950dc67a6&l=2761
• https://dev.azure.com/cncf/envoy/_build/results?buildId=113688&view=logs&j=7b6d60aa-b5de-5f9b-678c-063d31881e76&t=0c8bcd19-14ac-5aba-0af8-facf9399c298&l=14

Timetamp or time range:

Hub Username:

envoyproxy

Error messages (on screen or in browser console)

For example:

Get "[https://registry-1.docker.io/v2/library/node/manifests/sha256:59e776513bcb08c151d8ed4162b83bbecf1d5c629b61e3c2b4c1cae33b0fbdc3"](https://registry-1.docker.io/v2/library/node/manifests/sha256:59e776513bcb08c151d8ed4162b83bbecf1d5c629b61e3c2b4c1cae33b0fbdc3%22): EOF

Error response from daemon: Get https://registry-1.docker.io/v2/: EOF

Task List

• [X] This is NOT a security issue
• [ ] I do NOT have a Docker subscription (We are "Sponsored OSS" - not sure if that counts as a subscription)
• [X] I have looked through other issues and they do NOT apply to me (or have been closed)

[milosgajdos]

Hi @phlax we have identified an issue in our ingress LB setup which in combination with autoscaling causes these errors. We have now taken the first steps in addressing these issues, so they should be alleviated. We will work on the proper fix in the upcoming days. Thanks for reporting. I shall leave this issue open for the time being.

[phlax]

@milosgajdos thanks for picking this up - unfortunately we are still seeing a lot of these errors (not sure if more, but doesnt seem like less)

[milosgajdos]

I appreciate that, we'll be making more changes going forward. It seems this is a more complex problem than what we've originally estimated 😞

[phlax]

this issue appears to have gotten a lot worse - it seems to have gone from a trickle to a flood

on our side we exploring various ways we can limit how much we pull from dockerhub - there is quite a bit of work done to those ends, but none of the approaches are trivial, and im not sure it would resolve - more just mitigate

[milosgajdos]

There is an active incident @phlax https://status.docker.com/

[mathstuf]

This seems to be happening again; we're seeing many instances of this starting yesterday morning (US East) and continuing into today. There doesn't seem to be anything on the status dashboard though.

[milosgajdos]

@mathstuf apologies for the interruptions caused yesterday -- we've been performing internal infrastructure updates and not everything has gone as expected. The upgrade has now been completed and things should be back to normal.

[ericlee42]

Not working for aws us-east-1 and us-east-2

Please fix it.

[ingshtrom]

@ericlee42 We reported an incident over the weekend. I believe you likely ran into that: https://www.dockerstatus.com/pages/incident/533c6539221ae15e3f000031/654faa887895d304cc474aed

[nicks]

Thanks for reporting! We believe the original issue has been resolved. The incident over the weekend has been resolved as well. If you have future Hub connectivity issues, please file a new issue.