Pulling images impossible: Unexpected EOF

[gingters]

• [X] I have tried with the latest version of my channel (Stable or Edge)
• [X] I have uploaded Diagnostics
• Diagnostics ID: FDE7E224-A933-4E0F-94F5-5BA825FF2155/20200401125255

Expected behavior

docker run hello-world should run.

Actual behavior

Docker fails at pulling the image with an unexpected EOF error:

PS C:\Dev> docker run hello-world
Unable to find image 'hello-world:latest' locally
C:\Program Files\Docker\Docker\resources\bin\docker.exe: Error response from daemon: Get https://registry-1.docker.io/v2
/library/hello-world/manifests/latest: Get https://auth.docker.io/token?scope=repository%3Alibrary%2Fhello-world%3Apull&
service=registry.docker.io: unexpected EOF.

This does also happen when downloading from other registries:

PS C:\Dev> docker pull mcr.microsoft.com/mssql/server:2019-latest
Error response from daemon: Get https://mcr.microsoft.com/v2/mssql/server/manifests/2019-latest: unexpected EOF

Information

• Is it reproducible? Yes.
• Is the problem new? Yes.
• Did the problem appear with an update? Probably. I could pull images bevor. But I can't tell when this was introduced.
• Windows Version: 1909 Build 18363.752
• Docker Desktop Version: 2.2.0.4 (43472)
• Are you running inside a virtualized Windows e.g. on a cloud server or on a mac VM: No

Additional Info:

I already tried:

• uninstalling Docker Desktop & reinstalling
  • Both Edge as well as Stable
• Reset to factory defaults
• Login with my dockerhub credentials
• logout again

There are different unexpected EOF issues on the community and here, but no answers:

• https://forums.docker.com/t/unexpected-eof-on-auth-docker-io-when-trying-to-pull-from-docker-hub/90652
• https://github.com/docker/for-win/issues/5868
• https://github.com/moby/moby/issues/40578

This is a complete showstopper, as reinstalling and factory reset of course cleared all images, and now I can't run anything on docker anymore.

[gingters]

Update:

I went back and installed several older Docker versions:

• This also happens with 2.2.0.3.
• Sadly 2.2.0.0 isn't available as a download from the Release Notes page.
• The Edge build 2.1.7.0 (the 2.2 RC ?) also has this issue.
• The Edge build 2.1.5.0 (new Filesharing Impl) also has this issue.
• 2.1.0.2 also.

I didn't go back further than that. I am pretty sure that it still worked with one of these previous versions, so my guess would be that the latest Windows update 18363.752 from March the 24th changed something that docker can't cope with.

[gingters]

Update: I now also uninstalled KB4541335 and KB4551762 as these were the latest two somewhat bigger Windows Updates to check if that changed something, but it didn't.

[gingters]

Update: I did a complete fresh install of Windows 10 (1909 18363.720) and a fresh install of Docker (2.2.0.5 (43884).

I also did a new diagnostic from the new installation: 4FCFC13E-A452-4269-8906-1C0E7B94F04F/20200402173141

Seriously: I need support here. This is a complete blocker. A fresh install of both Windows and Docker, and it can't download any image at all. This is absolutely unusable in this state.

I am a developer and I am willing to support you in diagnosing this issue as much as possible, so please get in contact.

[gingters]

Update: Someone mentioned to me that it could be an issue with some network chips freaking out on a gigabit connection. Tried another throttled network too: Same issue.

[SR-Lut3t1um]

Same issue for me!

[McGregsen]

We are also seeing this issue on multiple windows clients in our network. There is no proxy or anything like that in use.

I think the issue started appearing about 1,5 weeks ago. Not sure about the exact timeframe but I am fairly certain it was still working 2 weeks ago. And as far as I could tell there was no update to docker4win when the issue started appearing (There was a recent update a few days ago but that didn't change the behaviour). So I am actually leaning towards a windows update issue maybe? Although we also already tried removing various recent windows updates without any success.

[twsl]

I am facing this issue as well. checked both latest stable and edge channel. Tried several reinstalls. I'm able to reproduce this with Linux containers and docker run hello-world

This seems to be the same issue as reported here, as it works with Windows containers for me.

• Windows Version: 1909 Build 18363.815
• Docker Desktop Version: 2.3.0.0 (44472)

PS C:\Users\User> docker --version
Docker version 19.03.8, build afacb8b

PS C:\Users\User> docker info
Client:
 Debug Mode: false
 Plugins:
  app: Docker Application (Docker Inc., v0.8.0)
  buildx: Build with BuildKit (Docker Inc., v0.3.1-tp-docker)
  mutagen: Synchronize files with Docker Desktop (Docker Inc., testing)

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 0
 Server Version: 19.03.8
 Storage Driver: overlay2
  Backing Filesystem: <unknown>
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 7ad184331fa3e55e52b890ea95e65ba581ae3429
 runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
 init version: fec3683
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 4.19.76-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 6
 Total Memory: 9.736GiB
 Name: docker-desktop
 ID: 7KGG:PIJ2:5JBI:5ZCR:KIML:AP7A:P64S:E65L:6DEY:DV2K:2VUB:V5H5
 Docker Root Dir: /var/lib/docker
 Debug Mode: true
  File Descriptors: 39
  Goroutines: 50
  System Time: 2020-04-23T11:17:46.490945716Z
  EventsListeners: 3
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: true
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
 Product License: Community Engine

[TSK-Dev]

Same issue for me. Restarting machine, Reinstall Docker für windows, set to docker desktop "Switch to windows containers" an back to linux containers. Nothing works for me.

[twsl]

Still the same behaviour with

• Docker Desktop Version: 2.3.0.1 (44875)

[medokin]

Having this issue for some weeks now, nothing helped, completely reinstalled Windows, issue is still there:

Diag: C5936D36-F1F8-4856-BECC-0FBD5B1A1B13/20200515174356 Docker Desktop Version: 2.3.0.2 (45183) Windows 10 (2004)

Pulling from official docker and microsoft does not work. Pulling from google is very instable, but works sometimes.

PS C:\Users\Niko> docker pull postgres
Using default tag: latest
Error response from daemon: Get https://registry-1.docker.io/v2/library/postgres/manifests/latest: Get https://auth.docker.io/token?account=medokin&scope=repository%3Alibrary%2Fpostgres%3Apull&service=registry.docker.io: unexpected EOF

PS C:\Users\Niko> docker pull mcr.microsoft.com/mssql/server:2019-latest
Error response from daemon: Get https://mcr.microsoft.com/v2/mssql/server/manifests/2019-latest: unexpected EOF

PS C:\Users\Niko> docker pull gcr.io/google_containers/cadvisor
Using default tag: latest
latest: Pulling from google_containers/cadvisor
9d48c3bd43c5: Pull complete
f7d6cbe0ad90: Pull complete
15f5311b080f: Pull complete
Digest: sha256:46d4d730ef886aaece9e0a65a912564cab0303cf88718d82b3df84d3add6885c
Status: Downloaded newer image for gcr.io/google_containers/cadvisor:latest
gcr.io/google_containers/cadvisor:latest

[gingters]

@medokin @twsl @TSK-Dev From your profiles I see you are from Germany, too. Could be important ;) @McGregsen I can't tell that from your profile, but I take a wild guess:

I figured out the problem lies with my router. I have an AVM Fritz!Box Cable modem (and a Vodafone / ex Kabel BW line). If you too have an AVM router, read further:

With the latest Fritz!OS firmware it seems there is an issue with the Hardware Packet Acceleration. One workaround for me is to disable that setting temporarily, the other one was using a VPN service.

See also this post for detailed instructions on how to disable hardware packet acceleration: https://github.com/docker/for-mac/issues/3493#issuecomment-616338458

[twsl]

@gingters I already found the thread. Thanks for sharing the solution tho. It's still weird, that it works with windows containers and not linux.

[medokin]

@gingters You are right, I have a brand new 6591.... It works now with the deactivation of packet acceleration.

[rGunti]

Can also confirm. Fritz!Box 6660 Cable (unbranded) with Vodafone 1Gbps plan. Disabling hardware acceleration works but slows down your internet speed to max 150 Mbps. I can get around the issue by using a VPN connection but that's not a solution.

[McGregsen]

@gingters Thanks for the update. You are correct I am from Germany as well. With your information we were able to implement a fix/workaround that works for us until an updated firmware for the box can be provided.

[TSK-Dev]

Thanks to you, @gingters. Same for me: Germany, Provider Vodafon. The problem started when I booked the new tarif "Vodafon Cable Max 1000" and get a new Fritz Box 6490. If I deactivate "Hardwarebeschleunigung", docker runs stable. I guess, some problems with Playstation Internet connection are also caused by this problem. What surprises me very much is that this problem does not always occur, when hardware acceleration is disabled. It occours mainly in the evenings and on weekends. Is there any filter causing problems that is only set at certain times when there is a lot of traffic?

[vietd24]

I can confirm with a Fritzbox Cable 6660 Unitymedia Region (now Vodafone) that pulling docker images are working as expected when disabling hardware acceleration in the support menu.

I have just contacted AVM, waiting for an answer.

[vietd24]

FYI: Using the new 7.19 BETA firmware on my Fritzbox 6660 seems to have fixed the problem.

[lpfann]

FYI: Using the new 7.19 BETA firmware on my Fritzbox 6660 seems to have fixed the problem.

Not for me. Still unexpected EOF when pulling images with packet acceleration.

[cybertschunk]

Same problem here, docker is currently unusable

[paulkre]

Same problem here! I work for an organization with a lot of employees. I can't just make changes to our router's configuration that will affect its network speed.

[lauff]

Same problem for me with a FRITZ!Box 6591 Cable FRITZ!OS 07.13

[mbaniasad]

so just to confirm. same problem. working in Germany and have a Gigabit connection. my colleague tried a throttled connection. didn't work. (update)

it was the router. fritzbox and vodafone connection problem. images can be downloaded if the hardware acceleration is disabled

[Cartman0]

there maybe same issue in usage of gcc image.

env:

• windows10 home 2004
• enabled wsl2
• docker desktop for windows: 2.3.0.3(45519)
• docker engine: 19.03.8, build afacb8b

Dockerfile:

FROM gcc:latest as env-gcc
....

cmd:

> docker build .
Sending build context to Docker daemon  2.048kB
Step 1/6 : FROM gcc:latest as env-gcc
latest: Pulling from library/gcc
e9afc4f90ab0: Pull complete
989e6b19a265: Pull complete
af14b6c2f878: Pull complete
5573c4b30949: Pull complete
11a88e764313: Downloading [==================================================>]   46.9MB/46.9MB
e2983d3e9d87: Download complete
b73e08d8a550: Download complete
ef564e387869: Download complete
2645c37e0992: Download complete
unexpected EOF

but docker build can correctly work by Dockerfile in https://github.com/docker/getting-started. What's the difference between these?

[ciB89]

I have got the exact same issue, also win 10, also Germany, also FritzBox router (6591 cable). At first after installing docker for windows everything worked fine and I was able to pull, build and push (altohugh I had to make sure to use google dns). Then it suddenly stopped. Using PIA now as a workaround and it works..

[lpfann]

I just installed the newest FritzOS version released today (07.19-80490 BETA) and hello-world can now pull without problem.

❯ docker pull hello-world
Using default tag: latest
latest: Pulling from library/hello-world
0e03bdcc26d7: Pull complete
Digest: sha256:49a1c8800c94df04e9658809b006fd8a686cab8028d33cfba2cc049724254202
Status: Downloaded newer image for hello-world:latest
docker.io/library/hello-world:latest

I dont know if anything else changed because I did not test it before updating.

[Spenhouet]

We have the same issue. We can't update the firmware of our Fritz!Box 6591 Cable since it is provided by the provider. There also must be something wrong on the docker side of things. No other application has these issues.

[gingters]

We have the same issue. We can't update the firmware of our Fritz!Box 6591 Cable since it is provided by the provider. There also must be something wrong on the docker side of things. No other application has these issues.

It's most likely not the Docker side but rather Amazon. Docker hosts on Amazon and there are other applications that also show these issues, i.e. the Elite: Dangerous game launcher can't download the game files (hosted on Amazon), and the JetBrains Toolbox for the JetBrains developer tools also can't download new versions of the tools (also hosted on Amazon) in this constellation.

Strangely enough, the Jetbrains Toolbox issue also occured with a colleague on his Mac, while the other issues only affect Windows machines.

Yesterday AVM released a new FritzBox! firmware that allegedly fixes these issues. I hope Vodafone is fast adopting the changes this time for their branded boxes.

[Spenhouet]

Yesterday AVM released a new Fritz!Box firmware that allegedly fixes these issues. I hope Vodafone is fast adopting the changes this time for their branded boxes.

We should not count on that. I have little hope that Vodafone will push any update (soon or not at all). This probably has to be fixed on behalf of Docker, Amazon, ... or who ever can.

Docker it not working for me on Windows at all. Not on WSL and not via Hyper-V.

EDIT: Docker does work on a pure Linux system in the same network. This therefore definitely is not a pure Issue of the Fritz!Box.

[Spenhouet]

An additional observation:

• 🟢 nvcr.io
• 🔴 docker.io

There seems to be some difference between registries. The NVIDIA registry works while docker hub does not.

[Cartman0]

i don't use Fritzbox router. so, behavior of Docker Desktop for win / maybe funny when network connection problem happen.. when network disconnect and reconnect, EOF may happen?

[JanFellner]

Same constellation here (Vodafone - FRITZ!Box 6591 Cable - FRITZ!OS 07.13). Strange unexpected eof issues while docker login.

[cybertschunk]

Update: I managed to get docker working again inside a virtualized linux installation. The windows host system is still unable to pull any images. This really seems to be a windows related problem.

[pluralistix]

It's not Windows only. I do have the same problem FritzBox 6660 (07.19-80490 BETA) with Vodafone Red Internet & Phone 1000 Cable. docker pull alpine on Fedora 32 "Client.Timeout exceeded while awaiting headers" while docker pull 000.dkr.ecr.eu-west-1.amazonaws.com/dummy succeeds. Disabling Hardware Packet Acceleration or using a VPN from my company (terrible slow) "fixes" it.

[fcolling]

It's not Windows only. I do have the same problem FritzBox 6660 (07.19-80490 BETA) with Vodafone Red Internet & Phone 1000 Cable. docker pull alpine on Fedora 32 "Client.Timeout exceeded while awaiting headers" while docker pull 000.dkr.ecr.eu-west-1.amazonaws.com/dummy succeeds. Disabling Hardware Packet Acceleration or using a VPN from my company (terrible slow) "fixes" it.

PROBLEM WAS ISOLATED!! I use windows 10 and FritzBox 6660 cable 500mbps and I have the same problem, but if I use my old fritzbox 7360 20mbps it works perfectly.

[SteveScott]

I have no idea what box I have but I am in an enterprise environment in America and I am getting the EOF error.

[molesmoke]

I have the same issue pulling Windows containers (Linux containers appear to be OK on a Linux host, but this could just be because the layers I'm pulling there are smaller). If I try repeatedly, it does eventually succeed. My network setup is a little unusual - a rural setup through a chain of Ubiquiti PowerBeam AC dishes. I've not noticed any issues downloading large files from other sources though.

[SteveScott]

I am pulling a Linux container on Windows. I work for government so it is behind so many firewalls. I don't know where to start. I think I am connecting to the proxy server. It just rejects the request?

On Mon, Jul 27, 2020, 6:07 PM molesmoke notifications@github.com wrote:

I have the same issue pulling Windows containers (Linux containers appears to be OK on a Linux host, but this could just be because the layers I'm pulling there are smaller). If I try repeatedly, it does eventually succeed. My network setup is a little unusual - a rural setup through a chain of Ubiquiti PowerBeam AC dishes. I've not noticed any issues downloading large files from other sources though.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/docker/for-win/issues/6192#issuecomment-664662677, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABHLFSLF6CNKHQI66SJYJJDR5X3BPANCNFSM4LY5HRTA .

[hgross]

I have the same issue: Windows 10 over 1GBit ethernet via 500 MBit UnityMedia (Vodafone) - FRITZ!Box 6591 Cable - FRITZ!OS 07.13 - DSLite connection

[tfrommen]

I have the same issue: Windows 10 over 1GBit ethernet via 500 MBit UnityMedia (Vodafone) - FRITZ!Box 6591 Cable - FRITZ!OS 07.13 - DSLite connection

💯 this one!

[SteveScott]

My problem was caused by the network. I entered an HTTP and an HTTPS connection, but the proxy only accepts HTTP requests. I changed both secure and insecure web server URLs to HTTP. I hear that this is common with many proxy servers: some only accept HTTP internally.

[SteffenMangold]

Same here with Vodafone - FRITZ!Box 6591 Cable - FRITZ!OS 07.13 Deactivating hardware acceleration fixed it.

Because I find it hard to find, maybe this helps otheres:

[KorbenDev]

Had the same issue. Also my NAS couldn't download images, but had a different error message: "Client.Timeout exceeded while awaiting headers". After disabling hardware acceleration on my Fritzbox it worked just fine. Router: AVM Fritzbox 6660 (cable) Firmware: Fritz!OS 07.19-80783 BETA Provider: Vodafone (1 Gbit cable)

PS: gave feedback to AVM about the issue with a link to this here.

[KorbenDev]

For me (as a AVM FB6660 user) the issue has been solved with today's new BETA Firmware (07.19-81008 BETA). Download of Docker images now works fine without disabling hardware acceleration.

[Fleshgrinder]

I have a FRITZ!Box 6591 Cable from Vodafone (branded) with a 1G connection in Berlin and disabling hardware acceleration also fixed the issue for me.

[VanillaFord]

same here with a branded FRITZ!Box 6591 Cable from Vodafone and with a 1G connection. Unfortunately the connection slows down to below 500mbps with hardware acceleration off. Vodafone is not known for updating the firmware frequently, so this is going to be a major issue, even if AVM will be able to fix it in a subsequent firmware update. My Linux NAS is able to pull images from docker-hub just fine, with hardware acceleration enabled.

[ThomasTosik]

I cant believe that this was the problem. At least i can work from home again... Same story as the others. FritzBox 6591, Vodafone, OS 07.13.

[emuuu]

I have a FRITZ!Box 6591 Cable from Vodafone (branded) with a 1G connection in Berlin and disabling hardware acceleration also fixed the issue for me.

Can confirm this fix on branded 6591 1G connection for ruhr valley region.

There seem to be some major issues with Fritz!OS. Previously I had problems with non functional port fowarding.

[thimo-seitz]

Recently my FritzBox 6660 updated to 07.21, which fixed the issue for me.

[shivan]

Here the same, FritzBox 6591, OS 7.13

Hopefully Vodafone will soon update to 7.20 or newer.