search

docker / for-mac

Bug reports for Docker Desktop for Mac
https://www.docker.com/products/docker#/mac
2.43k stars 118 forks source link

ERROR: failed to register layer downloading images after 17.12 upgrade #2388

Open hallatech opened 6 years ago

hallatech commented 6 years ago

Expected behavior

After installing new version and resetting all data to take advantage of RAW mode in APFS: Run docker-compose up Download missing images as before

Actual behavior

Run docker-compose up On downloading images as part of extraction of one of the layers encounters unexpected EOF: Tried running docker pull but most failed with the same issue as an example below:

5881d531c681: Pull complete 03c1eba1045d: Pull complete 3542ea36e713: Pull complete a967fabdedb2: Pull complete d8f89e6fbd56: Extracting [==================================================>] 720.5MB/720.5MB fc04d82d0e99: Download complete e18dcb4676b0: Download complete ERROR: failed to register layer: Error processing tar file(exit status 1): unexpected EOF

Information

Steps to reproduce the behavior

  1. Upgraded Docker to latest version Version 17.12.0-ce-mac46 (21698). Yesterday was on previous latest version.
  2. Docker -> Reset -> Remove All Data
  3. Run docker-compose up also tried: docker pull

One or 2 smaller images worked and one worked after numerous retries but most failed consistently to extract. Looking in ~/Library/Containers/com.docker.docker/Data/ there is no longer a .dcow2 file and there is no .raw either.

hallatech commented 6 years ago

If I look under the Docker UI under Disk there is a .raw file here: ~/Library/Containers/com.docker.helper/Data/Data/com.docker.driver.amd64-linux/Docker.raw

hallatech commented 6 years ago

I reset and ran "Remove All Data" and then reset->"Reset to factory defaults" Then ran individual downloads. The first 2 worked (as before) then the next failed again with the same error. I looked at Preferences->Disk and now the raw file is at ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/Docker.raw Notice in the previous one it had Data/Data now there was just (correctly) 1 level of Data/ dirs. But overall problem is the same

hallatech commented 6 years ago

I tried the beta/edge version and had the same issues on raw. Luckily I had a copy of 17.09 in my trash. I've reverted to that version and all is working again (on qcow).

endzyme commented 6 years ago

We too have seen this error on larger images. We've seen it inside the container as well with decompressing larger files (database restore testing).

TechnotronicOz commented 6 years ago

I have this same issue after upgrading to utilize the raw format. If I try 4-5 times, it'll eventually work but it is quite a pain.

Unable to find image 'internal.server/large-test-image:latest' locally
latest: Pulling from large-test-image
94b97ecd508d: Pull complete
eac724b14380: Pull complete
7660cea5417b: Pull complete
a35168133d19: Pull complete
8cc5c503e949: Pull complete
b5cf75bafa45: Pull complete
c113fafcbf3c: Pull complete
d730356d5d49: Pull complete
e963dea183e7: Pull complete
7156fbd3c2c9: Pull complete
d39f751d6fbe: Pull complete
85732aa65078: Pull complete
4a5278c25897: Extracting [==================================================>]  7.922GB/7.922GB
docker: failed to register layer: Error processing tar file(exit status 1): unexpected EOF.
Docker for Mac: version: 17.12.0-ce-mac49 (d1778b704353fa5b79142a2055a2c11c8b48a653)
macOS: version 10.13.2 (build: 17C88)
logs: /tmp/737F9D25-841A-4776-B343-EECE39E2D9A8/20180123-102324.tar.gz
[OK]     db.git
[OK]     vmnetd
[OK]     dns
[OK]     driver.amd64-linux
[OK]     virtualization VT-X
[OK]     app
[OK]     moby
[OK]     system
[OK]     moby-syslog
[OK]     kubernetes
[OK]     env
[OK]     virtualization kern.hv_support
[OK]     slirp
[OK]     osxfs
[OK]     moby-console
[OK]     logs
[OK]     docker-cli
[OK]     menubar
[OK]     disk
tvtamas commented 6 years ago

Experiencing the same issue:

docker run -d -P --name android tracer0tong/android-emulator
Unable to find image 'tracer0tong/android-emulator:latest' locally
latest: Pulling from tracer0tong/android-emulator
ae79f2514705: Pull complete 
5ad56d5fc149: Pull complete 
170e558760e8: Pull complete 
395460e233f5: Pull complete 
6f01dc62e444: Pull complete 
7b14867b4577: Pull complete 
dca319f54e7a: Pull complete 
7d0956d6bae2: Pull complete 
291a34750f4c: Extracting [==================================================>]  1.405GB/1.405GB
fe5a951d2b52: Download complete 
da51e890cdee: Download complete 
724943869aee: Download complete 
docker: failed to register layer: Error processing tar file(exit status 1): unexpected EOF.
Docker for Mac: version: 17.12.0-ce-mac49 (d1778b704353fa5b79142a2055a2c11c8b48a653)
macOS: version 10.13.3 (build: 17D47)
logs: /tmp/B8011AB7-0CE4-409A-818A-8C8C4170D222/20180205-140929.tar.gz
[OK]     db.git
[OK]     vmnetd
[OK]     dns
[OK]     driver.amd64-linux
[OK]     virtualization VT-X
[OK]     app
[OK]     moby
[OK]     system
[OK]     moby-syslog
[OK]     kubernetes
[OK]     env
[OK]     virtualization kern.hv_support
[OK]     slirp
[OK]     osxfs
[OK]     moby-console
[OK]     logs
[OK]     docker-cli
[OK]     menubar
[OK]     disk
walkingpendulum commented 6 years ago

Same issue

YouStock commented 6 years ago

I was getting this error on another project, and this stackoverflow answer fixed it for me https://stackoverflow.com/a/45038500/1114724 . Somehow permissions are getting messed up. Need to run chown -R on the project folder

endzyme commented 6 years ago

@YouStock. FYI most people in this thread appear to be receiving this on a docker pull/run so file permissions should not be involved here.

TechnotronicOz commented 6 years ago

This is related to something with the new raw format. Using the same image, it works with qcow2, but with raw, you seem to see this issue all the time. Sometimes it'll take 4 or 5 times before you get one successful, other times it has failed twelve in a row.

hallatech commented 6 years ago

Latest version 17.12.0-ce-mac55 2018-02-27 now resolves the large file download corruption after factory reset issue by reverting to qcow. So I have upgraded and tried the factory reset so far without issues on the large files. (Obviously this doesn't fix the root cause of switching to raw). Thank you.

brock-bouchard commented 6 years ago

We've seen it inside the container as well with decompressing larger files (database restore testing)

I just upgraded to Docker for Mac 18.06.0-ce-mac70, and with the raw format enabled by default I'm now seeing this problem of decompressing larger files as part of a database restore.

djs55 commented 6 years ago

@brock-bouchard could you upload a set of diagnostics? Also could you let me know which macOS version you're using? There were a set of APFS-related bugs which were fixed in macOS 10.13.4 which I assumed also explained this issue, but perhaps not. Do you have any repro steps you could share?

brock-bouchard commented 6 years ago

@djs55:

djs55 commented 6 years ago

@brock-bouchard to upload diagnostics: click on the whale menu, then "Diagnose and Feedback..." then wait a minute or two for it to gather data, then hit the "Upload" button and then it prints an ID which you should quote here. I can then download the logs and take a look at them.

Having said that though, a local reproduction would be much more valuable for me. If you can create something you can share that would be really helpful! I'll also see if I can make something fail with a similar large archive upload.

robertmiles3 commented 6 years ago

I've also had this issue for a while now with the raw format (everything works fine with qcow2). I posted an issue on moby/moby#37305 but never got any response. It's definitely related to the raw file format.

robertmiles3 commented 6 years ago

With the latest Docker Edge (18.06.0-ce-mac69) I am now even getting this error while trying to ADD a 8GB database backup file into a container like so...

FROM microsoft/mssql-server-linux:2017-CU8

RUN mkdir -p /usr/db/backup
ADD setup.sql /usr/db/backup
ADD foo.bak /usr/db/backup

My diagnostic ID is...

1B273E59-D204-4EEB-B2C3-B6DAB9712BBE/20180730123740

flagbug commented 6 years ago

Having basically the same issue as @robertmiles3 has. When restoring a large database file inside SQL Server For Linux, I'm getting Error processing tar file(exit status 1): unexpected EOF on Docker Stable (18.06.0-ce-mac70)

flagbug commented 6 years ago

@djs55 I've managed to reliably reproduce this with a super simple Dockerfile:

FROM ubuntu:18.04

RUN head -c 8G </dev/urandom >file

docker build -t test .

Output:

Step 1/2 : FROM ubuntu:18.04
 ---> 735f80812f90
Step 2/2 : RUN head -c 8G </dev/urandom >file
 ---> Running in 24eba72a7885
Error processing tar file(exit status 1): unexpected EOF

Interestingly, this chokes on an 8GB file, but not on a 7GB file

My diagnostics ID is E609D3B9-5C94-42EC-A383-C752066BD4AF/20180801104256, if that's relevant

ExitStatus commented 6 years ago

Same happens in Windows and Linux. Restoring a large sql server backup in a mssql container gives error.

Error processing tar file(exit status 1): unexpected EOF

djs55 commented 6 years ago

@flagbug thanks for the repro example. I think this is a engine bug triggered when a file size hits 8GiB due to a limitation in the size of a tar header. I've filed: https://github.com/moby/moby/issues/37581

flagbug commented 6 years ago

@djs55 Sure thing, thanks for moving this to the correct repo!

robertmiles3 commented 6 years ago

Yes, thanks @djs55! Nice digging on the 8G header limit. I'm glad there is some movement on this.

docker-robott commented 5 years ago

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale comment. Stale issues will be closed after an additional 30d of inactivity.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so.

Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows. /lifecycle stale

djs55 commented 5 years ago

/remove-lifecycle stale /lifecycle frozen

djs55 commented 5 years ago

The fix should be in the next release (18.09)