17.12.0-ce-mac46 build image, files inside of container corrupted

[Dani3lSun]

After updating to the newest stable version and replacing the old VM with the new raw VM I´m not able to build my images any more. It seems that some Zip files which get added via ADD command are corrupted, the unzip inside of the build container brings up CRC errors and the files are not unzipped properly...

I downloaded all Zip files again, but same behavior...

My dockerfile looks like this:

FROM oraclelinux:7.4

MAINTAINER Daniel Hochleitner <...>

# environment variables
ENV INSTALL_APEX=true \
    INSTALL_FOEX=true \
    INSTALL_SQLCL=true \
    INSTALL_LOGGER=true \
    DBCA_TOTAL_MEMORY=2048 \
    ORACLE_SID=orcl \
    SERVICE_NAME=orcl \
    ORACLE_BASE=/u01/app/oracle \
    ORACLE_HOME=/u01/app/oracle/product/12.2.0.1/dbhome \
    ORACLE_INVENTORY=/u01/app/oraInventory \
    PASS=oracle \
    ORDS_HOME=/u01/ords \
    JAVA_HOME=/opt/java \
    TOMCAT_HOME=/opt/tomcat \
    APEX_PASS=XXXXXX \
    APEX_ADDITIONAL_LANG= \
    TIME_ZONE=Europe/Berlin

# copy all scripts
ADD scripts /scripts/

# copy all files
ADD files /files/

# image setup via shell script to reduce layers and optimize final disk usage
RUN /scripts/install_main.sh

# ssh, database and apex port
EXPOSE 22 1521 8080

# entrypoint for database creation, startup and graceful shutdown
ENTRYPOINT ["/scripts/entrypoint.sh"]

Information

Docker for Mac: version: 17.12.0-ce-mac46 (a61e84b8bca06b1ae6ce058cdd7beab1520ad622)
macOS: version 10.13.2 (build: 17C205)
logs: /tmp/74D02CC7-2977-4D69-BEFA-DE2CB096DCB8/20180112-094233.tar.gz
[OK]     db.git
[OK]     vmnetd
[OK]     dns
[OK]     driver.amd64-linux
[OK]     virtualization VT-X
[OK]     app
[OK]     moby
[OK]     system
[OK]     moby-syslog
[OK]     kubernetes
[OK]     env
[OK]     virtualization kern.hv_support
[OK]     slirp
[OK]     osxfs
[OK]     moby-console
[ERROR]  logs
         logs check failed with: (Failure
  "exec: /usr/bin/log show --style syslog --last 1d | grep -i docker > \"/tmp/74D02CC7-2977-4D69-BEFA-DE2CB096DCB8/20180112-094233/docker-system-os.log\": exit 1")
[OK]     docker-cli
[OK]     menubar
[OK]     disk

[djs55]

Thanks for your report. We'll try to reproduce this locally but in the meantime you can revert to the Docker.qcow2 format following these instructions: https://github.com/docker/for-mac/issues/2327#issuecomment-351842252

[Dani3lSun]

Thanks, will try that! I´ll report how things are gone...

[Dani3lSun]

Thus before my tests a new version 17.12.0-ce-mac47 arrived, I tested with this and the raw VM. But same erros, the files were corrupted... So I changed to the older qcow2 format and this indeed fixed the errors, my builds are running fine until end and everything is good now! The errors must be depending on this new VM file format...I hope this will get fixed soon, thus the performance boost of this new format would be great to have!

[benhoskings]

I'm seeing the same behaviour. On an alpine-3.7 based image, running apk add pkg results in a corrupted binary, and bundling a rails app results in a few of the files within the bundle being corrupted.

I was using docker build . to create an image at the time, so was left with an image containing a corrupted file. I added a no-op above that point in the Dockerfile and re-ran, resulting in a good image. Extracting the binaries from both using docker cp and comparing them showed their file sizes were identical, and the bad file was intact until exactly the 21MB mark, after which point it was garbled. (In this case the binary was /usr/bin/node.)

$ ls -l
total 86272
-rwxr-xr-x  1 ben  staff  22082368  9 Dec 11:48 node.bad
-rwxr-xr-x  1 ben  staff  22082368  9 Dec 11:48 node.good
$ cmp node.good node.bad
node.good node.bad differ: char 22020097, line 98942
$ echo '21 * 1024 * 1024' | bc
22020096

[benhoskings]

@djs55 Switching back to qcow seems to have fixed the problem for now - thanks for the tip.

[ericbuehl]

I am also seeing this issue. I used to use the following command to check the consistency of images but it doesn't seem to be working as expected in later versions of docker.

docker save <image> -o /dev/null && echo 'OK' || echo 'Corrupted'

Always results in:

open /dev/.docker_temp_414428247: operation not permitted
Corrupted

Not sure if the change in the save operation is necessarily a regression, but it does leave me without a good way to verify that images aren't corrupted.

[docker-robott]

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale comment. Stale issues will be closed after an additional 30d of inactivity.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so.

Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows. /lifecycle stale

[PhilippSalvisberg]

Version 18.03.1-ce-mac65 (24312) is using the qcow2 format as default. If there are plans to support the raw format then I suggest to keep this issue open. /remove-lifecycle stale

[djs55]

/lifecycle frozen

[djs55]

A quick update: we've been testing the raw format on the recent macOS 10.13.4 release and it seems to be reliable-- we've had no instances of corruption so far (but we could reproduce the problem on previous versions). Therefore future versions of Docker will use raw by default if the version of macOS is new enough and the feature is available.

Thanks for your patience!

[YRM64]

Djs55, thank you so much; we are following you, watching, and patiently awaiting all updates.

[PhilippSalvisberg]

@djs55 I switched to raw format in Version 18.06.0-ce-mac70 (26399). Looks good so far. Thanks.

[Dani3lSun]

@djs55 for me the same, I switched to raw (also build 18.06.0-ce-mac70), and for now everything is fine! Thanks!