yum commands (layers?) in docker build unreliable

[pdmack]

oc v3.2.1.4-1-g1864c8f kubernetes v1.2.0-36-g4a3f9c5 Docker version 1.10.3, build 02f13c2-unsupported

95% reproducible in my env. yum commands in the docker build seem to render missing or corrupt layers in the incremental image build for some reason.

Step 5 : RUN yum install -y nss_wrapper && yum clean all
 ---> Running in e0bfafd00c68
Loaded plugins: fastestmirror, ovl

 One of the configured repositories failed (Unknown),
 and yum doesn't have enough cached data to continue. At this point the only
 safe thing yum can do is fail. There are a few ways to work "fix" this:

     1. Contact the upstream for the repository and get them to fix the problem.

     2. Reconfigure the baseurl/etc. for the repository, to point to a working
        upstream. This is most often useful if you are using a newer
        distribution release than is supported by the repository (and the
        packages for the previous distribution release still work).

     3. Disable the repository, so yum won't use it by default. Yum will then
        just ignore the repository until you permanently enable it again or use
        --enablerepo for temporary usage:

            yum-config-manager --disable <repoid>

     4. Configure the failing repository to be skipped, if it is unavailable.
        Note that yum will try to contact the repo. when it runs most commands,
        so will have to try and fail each time (and thus. yum will be be much
        slower). If it is a very temporary problem though, this is often a nice
        compromise:

            yum-config-manager --save --setopt=<repoid>.skip_if_unavailable=true

Cannot find a valid baseurl for repo: base/$releasever/x86_64
The command '/bin/sh -c yum install -y nss_wrapper && yum clean all' returned a non-zero code: 1
make: *** [build] Error 1

[pdmack]

Note the discrepancy at the yum install step between what should be equivalent images.

# docker history spark
IMAGE               CREATED              CREATED BY                                      SIZE                COMMENT
ba5db1137267        About a minute ago   /bin/sh -c #(nop) CMD ["/opt/spark/bin/launch   0 B
f8e7d841254d        About a minute ago   /bin/sh -c #(nop) USER [185]                    0 B
1e87af517fc7        About a minute ago   /bin/sh -c rm -rf /tmp/scripts                  0 B
51d89734276a        About a minute ago   /bin/sh -c #(nop) USER [root]                   0 B
312a372bd5d0        About a minute ago   bash -x /tmp/scripts/spark/install              160.1 MB
4c021ee96b43        About a minute ago   /bin/sh -c #(nop) USER [root]                   0 B
11de19b447c2        About a minute ago   /bin/sh -c #(nop) COPY dir:d0ec2e0b58c72c428e   2.632 kB
c7425d579d89        About a minute ago   /bin/sh -c #(nop) COPY file:39370d29b5da736c0   140.4 MB
10d3ba97c387        2 minutes ago        /bin/sh -c yum install -y --disablerepo=\* --   0 B
563f11a0198d        2 minutes ago        /bin/sh -c #(nop) COPY file:d9489eab7f56d1baf   1.152 kB
d9b83fa709f2        2 minutes ago        /bin/sh -c #(nop) USER [root]                   0 B
e93b2de9da72        2 minutes ago        /bin/sh -c #(nop) LABEL Name=apache-spark-2/s   0 B
dded24043a49        2 minutes ago        /bin/sh -c #(nop) ENV JBOSS_IMAGE_NAME=apache   0 B
f45a59bd1606        3 months ago                                                         195.4 MB
<missing>           3 months ago                                                         12.22 MB
<missing>           5 months ago                                                         203.3 MB            Imported from -
# docker history 172.24.253.57:5000/spark/spark
IMAGE               CREATED             CREATED BY                                      SIZE                COMMENT
8516b6a2fac0        6 days ago          /bin/sh -c #(nop) CMD ["/opt/spark/bin/launch   0 B
086874f5378c        6 days ago          /bin/sh -c #(nop) USER [185]                    0 B
4c6cf4297d92        6 days ago          /bin/sh -c rm -rf /tmp/scripts                  0 B
06638280887f        6 days ago          /bin/sh -c #(nop) USER [root]                   0 B
173f6f0d9bcb        6 days ago          bash -x /tmp/scripts/spark/install              160.1 MB
0e9ab4c0febe        6 days ago          /bin/sh -c #(nop) USER [root]                   0 B
abbe4c29ac97        6 days ago          /bin/sh -c #(nop) COPY dir:d0ec2e0b58c72c428e   2.632 kB
18d085f493ff        6 days ago          /bin/sh -c #(nop) COPY file:39370d29b5da736c0   140.4 MB
ca1354bfe2a0        6 days ago          /bin/sh -c yum install -y --disablerepo=\* --   49.42 MB
889793b021a9        6 days ago          /bin/sh -c #(nop) COPY file:d9489eab7f56d1baf   1.152 kB
10644e1d0321        6 days ago          /bin/sh -c #(nop) USER [root]                   0 B
1388268a0628        6 days ago          /bin/sh -c #(nop) LABEL Name=apache-spark-2/s   0 B
15dd4cb9b9ae        6 days ago          /bin/sh -c #(nop) ENV JBOSS_IMAGE_NAME=apache   0 B
f45a59bd1606        3 months ago                                                        195.4 MB
<missing>           3 months ago                                                        12.22 MB
<missing>           5 months ago                                                        203.3 MB            Imported from -

[pdmack]

BTW, docker hub layer visualization here.

[pdmack]

Can't reproduce with latest docker using loop device storage. Will retry with thinpool.

[pdmack]

Still see this which I presume is the detection of the yum sqlitedb being copied up in the Overlay (?). Warning: RPMDB altered outside of yum. However, no longer able to reproduce with upgraded docker.

# docker version
Client:
 Version:         1.10.3
 API version:     1.22
 Package version: docker-common-1.10.3-54.el7.x86_64
 Go version:      go1.6.2
 Git commit:      0b52dd1-unsupported
 Built:           Mon Sep 12 10:58:52 2016
 OS/Arch:         linux/amd64

Server:
 Version:         1.10.3
 API version:     1.22
 Package version: docker-common-1.10.3-54.el7.x86_64
 Go version:      go1.6.2
 Git commit:      0b52dd1-unsupported
 Built:           Mon Sep 12 10:58:52 2016
 OS/Arch:         linux/amd64