fabric8io-images / s2i

OpenShift S2I images for Java and Karaf applications
Apache License 2.0
70 stars 84 forks source link

Broken (community) image due to today's release of CentOS 7.6 #206

Closed vorburger closed 5 years ago

vorburger commented 5 years ago

Unless I'm stupid, I think the new CentOS just released to Docker Hub, hours ago, broke ./test.sh for the existing Java 8 image (forget about Java 11 for #193) ... this seems weird, and worth investigating and possibly escalating to the CentOS community, if confirmed.

vorburger commented 5 years ago

Yup, it would seem to me that despite using the completely fixed versions (which is defined here) using the Base Container FROM "centos:7.5.1804" and RPM yum install "java-1.8.0-openjdk-1.8.0.181-3.b13.el7_5", the release of the new CentOS 7.6 container image just broke the old one 7.5.1804, check this out, which worked before:

Step 7/37 : RUN yum install -y        unzip        java-1.8.0-openjdk-1.8.0.181-3.b13.el7_5        java-1.8.0-openjdk-devel-1.8.0.181-3.b13.el7_5   && curl https://archive.apache.org/dist/maven/maven-3/3.5.4/binaries/apache-maven-3.5.4-bin.tar.gz |     tar -xzf - -C /opt   && ln -s /opt/apache-maven-3.5.4 /opt/maven   && ln -s /opt/maven/bin/mvn /usr/bin/mvn   && groupadd -r jboss -g 1000   && useradd -u 1000 -r -g jboss -m -d /opt/jboss -s /sbin/nologin -c "JBoss user" jboss   && chmod 755 /opt/jboss
 ---> Using cache
 ---> 093f954f1101
Step 8/37 : ENV JAVA_HOME /etc/alternatives/jre
 ---> Using cache
 ---> 4720429c8d1d
Step 9/37 : RUN echo securerandom.source=file:/dev/urandom >> /usr/lib/jvm/java/jre/lib/security/java.security  && usermod -g root -G jboss jboss
 ---> Running in fc3c875102bb
/bin/sh: /usr/lib/jvm/java/jre/lib/security/java.security: No such file or directory
The command '/bin/sh -c echo securerandom.source=file:/dev/urandom >> /usr/lib/jvm/java/jre/lib/security/java.security  && usermod -g root -G jboss jboss' returned a non-zero code: 1

What's curious is that centos:7.5.1804 on https://hub.docker.com/r/library/centos/tags/ says "2 months ago" ... unless I'm not getting it from there? This is weird... @ctron @rhuss please jump in and shout if you get what' going on here...

vorburger commented 5 years ago

Also seen here https://circleci.com/gh/fabric8io-images/s2i/498 following merge of #205 - not caused by it.

ctron commented 5 years ago

Looks like the parent directory is missing

vorburger commented 5 years ago

Looks like the parent directory is missing

sure but how did it suddenly disappear given a fixed version of base image and OpenJDK?

This is really weird, the CentOS (7.5.1804) image hasn't changed in 4 months even locally:

docker.io/tripleomaster/centos-binary-opendaylight   current-tripleo-rdo   78eaf446417a        11 days ago         977 MB
docker.io/centos                                     7.5.1804              fdf13fa91c6e        4 months ago        200 MB
docker.io/centos                                     7                     e934aafc2206        8 months ago        199 MB

I'm going to look more at the OpenJDK (RPM) installation side of things...

vorburger commented 5 years ago

Commenting out that echo securerandom.source=file:/dev/urandom >> /usr/lib/jvm/java/jre/lib/security/java.security and building it without that to have a look inside, we see that somehow the Java installation is completely broken:

docker build java/images/centos/ -t test
docker run --rm -it test bash
[jboss@e4f73e0a4975 ~]$ java -version
bash: java: command not found

Did I just completely break this with #205 ? I had tested it, and ./test.sh had passed, that seems unlikely.

vorburger commented 5 years ago

Did I just completely break this with #205 ? I had tested it, and ./test.sh had passed

Looks like it.. does the exact position of the EXPOSE matter? Top NOK needs to be last for OK? Trying..

vorburger commented 5 years ago

Ignore earlier comments above re. CentOS, this was a misinterpretation; it's got absolutely nothing to do with that, but it's the ARG EXPOSE_PORT=8080 (not the EXPOSE ${EXPOSE_PORT}) which somehow, weirdly, seemed to cause this. And only for the CentOS 7.5 Java 8 image, whereas for the Fedora 29 one wasn't affected by this; makes no sense at all, to me. I'm just removing that ARG again and hard-coding 8080, for now - that fixes it.

vorburger commented 5 years ago

It still failed on https://circleci.com/gh/fabric8io-images/s2i/500, and now that I'm looking more closely I am seeing what already on https://circleci.com/gh/fabric8io-images/s2i/498 4 hours ago but I initially missed:

No package java-1.8.0-openjdk-1.8.0.181-3.b13.el7_5 available.
No package java-1.8.0-openjdk-devel-1.8.0.181-3.b13.el7_5 available.
vorburger commented 5 years ago

It likely had nothing at all to do with #115, so reverting a9bd456 in c23d43fcbf0b53dc53a4fec87abed75f73337761.

80d07d6babad87081ddb6226aa1e9a4b1bc3cb9c ensures future non-regression and helps to sure that we get a much clearer error now:

Step 5/36 : RUN yum install -y --setopt=skip_missing_names_on_install=False        unzip        java-1.8.0-openjdk-1.8.0.181-3.b13.el7_5        java-1.8.0-openjdk-devel-1.8.0.181-3.b13.el7_5   && curl https://archive.apache.org/dist/maven/maven-3/3.5.4/binaries/apache-maven-3.5.4-bin.tar.gz |     tar -xzf - -C /opt   && ln -s /opt/apache-maven-3.5.4 /opt/maven   && ln -s /opt/maven/bin/mvn /usr/bin/mvn   && groupadd -r jboss -g 1000   && useradd -u 1000 -r -g jboss -m -d /opt/jboss -s /sbin/nologin -c "JBoss user" jboss   && chmod 755 /opt/jboss
 ---> Running in 8baf5142e639
Loaded plugins: fastestmirror, ovl
Determining fastest mirrors
 * base: mirror.init7.net
 * extras: mirror.init7.net
 * updates: mirror.init7.net
No package java-1.8.0-openjdk-1.8.0.181-3.b13.el7_5 available.
Error: Not tolerating missing names on install, stopping.
The command '/bin/sh -c yum install -y --setopt=skip_missing_names_on_install=False        unzip        java-1.8.0-openjdk-1.8.0.181-3.b13.el7_5        java-1.8.0-openjdk-devel-1.8.0.181-3.b13.el7_5   && curl https://archive.apache.org/dist/maven/maven-3/3.5.4/binaries/apache-maven-3.5.4-bin.tar.gz |     tar -xzf - -C /opt   && ln -s /opt/apache-maven-3.5.4 /opt/maven   && ln -s /opt/maven/bin/mvn /usr/bin/mvn   && groupadd -r jboss -g 1000   && useradd -u 1000 -r -g jboss -m -d /opt/jboss -s /sbin/nologin -c "JBoss user" jboss   && chmod 755 /opt/jboss' returned a non-zero code: 1

The observed very confusing behaviour which lead me down a wrong interpretation re. ARG, EXPOSE and CentOS 7.5 Java 8 vs Fedora 29 Java 11 was perhaps somehow related to (a bug with not rebuilding when it should have?) local Docker Image layer caching?

Now we're back to square one, and I do suspect something is really wrong, in Docker itself, or on Docker Hub, or with the CentOS image update, because how on earth is this possible:

docker run --rm -it centos:7.5.1804 yum list "java-*"
Loaded plugins: fastestmirror, ovl
Determining fastest mirrors
 * base: pkg.adfinis-sygroup.ch
 * extras: mirror.infonline.de
 * updates: mirror.init7.net
base                                                                                                                       | 3.6 kB  00:00:00     
extras                                                                                                                     | 3.4 kB  00:00:00     
updates                                                                                                                    | 3.4 kB  00:00:00     
(1/4): base/7/x86_64/group_gz                                                                                              | 166 kB  00:00:00     
(2/4): extras/7/x86_64/primary_db                                                                                          | 154 kB  00:00:00     
(3/4): updates/7/x86_64/primary_db                                                                                         | 1.3 MB  00:00:01     
(4/4): base/7/x86_64/primary_db                                                                                            | 6.0 MB  00:00:02     
Available Packages
java-1.6.0-openjdk.x86_64                                                      1:1.6.0.41-1.13.13.1.el7_3                                  base   
java-1.6.0-openjdk-demo.x86_64                                                 1:1.6.0.41-1.13.13.1.el7_3                                  base   
java-1.6.0-openjdk-devel.x86_64                                                1:1.6.0.41-1.13.13.1.el7_3                                  base   
java-1.6.0-openjdk-javadoc.x86_64                                              1:1.6.0.41-1.13.13.1.el7_3                                  base   
java-1.6.0-openjdk-src.x86_64                                                  1:1.6.0.41-1.13.13.1.el7_3                                  base   
java-1.7.0-openjdk.x86_64                                                      1:1.7.0.201-2.6.16.1.el7_6                                  updates
java-1.7.0-openjdk-accessibility.x86_64                                        1:1.7.0.201-2.6.16.1.el7_6                                  updates
java-1.7.0-openjdk-demo.x86_64                                                 1:1.7.0.201-2.6.16.1.el7_6                                  updates
java-1.7.0-openjdk-devel.x86_64                                                1:1.7.0.201-2.6.16.1.el7_6                                  updates
java-1.7.0-openjdk-headless.x86_64                                             1:1.7.0.201-2.6.16.1.el7_6                                  updates
java-1.7.0-openjdk-javadoc.noarch                                              1:1.7.0.201-2.6.16.1.el7_6                                  updates
java-1.7.0-openjdk-src.x86_64                                                  1:1.7.0.201-2.6.16.1.el7_6                                  updates
java-1.8.0-openjdk.i686                                                        1:1.8.0.191.b12-1.el7_6                                     updates
java-1.8.0-openjdk.x86_64                                                      1:1.8.0.191.b12-1.el7_6                                     updates
java-1.8.0-openjdk-accessibility.i686                                          1:1.8.0.191.b12-1.el7_6                                     updates
java-1.8.0-openjdk-accessibility.x86_64                                        1:1.8.0.191.b12-1.el7_6                                     updates
java-1.8.0-openjdk-accessibility-debug.i686                                    1:1.8.0.191.b12-1.el7_6                                     updates
java-1.8.0-openjdk-accessibility-debug.x86_64                                  1:1.8.0.191.b12-1.el7_6                                     updates
java-1.8.0-openjdk-debug.i686                                                  1:1.8.0.191.b12-1.el7_6                                     updates
java-1.8.0-openjdk-debug.x86_64                                                1:1.8.0.191.b12-1.el7_6                                     updates
java-1.8.0-openjdk-demo.i686                                                   1:1.8.0.191.b12-1.el7_6                                     updates
java-1.8.0-openjdk-demo.x86_64                                                 1:1.8.0.191.b12-1.el7_6                                     updates
java-1.8.0-openjdk-demo-debug.i686                                             1:1.8.0.191.b12-1.el7_6                                     updates
java-1.8.0-openjdk-demo-debug.x86_64                                           1:1.8.0.191.b12-1.el7_6                                     updates
java-1.8.0-openjdk-devel.i686                                                  1:1.8.0.191.b12-1.el7_6                                     updates
java-1.8.0-openjdk-devel.x86_64                                                1:1.8.0.191.b12-1.el7_6                                     updates
java-1.8.0-openjdk-devel-debug.i686                                            1:1.8.0.191.b12-1.el7_6                                     updates
java-1.8.0-openjdk-devel-debug.x86_64                                          1:1.8.0.191.b12-1.el7_6                                     updates
java-1.8.0-openjdk-headless.i686                                               1:1.8.0.191.b12-1.el7_6                                     updates
java-1.8.0-openjdk-headless.x86_64                                             1:1.8.0.191.b12-1.el7_6                                     updates
java-1.8.0-openjdk-headless-debug.i686                                         1:1.8.0.191.b12-1.el7_6                                     updates
java-1.8.0-openjdk-headless-debug.x86_64                                       1:1.8.0.191.b12-1.el7_6                                     updates
java-1.8.0-openjdk-javadoc.noarch                                              1:1.8.0.191.b12-1.el7_6                                     updates
java-1.8.0-openjdk-javadoc-debug.noarch                                        1:1.8.0.191.b12-1.el7_6                                     updates
java-1.8.0-openjdk-javadoc-zip.noarch                                          1:1.8.0.191.b12-1.el7_6                                     updates
java-1.8.0-openjdk-javadoc-zip-debug.noarch                                    1:1.8.0.191.b12-1.el7_6                                     updates
java-1.8.0-openjdk-src.i686                                                    1:1.8.0.191.b12-1.el7_6                                     updates
java-1.8.0-openjdk-src.x86_64                                                  1:1.8.0.191.b12-1.el7_6                                     updates
java-1.8.0-openjdk-src-debug.i686                                              1:1.8.0.191.b12-1.el7_6                                     updates
java-1.8.0-openjdk-src-debug.x86_64                                            1:1.8.0.191.b12-1.el7_6                                     updates
java-11-openjdk.i686                                                           1:11.0.1.13-3.el7_6                                         updates
java-11-openjdk.x86_64                                                         1:11.0.1.13-3.el7_6                                         updates
java-11-openjdk-debug.i686                                                     1:11.0.1.13-3.el7_6                                         updates
java-11-openjdk-debug.x86_64                                                   1:11.0.1.13-3.el7_6                                         updates
java-11-openjdk-demo.i686                                                      1:11.0.1.13-3.el7_6                                         updates
java-11-openjdk-demo.x86_64                                                    1:11.0.1.13-3.el7_6                                         updates
java-11-openjdk-demo-debug.i686                                                1:11.0.1.13-3.el7_6                                         updates
java-11-openjdk-demo-debug.x86_64                                              1:11.0.1.13-3.el7_6                                         updates
java-11-openjdk-devel.i686                                                     1:11.0.1.13-3.el7_6                                         updates
java-11-openjdk-devel.x86_64                                                   1:11.0.1.13-3.el7_6                                         updates
java-11-openjdk-devel-debug.i686                                               1:11.0.1.13-3.el7_6                                         updates
java-11-openjdk-devel-debug.x86_64                                             1:11.0.1.13-3.el7_6                                         updates
java-11-openjdk-headless.i686                                                  1:11.0.1.13-3.el7_6                                         updates
java-11-openjdk-headless.x86_64                                                1:11.0.1.13-3.el7_6                                         updates
java-11-openjdk-headless-debug.i686                                            1:11.0.1.13-3.el7_6                                         updates
java-11-openjdk-headless-debug.x86_64                                          1:11.0.1.13-3.el7_6                                         updates
java-11-openjdk-javadoc.i686                                                   1:11.0.1.13-3.el7_6                                         updates
java-11-openjdk-javadoc.x86_64                                                 1:11.0.1.13-3.el7_6                                         updates
java-11-openjdk-javadoc-debug.i686                                             1:11.0.1.13-3.el7_6                                         updates
java-11-openjdk-javadoc-debug.x86_64                                           1:11.0.1.13-3.el7_6                                         updates
java-11-openjdk-javadoc-zip.i686                                               1:11.0.1.13-3.el7_6                                         updates
java-11-openjdk-javadoc-zip.x86_64                                             1:11.0.1.13-3.el7_6                                         updates
java-11-openjdk-javadoc-zip-debug.i686                                         1:11.0.1.13-3.el7_6                                         updates
java-11-openjdk-javadoc-zip-debug.x86_64                                       1:11.0.1.13-3.el7_6                                         updates
java-11-openjdk-jmods.i686                                                     1:11.0.1.13-3.el7_6                                         updates
java-11-openjdk-jmods.x86_64                                                   1:11.0.1.13-3.el7_6                                         updates
java-11-openjdk-jmods-debug.i686                                               1:11.0.1.13-3.el7_6                                         updates
java-11-openjdk-jmods-debug.x86_64                                             1:11.0.1.13-3.el7_6                                         updates
java-11-openjdk-src.i686                                                       1:11.0.1.13-3.el7_6                                         updates
java-11-openjdk-src.x86_64                                                     1:11.0.1.13-3.el7_6                                         updates
java-11-openjdk-src-debug.i686                                                 1:11.0.1.13-3.el7_6                                         updates
java-11-openjdk-src-debug.x86_64                                               1:11.0.1.13-3.el7_6                                         updates
java-atk-wrapper.i686                                                          0.30.4-5.el7                                                base   
java-atk-wrapper.x86_64                                                        0.30.4-5.el7                                                base  

How can the (old) centos:7.5.1804 which according to https://hub.docker.com/r/library/centos/tags/ has not changed since 2 months ago not contain java-1.8.0-openjdk-1.8.0.181-3.b13.el7_5 anymore, which used to still work 1 week ago, as proven here https://circleci.com/gh/fabric8io-images/s2i/496, and only contain 1.8.0.191.b12-1.el7_6, and more fun also have 11.0.1.13-3.el7_6 which should be only in 7.6?

Something is really off here.

vorburger commented 5 years ago

Just to make sure this isn't some local screw-up, even though it clearly does happen on Circle CI as well, I've just done a docker rm $(docker ps -a -q) ; docker rmi $(docker images -q) but the result is the same as above.

TrevorPH commented 5 years ago

It's BECAUSE you are using fixed versions that you have the error. Don't. Install just the package name and you get the latest.

vorburger commented 5 years ago

b57b0df ditches Java 8 package minor version number which fixes this; https://circleci.com/gh/fabric8io-images/s2i/504 passed again.

8bad895 does the same for the karaf/ image (b57b0df just took care of java/ to unblock); https://circleci.com/gh/fabric8io-images/s2i/506 passed again.

172 has something somewhat related to this problem; basically this project may have misunderstood CentOS versions - for both packages and the images themselves: Full transcript of what I learnt today in this regard preserved on https://gist.github.com/vorburger/ea367a76f69aafb8260c62f6bedf2788, but the gist of it is basically that there is FROM centos:7.5.1804 since today's release "becomes" a 7.6 as soon as you do any dnf install (not just dnf update, but we should probably also do that) and specific minor package versions like java-1.8.0-openjdk-1.8.0.181-3.b13.el7_5 are not available on the repos anymore. Someone commented on private IRC pointing out that on http://vault.centos.org old package versions remain available, if "you modify the mirror paths in /etc/yum.repos.d/CentOS-Base.repo e.g. baseurl=http://vault.centos.org/7.5.1804/os/x86_64/ but I doubt we want to use that here in this project (I personally certainly don't).

@rhuss @ctron FYI