Proper multi-architecture Dockerfiles

[tianon]

The Official Images finally have official multi-architecture support (see https://github.com/docker-library/official-images#architectures-other-than-amd64 and https://github.com/docker-library/official-images#multiple-architectures for details). It looks like the hard work to support this for ibmjava has already been done (figuring out where the upstream multi-arch bits live), and all that's really left is to combine all that into a single Dockerfile that transparently handles multi-arch and then to add Architectures entries to library/ibmjava. :+1:

I didn't want to send a PR to re-organize this whole repo to remove the intermediate architecture-specific folders without discussing it in an issue first. :heart:

So, for a Dockerfile like https://github.com/ibmruntimes/ci.docker/blob/9b62a599bd4bcdba1983c71880dff055ef5a488b/ibmjava/8/jre/x86_64/ubuntu/Dockerfile, here's how I'd propose that changes:

FROM ubuntu:16.04

RUN apt-get update \
    && apt-get install -y --no-install-recommends wget ca-certificates \
    && rm -rf /var/lib/apt/lists/*

ENV JAVA_VERSION 1.8.0_sr4fp10

RUN set -ex; \
    dpkgArch="$(dpkg --print-architecture)"; \
    case "${dpkgArch##*-}" in \
        amd64) \
            ESUM="18b756d0bd81a605a7a0ac722e36e3321671478b11a1bb685c78d5b59164074c"; \
            YML_FILE="jre/linux/x86_64/index.yml"; \
            ;; \
        i386) \
            ESUM="8197212d492c0b76a3e01805a61efab6d5c45ff248e045a3742d51e3ac402902"; \
            YML_FILE="jre/linux/i386/index.yml"; \
            ;; \
        ppc64el) \
            ESUM="fb87faf943c9f51df4cf48fcc537c045c96507f5f2abc812500882e9ba1e94d8"; \
            YML_FILE="jre/linux/ppc64le/index.yml"; \
            ;; \
        s390) \
            ESUM="5cb294b038d16098f0982a096104b42e761e3cb09731bff954b6141061e60ef1"; \
            YML_FILE="jre/linux/s390/index.yml"; \
            ;; \
        s390x) \
            ESUM="996d9f51c858d4cba4720a88493263d2a281f24a826c4714d2c92fcff9dfd7d9"; \
            YML_FILE="jre/linux/s390x/index.yml"; \
            ;; \
    esac; \
    BASE_URL="https://public.dhe.ibm.com/ibmdl/export/pub/systems/cloud/runtimes/java/meta/"; \
    wget -q -U UA_IBM_JAVA_Docker -O /tmp/index.yml "$BASE_URL/$YML_FILE"; \
    JAVA_URL=$(cat /tmp/index.yml | sed -n '/'$JAVA_VERSION'/{n;p}' | sed -n 's/\s*uri:\s//p' | tr -d '\r'); \
    wget -q -U UA_IBM_JAVA_Docker -O /tmp/ibm-java.bin "$JAVA_URL"; \
    echo "$ESUM */tmp/ibm-java.bin" | sha256sum -c -; \
    echo "INSTALLER_UI=silent" > /tmp/response.properties; \
    echo "USER_INSTALL_DIR=/opt/ibm/java" >> /tmp/response.properties; \
    echo "LICENSE_ACCEPTED=TRUE" >> /tmp/response.properties; \
    mkdir -p /opt/ibm; \
    chmod +x /tmp/ibm-java.bin; \
    /tmp/ibm-java.bin -i silent -f /tmp/response.properties; \
    rm -f /tmp/response.properties; \
    rm -f /tmp/index.yml; \
    rm -f /tmp/ibm-java.bin

ENV JAVA_HOME=/opt/ibm/java/jre \
    PATH=/opt/ibm/java/jre/bin:$PATH

@@ -7,8 +7,29 @@
 ENV JAVA_VERSION 1.8.0_sr4fp10

 RUN set -ex; \
-    ESUM="18b756d0bd81a605a7a0ac722e36e3321671478b11a1bb685c78d5b59164074c"; \
-    YML_FILE="jre/linux/x86_64/index.yml"; \
+    dpkgArch="$(dpkg --print-architecture)"; \
+    case "${dpkgArch##*-}" in \
+        amd64) \
+            ESUM="18b756d0bd81a605a7a0ac722e36e3321671478b11a1bb685c78d5b59164074c"; \
+            YML_FILE="jre/linux/x86_64/index.yml"; \
+            ;; \
+        i386) \
+            ESUM="8197212d492c0b76a3e01805a61efab6d5c45ff248e045a3742d51e3ac402902"; \
+            YML_FILE="jre/linux/i386/index.yml"; \
+            ;; \
+        ppc64el) \
+            ESUM="fb87faf943c9f51df4cf48fcc537c045c96507f5f2abc812500882e9ba1e94d8"; \
+            YML_FILE="jre/linux/ppc64le/index.yml"; \
+            ;; \
+        s390) \
+            ESUM="5cb294b038d16098f0982a096104b42e761e3cb09731bff954b6141061e60ef1"; \
+            YML_FILE="jre/linux/s390/index.yml"; \
+            ;; \
+        s390x) \
+            ESUM="996d9f51c858d4cba4720a88493263d2a281f24a826c4714d2c92fcff9dfd7d9"; \
+            YML_FILE="jre/linux/s390x/index.yml"; \
+            ;; \
+    esac; \
     BASE_URL="https://public.dhe.ibm.com/ibmdl/export/pub/systems/cloud/runtimes/java/meta/"; \
     wget -q -U UA_IBM_JAVA_Docker -O /tmp/index.yml "$BASE_URL/$YML_FILE"; \
     JAVA_URL=$(cat /tmp/index.yml | sed -n '/'$JAVA_VERSION'/{n;p}' | sed -n 's/\s*uri:\s//p' | tr -d '\r'); \

See golang for another good example of this pattern (https://github.com/docker-library/golang/blob/f34c645c8402bd8f6b70f530545ad7845dfefbcc/1.9/stretch/Dockerfile#L16-L27).

I'm more than happy to make a PR to implement this (where we can discuss details of the actual implementation further, and iterate on it until you're happy with it), but again, it's a big change, so I figured it's probably worth discussing before I do a huge PR dump. :smile:

The primary benefit of doing this is that once https://github.com/docker-library/official-images/issues/2289 is resolved (which should hopefully be soon!), users of this image will be able to just use ibmjava on every architecture and it'll do-the-right-thing appropriately out-of-the-box. :+1:

If you want more examples of multi-architecture official images, see any of the following:

• buildpack-deps
• busybox
• cassandra
• debian
• drupal
• fedora
• gcc
• ghost
• golang
• haproxy
• hello-seattle
• hello-world
• hola-mundo
• httpd
• hylang
• irssi
• julia
• memcached
• mono
• nextcloud
• node
• openjdk
• owncloud
• perl
• php
• postgres
• pypy
• python
• rabbitmq
• redis
• redmine
• ros
• ruby
• rust
• spiped
• tomcat
• ubuntu
• wordpress

[dinogun]

Thanks @tianon for reaching out, I look forward to add multi-arch support to ibmjava . Will work on it later this week. Expecting to have lots of questions for you as well !

[tianon]

Going to also CC the websphere-liberty image maintainers (@davidcurrie @jamiecoleman92 @dibbles) here so they can track the status of this, because once this is implemented, we're going to want to update that image too (which will hopefully be really minimal changes :muscle:)! :smile: :heart:

[davidcurrie]

Catching up post-vacation. Thanks for the heads up @tianon. I suspect at your behest, @duglin has been making noises about moving IBM built images out of ppc64le and s390x. It would be good to get the websphere-liberty images there replaced by Docker built images before that clearout takes place.

[dinogun]

Hi @tianon, here is a first cut of the multi-arch support for ibmjava, let me know if this works for you. Dockerfile and script changes are as below (alpine+ibmjava currently only works on amd64, please ignore the alpine Dockerfiles for other arches for now) https://github.com/ibmruntimes/ci.docker/pull/12 Updates to the library (I havent raised a PR yet) https://github.com/docker-library/official-images/compare/master...dinogun:ibmjava-multi-arch

Also as promised a few questions ! Thanks in advance !

1. s390 is not supported as an arch right (only s390x)
2. Is there any specific docker version that users need to be on for this to work ?
3. If I am maintaining a local docker registry, what do I need to do to get this to work.
4. Doing a docker pull arch/ibmjava will continue to work ?

[tianon]

Commented on #12 -- looks great!

Your library diff looks fine too; I'd note that it is valid to put GitCommit up at the top if you'd like to deduplicate that (we usually only intersperse it because we generate the files so we like to put the exact latest commit of the Dockerfile in each entry's section).

1. s390 isn't well-supported anywhere these days, is it? Looks like even Debian removed support for it back in Wheezy (when s390x was added), and Go definitely doesn't officially support it.
2. manifest list support was added initially in Docker 1.10.0 (2016-02-04), but I believe some registries will only respond with amd64 if the client does not support manifest lists (not sure if that's changed)
3. in a local Docker registry, you'll either need to use @estesp's manifest-tool directly, or use our bashbrew tool which wraps it (via bashbrew put-shared ...) -- you have to push each arch image somewhere, then combine them into a manifest list afterwards
4. yes, docker pull s390x/ibmjava will continue to work, and will be built from the same sources as amd64/ibmjava, etc (and will be what then gets combined into a manifest list to create /_/ibmjava) :+1:

[utzb]

Right, s390 is how it all started in 31-bit mode, almost 20 years ago. In some places it still stands for general platform support, and some old 31 bit applications are still running today on 64bit kernels. However, the current and only relevant mode of operation is s390x = 64 bit; no need to consider 31bit=s390.

[dinogun]

I see it is working now for both s390x and ppc64le, yay !

$ docker run --rm -it ibmjava:8-sfj java -version
Unable to find image 'ibmjava:8-sfj' locally
8-sfj: Pulling from library/ibmjava
c58ca33e0c24: Already exists
fc3fd4e62755: Already exists
0c7fa8f00fe2: Already exists
1618266695ce: Already exists
7c037ccce0cb: Already exists
2c3f4a21f2a7: Already exists
9d2c11061bd0: Pull complete
Digest: sha256:bcf5667a2accda4e70b797b03de41c8b3d7b5494010df641d8159cc8b971e34f
Status: Downloaded newer image for ibmjava:8-sfj
java version "1.8.0"
Java(TM) SE Runtime Environment (build pxz6480sr4fp11-20170823_01(SR4 FP11) Small Footprint)
IBM J9 VM (build 2.8, JRE 1.8.0 Linux s390x-64 Compressed References 20170812_360168 (JIT enabled, AOT enabled)
J9VM - R28_20170812_0201_B360168
JIT  - tr.r14.java_20170812_360168
GC   - R28_20170812_0201_B360168_CMPRSS
J9CL - 20170812_360168)
JCL - 20170726_01 based on Oracle jdk8u144-b01

$ docker run --rm -it ibmjava:8-jre java -version
Unable to find image 'ibmjava:8-jre' locally
8-jre: Pulling from library/ibmjava
0ad9ca03f1b2: Pull complete
7d2491df9494: Pull complete
9a961cbb08c0: Pull complete
9f38c9e24bdd: Pull complete
36070ab6f935: Pull complete
e44e9115c675: Pull complete
a19364f87895: Pull complete
Digest: sha256:5a63b4657da717067c9f11f7adebc606eacb6449d32ec81ec48265896cfbeb26
Status: Downloaded newer image for ibmjava:8-jre
java version "1.8.0"
Java(TM) SE Runtime Environment (build pxl6480sr4fp11-20170823_01(SR4 FP11))
IBM J9 VM (build 2.8, JRE 1.8.0 Linux ppc64le-64 Compressed References 20170812_360168 (JIT enabled, AOT enabled)
J9VM - R28_20170812_0201_B360168
JIT  - tr.r14.java_20170812_360168
GC   - R28_20170812_0201_B360168_CMPRSS
J9CL - 20170812_360168)
JCL - 20170726_01 based on Oracle jdk8u144-b01