Add new Centos release (7.3.1611)

frivoire commented 7 years ago

Hello,

Currently, the latest version available in multiarch is 7.2. It would be very appreciated if you could build version 7.3 (released some months ago) too.

NB: I've tried a little to prepare some patchs for a PR, but I've had difficulties to run successfully update.sh, and couldn't make it myself.

During the building, could you check that aarch64 has its variant -clean in dockerhub (as there is x86_64 and armhf). This is the variant that we used for scaleway/image-centos, and it's currently missing (no tag 7.2.1511-aarch64-clean).

Thanks a lot for help !

moul commented 7 years ago

Do you particularly want the 1611 ? I can find a more recent one:

frivoire commented 7 years ago

Not particularly the 1611, I was just refering to 7.3 but got confused with Centos naming. Sorry.

moul commented 7 years ago

I successfully pushed some architectures but I still have some issues for armhf and aarch64, maybe you can give a try using the branch in the PR #3

Maybe it comes from my current environment

frivoire commented 7 years ago

I made several tests and I discovered the following elements :

1) Recent change in file naming of qemu-user-static releases :

flo@flopc:~$ wget -q https://github.com/multiarch/qemu-user-static/releases/download/v2.8.0/x86_64_qemu-aarch64-static.tar.gz -O - |tar t
qemu-aarch64-static
flo@flopc:~$ wget -q https://github.com/multiarch/qemu-user-static/releases/download/v2.9.0/x86_64_qemu-aarch64-static.tar.gz -O - |tar t
qemu-aarch64

This is important, because register.sh (binfmt_misc) uses path : '${QEMU_BIN_DIR}'/qemu-aarch64-static:'

2) Invalid URL in update.sh :

    mkdir -p iso
    if [ -n "${qemu_arch}" -a ! -f "iso/qemu-${qemu_arch}-static" ]; then
       wget https://github.com/multiarch/qemu-user-static/releases/download/v2.9.0/x86_64_qemu-${qemu_arch}-static -O "iso/qemu-${qemu_arch}-static"
    fi

==> missing ".tar.gz" at the end of url + local-filename ==> missing a "tar xf" command after the wget

3) Previous errors have been commited (in current repo) and should be cleaned :

git checkout dev/moul/bump-7.3
git rm 7.3.1611-aarch64/iso/qemu-aarch64-static
git ci -m "Remove empty file (should not have been committed, because it's downloaded by update.sh)"

moul commented 7 years ago

1) Recent change in file naming of qemu-user-static releases :

Already done here: https://github.com/multiarch/centos/pull/3/files#diff-25e6ad8ebc5d26ba06d2bcb0b220149cR91

2) Invalid URL in update.sh :

Not a problem, I updated the download URL in the update.sh file and the ADD command in the Dockerfile accordingly

3) Previous errors have been commited (in current repo) and should be cleaned :

Yes, I forgot to update the .gitignore file

moul commented 7 years ago

Previous errors have been commited (in current repo) and should be cleaned :

Removed, added .gitignore, force-pushed

frivoire commented 7 years ago

1 . and 2. I still think there is an issue : In your commit : https://github.com/multiarch/centos/pull/3/files#diff-25e6ad8ebc5d26ba06d2bcb0b220149cR91 there is the line : wget https://github.com/multiarch/qemu-user-static/releases/download/v2.9.0/x86_64_qemu-${qemu_arch}-static -O "iso/qemu-${qemu_arch}-static"

But, when I try : qemu_arch=aarch64; wget https://github.com/multiarch/qemu-user-static/releases/download/v2.9.0/x86_64_qemu-${qemu_arch}-static ==> 404 You should change the URL to qemu_arch=aarch64; wget https://github.com/multiarch/qemu-user-static/releases/download/v2.9.0/qemu-${qemu_arch} ==> 200

Can you re-check on your side please ? NB: I now see that tar xf is not needed (but you still need to fix the URL as I shown just above)

OK for me :)

frivoire commented 7 years ago

Another element that I just found :

4) qemu-user-static is not static anymore :

flo@flopc:~$ wget -q https://github.com/multiarch/qemu-user-static/releases/download/v2.8.4/qemu-aarch64-static -O - |file -
/dev/stdin: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), statically linked, for GNU/Linux 2.6.32, BuildID[sha1]=91fb924228a0814f7ae7d13947cb1f13c02fa36f, stripped

flo@flopc:~$ wget -q https://github.com/multiarch/qemu-user-static/releases/download/v2.9.0/qemu-aarch64 -O - |file -
/dev/stdin: ELF 64-bit LSB shared object, x86-64, version 1 (GNU/Linux), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.32, BuildID[sha1]=48894fc6b031f7dfecf8ac90b6f420dd8d94b34f, stripped

.

So, to fix 1. and 2. and 4. (which are all related), I did that :

diff --git a/update.sh b/update.sh
index 0d2929c..8e54e21 100755
--- a/update.sh
+++ b/update.sh
@@ -88,7 +88,8 @@ EOF
     # create iso dockerfile
     mkdir -p iso
     if [ -n "${qemu_arch}" -a ! -f "iso/qemu-${qemu_arch}-static" ]; then
-       wget https://github.com/multiarch/qemu-user-static/releases/download/v2.9.0/x86_64_qemu-${qemu_arch}-static -O "iso/qemu-${qemu_arch}-static"
+       wget https://github.com/multiarch/qemu-user-static/releases/download/v2.8.4/qemu-${qemu_arch}-static -O "iso/qemu-${qemu_arch}-static"
+       chmod +x "iso/qemu-${qemu_arch}-static"
     fi
     if [ -n "${qemu_arch}" ]; then
        cat > iso/Dockerfile <<EOF

And with that, I manage to have the building process (update.sh) to go much further, but not to the end. I'm still working on it.

frivoire commented 7 years ago

I cleaned everything and relaunch the build (with my patch of update.sh, cf previous comment) of 7.3.1611-aarch64 : ==> it is successful !

frivoire commented 7 years ago

Here are some packages that you should add in update.sh to be yum removed (when writting the iso-clean/Dockerfile) : => perl gcc cpp doxygen emacs-nox

They were not present in previous images (7.2-xxx-clean and 7.3-x86_64-clean). So removing them seems logic, helping to keep an image "not too large".

moul commented 7 years ago

Can you open a PR that continues mine?

On 1 Jun 2017, 15:09 +0200, Florent Rivoire notifications@github.com, wrote:

Here are some packages that you should add in update.sh to be yum removed (when writting the iso-clean/Dockerfile) : => perl gcc cpp doxygen emacs-nox They were not present in previous images (7.2-xxx-clean and 7.3-x86_64-clean). So removing them seems logic, helping to keep an image "not too large". — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub, or mute the thread.

frivoire commented 7 years ago

Sorry, I miss the notification for your reply last week :( I just created PR #4.

frivoire commented 7 years ago

Thanks for merging my PR. With this fix, can you build successfully Centos 7.3 for all archs on your machine ? If yes, can you publish them on dockerhub ?

By the way, how do you validate that an image is "ok" to be published ? Do you have some tests that you run ?

On my side, the only thing I've tested : ./update.sh ==> build ok docker run -ti --rm multiarch/centos:7.3.1611-aarch64-clean bash ==> ok [some simple commands : top, uname -a, less, yum install vim] ==> everything seems ok

moul commented 7 years ago

Done :)

frivoire commented 7 years ago

I've seen the new images on : https://hub.docker.com/r/multiarch/centos/tags/ Thanks !

But the 7.3.1611-aarch64-clean (clean, the variant that we used for scaleway/image-centos) seems still missing. Can you re-check on your side please ?

frivoire commented 7 years ago

Sorry to ask again, but did you have a little time to check the -clean tag of aarch64 ? Thanks in advance !

moul commented 7 years ago

Yep, I just pushed a fix, can you retry?

frivoire commented 7 years ago

There is still an issue somewhere :

flo@flopc:~$ docker images |grep aarch64
multiarch/centos                                7.3.1611-aarch64-clean   2bbb01f2cce5        55 minutes ago      0 B
flo@flopc:~$ docker run -ti --rm multiarch/centos:7.3.1611-aarch64-clean bash
docker: Error response from daemon: oci runtime error: container_linux.go:247: starting container process caused "exec: \"bash\": executable file not found in $PATH".

==> cf the size "0B" for the image

.

So, I've tested with your latest patch (pulling master, so last commit included is : https://github.com/multiarch/centos/commit/4a5d4ac81aa143025edac8155e40c5a7ccb6185d) :

docker rmi -f $(docker images -q)
git clean -fx
git reset --hard
./update.sh 7.3.1611-aarch64

After a few minutes :

[... logs of "yum remove list-of-packages" that is ok]
Complete!
 ---> 4a7ebd5e72df
Removing intermediate container 99385f2da7b7
Step 3 : RUN rm -rf /boot
 ---> Running in 2365255aa3a2
 ---> 4db4816ec9c0
Removing intermediate container 2365255aa3a2
Successfully built 4db4816ec9c0
Error response from daemon: No such container: export-7ZSugjB+x1u6tA
sha256:465af938b23a8d7db6e4c16bdb92c3e4487fb96a98f6cf28292b25e74b4195de
Error response from daemon: No such container: export-7ZSugjB+x1u6tA

$ docker images -a
REPOSITORY             TAG                            IMAGE ID            CREATED             SIZE
multiarch/centos       7.3.1611-aarch64-clean         465af938b23a        21 seconds ago      0 B
tmp-multiarch/centos   7.3.1611-aarch64-iso-cleaner   4db4816ec9c0        21 seconds ago      1.402 GB
<none>                 <none>                         4a7ebd5e72df        23 seconds ago      1.402 GB
multiarch/centos       7-aarch64-iso                  8763541e4a18        2 minutes ago       1.369 GB
multiarch/centos       7.3-aarch64-iso                8763541e4a18        2 minutes ago       1.369 GB
multiarch/centos       7.3.1611-aarch64-iso           8763541e4a18        2 minutes ago       1.369 GB
multiarch/centos       aarch64-iso                    8763541e4a18        2 minutes ago       1.369 GB
multiarch/centos       7-aarch64-iso-slim             c54912f761fc        2 minutes ago       1.366 GB
multiarch/centos       7.3-aarch64-iso-slim           c54912f761fc        2 minutes ago       1.366 GB
multiarch/centos       7.3.1611-aarch64-iso-slim      c54912f761fc        2 minutes ago       1.366 GB
multiarch/centos       aarch64-iso-slim               c54912f761fc        2 minutes ago       1.366 GB
<none>                 <none>                         c0d96b2e7e23        2 minutes ago       1.366 GB

==> I'm guessing there are issue with the "+" character that has been randomly generated in update.sh.

So, I've created https://github.com/multiarch/centos/pull/5 with a simple fix. If ok for you, can you rebuild on your side and publish everything (and especially the 7.3.1611-aarch64-clean) on dockerhub.

Thanks :)

moul commented 7 years ago

$ docker images | grep multiarch/centos
multiarch/centos                                 7-aarch64-clean                 3ed1835dbbcc        37 minutes ago      684.6 MB
multiarch/centos                                 7.3-aarch64-clean               3ed1835dbbcc        37 minutes ago      684.6 MB
multiarch/centos                                 7.3.1611-aarch64-clean          3ed1835dbbcc        37 minutes ago      684.6 MB
multiarch/centos                                 aarch64-clean                   3ed1835dbbcc        37 minutes ago      684.6 MB

Can you retry?

frivoire commented 7 years ago

aarch64 seems good for me (after very quick testing). Thanks !!

I've also looked at dockerhub and some other archs seem borken too : "empty" image (0 byte or almost 0 byte) :

7.3.1611-x86_64-clean ==> probably not a big impact, since 7.3.1704 is there and ok
*-ppc64-* / *-ppc64le-* ==> we dont use them for scw, so not an issue for us

frivoire commented 7 years ago

@moul : you should check your building server, your are still probably building in a "while true" all 7.3 images. Cf the last updated column of https://hub.docker.com/r/multiarch/centos/tags/ that seems to be always very recent (< 15min), the 3 different times I checked today.

frivoire commented 7 years ago

I've done some more testing (integrating into scw) and found a missing config. So : https://github.com/multiarch/centos/pull/6

Thanks in advance for you help again :)

moul commented 7 years ago

@moul : you should check your building server, your are still probably building in a "while true" all 7.3 images. Cf the last updated column of https://hub.docker.com/r/multiarch/centos/tags/ that seems to be always very recent (< 15min), the 3 different times I checked today.

Loop closed

I've done some more testing (integrating into scw) and found a missing config. So : #6

Merged

I will rebuild the aarch64 image right now for scw and look for the 3 missing images a little bit later

frivoire commented 7 years ago

Thanks ! Aarch64 tag is ok now.

frivoire commented 7 years ago

I've just tested the armhf release, and there is an issue :

flo@flopc:~$ docker run -ti --rm multiarch/centos:7.3.1611-armhfp-clean bash
Unable to find image 'multiarch/centos:7.3.1611-armhfp-clean' locally
7.3.1611-armhfp-clean: Pulling from multiarch/centos
7556d73f071f: Pull complete
Digest: sha256:51a82f44429a150592be0ff7a64a85df404f12a4bee43295898fc5f0c38431a4
Status: Downloaded newer image for multiarch/centos:7.3.1611-armhfp-clean

[root@1de8111df0bd /]# id
uid=0(root) gid=0(root) groups=0(root)

[root@1de8111df0bd /]# uname -a
Linux 1de8111df0bd 4.9.27-moby #1 SMP Thu May 11 04:01:18 UTC 2017 armv7l armv7l armv7l GNU/Linux

[root@1de8111df0bd /]# yum update
qemu: uncaught target signal 11 (Segmentation fault) - core dumped
Segmentation fault

==> freshly-downloaded image does segfaults for some commands (but other commands work)

I have no explanation for now, I'll look into it soon.

moul commented 7 years ago

Last time it happen was because of qemu incompatibility for some syscalls on some archs

Can you give a try on real hardware?

On 22 Jun 2017, 14:27 +0200, Florent Rivoire notifications@github.com, wrote:

I've just tested the armhf release, and there is an issue :

flo@flopc:~$ docker run -ti --rm multiarch/centos:7.3.1611-armhfp-clean bash Unable to find image 'multiarch/centos:7.3.1611-armhfp-clean' locally 7.3.1611-armhfp-clean: Pulling from multiarch/centos 7556d73f071f: Pull complete Digest: sha256:51a82f44429a150592be0ff7a64a85df404f12a4bee43295898fc5f0c38431a4 Status: Downloaded newer image for multiarch/centos:7.3.1611-armhfp-clean

[root@1de8111df0bd /]# id uid=0(root) gid=0(root) groups=0(root)

[root@1de8111df0bd /]# uname -a Linux 1de8111df0bd 4.9.27-moby #1 SMP Thu May 11 04:01:18 UTC 2017 armv7l armv7l armv7l GNU/Linux

[root@1de8111df0bd /]# yum update qemu: uncaught target signal 11 (Segmentation fault) - core dumped Segmentation fault

==> freshly-downloaded image does segfaults for some commands (but other commands work) I have no explanation for now, I'll look into it soon. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

multiarch / centos

Add new Centos release (7.3.1611) #2