Alpine armhf on x86_64 Ubuntu via qemu returns 127

[ghost]

I am not sure if it is supported, but README suggests that the script uses qemu so I tried and ended up getting 4 errors:

# in ubuntu artful x86_64

# download
wget https://raw.githubusercontent.com/alpinelinux/alpine-chroot-install/v0.7.0/alpine-chroot-install     && echo '090d323d887ef3a2fd4e752428553f22a52b87bb  alpine-chroot-install' | sha1sum -c
# set permissions
sudo chmod 777 alpine-chroot-install

sudo ./alpine-chroot-install -a armhf -b v3.7 -m http://dl-cdn.alpinelinux.org/alpine

> Installing and enabling binfmt-support on host system...
Reading package lists... Done
Building dependency tree
Reading state information... Done
binfmt-support is already the newest version (2.1.8-1).
0 upgraded, 0 newly installed, 0 to remove and 88 not upgraded.
'/usr/bin/qemu-arm-static' -> 'usr/bin/qemu-arm-static'

> Downloading static apk-tools
2018-04-25 08:15:10 URL:https://github-production-release-asset-2e65be.s3.amazonaws.com/56234603/7fed7126-f51c-11e7-9a23-c622dbccf101?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20180425%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20180425T051508Z&X-Amz-Expires=300&X-Amz-Signature=4ddefa9f0a0304527d481b59b8791d94b08b9906a6dd1506796810b22db67b06&X-Amz-SignedHeaders=host&actor_id=0&response-content-disposition=attachment%3B%20filename%3Dapk-tools-2.8.2-x86_64-linux.tar.gz&response-content-type=application%2Foctet-stream [814198/814198] -> "apk-tools-2.8.2-x86_64-linux.tar.gz" [1]
apk-tools-2.8.2-x86_64-linux.tar.gz: OK

> Downloading APK keys
2018-04-25 08:15:11 URL:https://alpinelinux.org/keys/alpine-devel@lists.alpinelinux.org-4a6a0840.rsa.pub [451/451] -> "alpine-devel@lists.alpinelinux.org-4a6a0840.rsa.pub" [1]
alpine-devel@lists.alpinelinux.org-4a6a0840.rsa.pub: OK
2018-04-25 08:15:11 URL:https://alpinelinux.org/keys/alpine-devel@lists.alpinelinux.org-5243ef4b.rsa.pub [451/451] -> "alpine-devel@lists.alpinelinux.org-5243ef4b.rsa.pub" [1]
alpine-devel@lists.alpinelinux.org-5243ef4b.rsa.pub: OK
2018-04-25 08:15:11 URL:https://alpinelinux.org/keys/alpine-devel@lists.alpinelinux.org-524d27bb.rsa.pub [451/451] -> "alpine-devel@lists.alpinelinux.org-524d27bb.rsa.pub" [1]
alpine-devel@lists.alpinelinux.org-524d27bb.rsa.pub: OK
2018-04-25 08:15:12 URL:https://alpinelinux.org/keys/alpine-devel@lists.alpinelinux.org-5261cecb.rsa.pub [451/451] -> "alpine-devel@lists.alpinelinux.org-5261cecb.rsa.pub" [1]
alpine-devel@lists.alpinelinux.org-5261cecb.rsa.pub: OK
2018-04-25 08:15:12 URL:https://alpinelinux.org/keys/alpine-devel@lists.alpinelinux.org-58199dcc.rsa.pub [451/451] -> "alpine-devel@lists.alpinelinux.org-58199dcc.rsa.pub" [1]
alpine-devel@lists.alpinelinux.org-58199dcc.rsa.pub: OK
2018-04-25 08:15:12 URL:https://alpinelinux.org/keys/alpine-devel@lists.alpinelinux.org-58cbb476.rsa.pub [451/451] -> "alpine-devel@lists.alpinelinux.org-58cbb476.rsa.pub" [1]
alpine-devel@lists.alpinelinux.org-58cbb476.rsa.pub: OK
2018-04-25 08:15:12 URL:https://alpinelinux.org/keys/alpine-devel@lists.alpinelinux.org-58e4f17d.rsa.pub [451/451] -> "alpine-devel@lists.alpinelinux.org-58e4f17d.rsa.pub" [1]
alpine-devel@lists.alpinelinux.org-58e4f17d.rsa.pub: OK

> Installing Alpine Linux v3.7 (armhf) into chroot
fetch http://dl-cdn.alpinelinux.org/alpine/v3.7/main/armhf/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.7/community/armhf/APKINDEX.tar.gz
(1/16) Installing musl (1.1.18-r3)
(2/16) Installing busybox (1.27.2-r8)
Executing busybox-1.27.2-r8.post-install
ERROR: busybox-1.27.2-r8.post-install: script exited with error 127
(3/16) Installing alpine-baselayout (3.0.5-r2)
Executing alpine-baselayout-3.0.5-r2.pre-install
ERROR: alpine-baselayout-3.0.5-r2.pre-install: script exited with error 127
Executing alpine-baselayout-3.0.5-r2.post-install
ERROR: alpine-baselayout-3.0.5-r2.post-install: script exited with error 127
(4/16) Installing openrc (0.24.1-r4)
Executing openrc-0.24.1-r4.post-install
ERROR: openrc-0.24.1-r4.post-install: script exited with error 127
(5/16) Installing alpine-conf (3.7.0-r1)
(6/16) Installing libressl2.6-libcrypto (2.6.3-r0)
(7/16) Installing libressl2.6-libssl (2.6.3-r0)
(8/16) Installing zlib (1.2.11-r1)
(9/16) Installing apk-tools (2.9.1-r2)
(10/16) Installing busybox-suid (1.27.2-r8)
(11/16) Installing busybox-initscripts (3.1-r2)
Executing busybox-initscripts-3.1-r2.post-install
ERROR: busybox-initscripts-3.1-r2.post-install: script exited with error 127
(12/16) Installing scanelf (1.2.2-r1)
(13/16) Installing musl-utils (1.1.18-r3)
(14/16) Installing libc-utils (0.7.1-r0)
(15/16) Installing alpine-keys (2.1-r1)
(16/16) Installing alpine-base (3.7.0-r0)
Executing busybox-1.27.2-r8.trigger
ERROR: busybox-1.27.2-r8.trigger: script exited with error 127
4 errors; 5 MiB in 16 packages

Can you point me what I am doing wrong and how to fix it?

Cheers!

[ghost]

qemu binfmt emulation failed here. Can you check that qemu is properly registered at /proc/sys/fs/binfmt_misc after the script has run?

[ghost]

@nero, I have just created a brand new VM, this time Ubuntu 16 LTS x64, and ran the following:

# first time ssh'd into brand-new install
uname -a
# Linux ubuntu 4.4.0-116-generic #140-Ubuntu SMP Mon Feb 12 21:23:04 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

cat /etc/*-release
# DISTRIB_ID=Ubuntu
# DISTRIB_RELEASE=16.04
# DISTRIB_CODENAME=xenial
# DISTRIB_DESCRIPTION="Ubuntu 16.04.4 LTS"
# NAME="Ubuntu"
# VERSION="16.04.4 LTS (Xenial Xerus)"
# ID=ubuntu
# ID_LIKE=debian
# PRETTY_NAME="Ubuntu 16.04.4 LTS"
# VERSION_ID="16.04"
# HOME_URL="http://www.ubuntu.com/"
# SUPPORT_URL="http://help.ubuntu.com/"
# BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
# VERSION_CODENAME=xenial
# UBUNTU_CODENAME=xenial

sudo apt update
sudo apt install -y binfmt-support qemu-user < /dev/null

wget https://raw.githubusercontent.com/alpinelinux/alpine-chroot-install/v0.7.0/alpine-chroot-install     && echo '090d323d887ef3a2fd4e752428553f22a52b87bb  alpine-chroot-install' | sha1sum -c
sudo chmod 777 alpine-chroot-install
sudo ./alpine-chroot-install -a armhf -b v3.7 -m http://dl-cdn.alpinelinux.org/alpine

#!! failed with same error

cat /proc/sys/fs/binfmt_misc/status
# enabled

[ghost]

The script setups an additional repo source from ubuntu artful, and installing qemu-user from that. This means manually installing qemu isn't necessary.

The newer qemu then doens't come with proper binfmt support. Could you try without manually installing qemu and using this patch?

--- alpine-chroot-install
+++ alpine-chroot-install
@@ -232,7 +232,7 @@
                target_rel="--target-release $QEMU_UBUNTU_REL"
        fi

-       apt_install $target_rel qemu-user-static \
+       apt_install $target_rel qemu-user-static binfmt-support \
                || die 'Failed to install qemu-user-static using apt-get!'
 }

[ghost]

@nero, you are right installing binfmt-support was extra step, I was being overefficient there :) The problem was that I was running Ubuntu in a VM on Windows 10 Hyper-V and enabled the nested VM support incorrectly. I have just tested with Ubuntu:artful in docker on same machine, with --privileged flag for docker-in-docker, and it doesn't fail with 127 anymore with or without applying the patch.

However, it is now failing on next step:

> Installing Alpine Linux v3.7 (armhf) into chroot
fetch http://dl-cdn.alpinelinux.org/alpine/v3.7/main/armhf/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.7/community/armhf/APKINDEX.tar.gz
(1/16) Installing musl (1.1.18-r3)
(2/16) Installing busybox (1.27.2-r8)
Executing busybox-1.27.2-r8.post-install
(3/16) Installing alpine-baselayout (3.0.5-r2)
Executing alpine-baselayout-3.0.5-r2.pre-install
Executing alpine-baselayout-3.0.5-r2.post-install
(4/16) Installing openrc (0.24.1-r4)
Executing openrc-0.24.1-r4.post-install
(5/16) Installing alpine-conf (3.7.0-r1)
(6/16) Installing libressl2.6-libcrypto (2.6.3-r0)
(7/16) Installing libressl2.6-libssl (2.6.3-r0)
(8/16) Installing zlib (1.2.11-r1)
(9/16) Installing apk-tools (2.9.1-r2)
(10/16) Installing busybox-suid (1.27.2-r8)
(11/16) Installing busybox-initscripts (3.1-r2)
Executing busybox-initscripts-3.1-r2.post-install
(12/16) Installing scanelf (1.2.2-r1)
(13/16) Installing musl-utils (1.1.18-r3)
(14/16) Installing libc-utils (0.7.1-r0)
(15/16) Installing alpine-keys (2.1-r1)
(16/16) Installing alpine-base (3.7.0-r0)
Executing busybox-1.27.2-r8.trigger
OK: 5 MiB in 16 packages

> Binding filesystems into chroot
mount: none mounted on /alpine/proc.
mount: /sys bound on /alpine/sys.
mount: /dev bound on /alpine/dev.
mount: /run bound on /alpine/run.
mount: / bound on /alpine.

> Setting up Alpine
mesg: ttyname failed: Success
sh: 2: apk: not found

and running /alpine/enter-chroot uname -a gives bash: /alpine/enter-chroot: No such file or directory (because script exited earlier before creating enter-chroot)

[ghost]

Ok, I have figured it out now. The issue is that if we don't provide BIND_DIR, it binds the root / of host in chroot and overwrite the /etc directory that has apk.

Passing mkdir /dummy and passing -i /dummy fixed the issue.

Here is the full docker and how to run it:

docker run -it --privileged ubuntu:bionic  # bionic will be next LTS in few days (on 4/29)

in docker:

apt update
apt install -y apt-utils wget < /dev/null
wget https://raw.githubusercontent.com/alpinelinux/alpine-chroot-install/v0.7.0/alpine-chroot-install     && echo '090d323d887ef3a2fd4e752428553f22a52b87bb  alpine-chroot-install' | sha1sum -c
chmod 777 alpine-chroot-install

# hack
mkdir /dummy

./alpine-chroot-install -a armhf -b v3.7 -m http://dl-cdn.alpinelinux.org/alpine -i /dummy

/alpine/enter-chroot uname -a

Should it be fixed by making BIND_DIR mounting optional (instead of flashing it with root /)?

[ghost]

#   -i BIND_DIR            Absolute path to the directory on the host system that
#                          should be mounted on the same path inside the chroot
#                          (default is PWD).

Other solution would be to have a cwd different from /.

[ghost]

What do you think about:

@@ -346,8 +346,14 @@ mount -v --rbind /sys sys
 mount -v --rbind /dev dev
 mount -v --rbind /run run

-mkdir -p "${CHROOT_DIR}${BIND_DIR}"
-mount -v --bind "$BIND_DIR" "${CHROOT_DIR}${BIND_DIR}"
+BIND_TARGET_DIR="${CHROOT_DIR}${BIND_DIR}"
+
+if diff "$BIND_DIR" "$BIND_TARGET_DIR" | grep "Common"; then
+  ewarn "Some directories will be overwritten in '$BIND_TARGET_DIR'"
+fi
+
+mkdir -p "$BIND_TARGET_DIR"
+mount -v --bind "$BIND_DIR" "$BIND_TARGET_DIR"

 einfo 'Setting up Alpine'

we can just warn the user or fail and if user specifies --force, only then continue in this case

[ghost]

An bind mount will always shadow the (non-bind-mounted) contents of the mountpoint. This is not an overlayfs, so the diff check is useless here - the contents of BIND_DIR will always be overwritten. This is the way bind mounts work.

[ghost]

The idea was to somehow identify if directories like bin etc are in BIND_DIR. If that is useless, then current defaults are reasonable. I personally think diff check has a potential of identifying accidental binds.

[ghost]

Other question: Why are you running that script in a docker container in the first place?

[ghost]

I was preparing the script for CI, and didn't had Ubuntu bare-metal handy; but that is irrelevant: cd /; ./alpine-chroot-install on bare-metal installation has same effect.

[ghost]

Why dont you use the alpine docker image then?

[ghost]

The final CI is not using docker, i was preparing the recipe script (thing that i ran "inside" the docker) on docker.

[jirutka]

@kasper3, @nero Thanks a lot for troubleshooting this issue!

Binding PWD to /$PWD inside chroot as default was really bad idea. It’s convenient on Travis, but dangerous in general. However, all existing .travis.yml using this script relies on this behaviour, so I cannot (shouldn’t) remove it completely. Thus I’ve just limited it to bind PWD by default only when PWD is under /home. Fixed version (0.8.0) is already released.

[ghost]

Thanks @jirutka! 👍

[Rich-Ireland-Bose]

I'm seeing a similar issue running on a Virtual Box instance of Mint 19 when attempting to build a chroot for aarch64. The some of the install scripts fail with "script exited with error 127" I've made sure that the proper qemu and binfmt bits are installed. I've given it a BIND_DIR value, there seems to be another root cause, but I'm too much of a newbie with alpine to see it.

[Rich-Ireland-Bose]

While testing to see if this would work without a specified architecture I ran into a small issue. It appears that if you aren't using a root directory for your BIND_DIR ("/something") then you end up with mashed directory names since the variables don't have a separator. This patch "fixes" this, but I suspect there's a better way:

diff --git a/alpine-chroot-install b/alpine-chroot-install
index 4c5c678..1969a44 100755
--- a/alpine-chroot-install
+++ b/alpine-chroot-install
@@ -371,9 +371,9 @@ if [ -L /dev/shm ] && [ -d /run/shm ]; then
 fi

 if [ "$BIND_DIR" ]; then
-       mkdir -p "${CHROOT_DIR}${BIND_DIR}"
-       mount -v --bind "$BIND_DIR" "${CHROOT_DIR}${BIND_DIR}"
-       mount --make-private "${CHROOT_DIR}${BIND_DIR}"
+       mkdir -p "${CHROOT_DIR}/${BIND_DIR}"
+       mount -v --bind "$BIND_DIR" "${CHROOT_DIR}/${BIND_DIR}"
+       mount --make-private "${CHROOT_DIR}/${BIND_DIR}"
 fi