`balena build` incorrectly reports ARM v7 or v8 instead of ARM v6

[pdcastro]

The balena build command is able to produce ARM v6 (rpi 1), ARM v7 (armv7hf) and ARM v8 (aarch64) images when base images of the respective architectures are selected, for example:

• FROM balenalib/raspberry-pi-alpine-node
• FROM balenalib/raspberrypi3-alpine-node
• FROM balenalib/raspberrypi4-64-alpine-node

But there is a limitation: the architecture "reported" to code executed in a Dockerfile RUN instruction is not always "correctly faked", if the code queries the current CPU architecture by running uname or similar:

• CLI running on an Intel machine with either QEMU emulation (e.g. balenaEngine on balenaOS on an Intel NUC device) or binfmt_misc (Docker Desktop for Windows or Mac), building ARM images:
  • Building for ARM v6: incorrectly reports ARM v7.
  • Building for ARM v7: correctly reports ARM v7.
  • Building for ARM v8: correctly reports ARM v8 (aarch64).
• CLI running on an ARM v8 (aarch64) machine, building ARM images (QEMU emulation not strictly required and not currently available):
  • Building for ARM v6: incorrectly reports ARM v8 (aarch64).
  • Building for ARM v7: incorrectly reports ARM v8 (aarch64).
  • Building for ARM v8: correctly reports ARM v8 (aarch64).

This only matters to code that dynamically queries the machine's architecture during the image build, for example Python's pip package manager that attempts to decide which pre-compiled executables to download and install on the image, or compilation tasks that decide the target architecture by dynamically querying the current CPU. @ab77 and @jakogut pointed out that the best solution for this is probably for the CLI to modify RUN instructions to set environment variables and inject a library that mocks the architecture to any code that queries it, for example:

RUN LD_PRELOAD=$(readlink -f libfakeuname.so) MACHINE_OVERRIDE=armv7l pip3 install https://www.piwheels.org/simple/pillow/Pillow-8.1.0-cp37-cp37m-linux_armv7l.whl

It should be possible to reuse the CLI's existing mechanism that copies QEMU to /tmp/ on the image during build time and already prefixes RUN instructions with the QEMU emulator:

• https://github.com/balena-io-modules/docker-qemu-transpose/
• https://github.com/balena-io/balena-cli/blob/v12.40.0/lib/utils/compose_ts.ts#L290
• https://github.com/balena-io/balena-cli/blob/v12.40.0/lib/utils/compose_ts.ts#L461

Originally reported on the following thread (private): https://www.flowdock.com/app/rulemotion/r-architecture/threads/tGKgFDu1Ofh2LCyGbBi_qIqHsYt

[pdcastro]

Additional information: An ARM v8 (aarch64) CPU is able to natively run ARM v7 or v6 code, without QEMU. For example, on a Raspberry Pi 4 (balenaOS 64-bit) in local mode, and I can successfully build an armv7hf (RPi 3) image as follows:

/tmp/t/Dockerfile

FROM balenalib/raspberrypi3-alpine-node
RUN apk add file
RUN uname -a
RUN file /usr/local/bin/node
RUN /usr/local/bin/node -e 'console.log("hello world")' > out.txt
RUN cat out.txt
CMD /bin/bash

root@d1dc447:/tmp/t# balena-engine build -t test-rpi3 --no-cache .
Sending build context to Docker daemon  2.048kB
Step 1/7 : FROM balenalib/raspberrypi3-alpine-node
 ---> 5dbf38d7379d
Step 2/7 : RUN apk add file
 ---> Running in bb7bc6323321
Here are a few details about this Docker image (For more information please visit https://www.balena.io/docs/reference/base-images/base-images/):
Architecture: ARM v7
OS: Alpine Linux 3.13
Variant: run variant
Default variable(s): UDEV=off
The following software stack is preinstalled:
Node.js v15.7.0, Yarn v1.22.4
Extra features:
- Easy way to install packages with `install_packages <package-name>` command
- Run anywhere with cross-build feature  (for ARM only)
- Keep the container idling with `balena-idle` command
- Show base image details with `balena-info` command
fetch https://dl-cdn.alpinelinux.org/alpine/v3.13/main/armv7/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.13/community/armv7/APKINDEX.tar.gz
(1/1) Installing file (5.39-r0)
Executing busybox-1.32.1-r2.trigger
OK: 57 MiB in 78 packages
Removing intermediate container bb7bc6323321
 ---> 24e4b8586103
Step 3/7 : RUN uname -a
 ---> Running in 6db379c53b1f
Linux f91ad8d0b1a4 5.4.58 #1 SMP PREEMPT Wed Dec 16 13:40:47 UTC 2020 aarch64 Linux
Removing intermediate container 6db379c53b1f
 ---> a3f0f79713c9
Step 4/7 : RUN file /usr/local/bin/node
 ---> Running in 29845da2edb0
/usr/local/bin/node: ELF 32-bit LSB pie executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /lib/ld-musl-armhf.so.1, with debug_info, not stripped
Removing intermediate container 29845da2edb0
 ---> be61510423e8
Step 5/7 : RUN /usr/local/bin/node -e 'console.log("hello world")' > out.txt
 ---> Running in 73c993a837f9
Removing intermediate container 73c993a837f9
 ---> e902e6967b57
Step 6/7 : RUN cat out.txt
 ---> Running in 4f1dfeca9d4b
hello world
Removing intermediate container 4f1dfeca9d4b
 ---> 46cb6d5c9531
Step 7/7 : CMD /bin/bash
 ---> Running in fed642579005
Removing intermediate container fed642579005
 ---> ee10e18e6c53
Successfully built ee10e18e6c53

Note in particular how the node executable provided by the base image is a 32-bit armv7hf binary:

RUN file /usr/local/bin/node
 ---> Running in 29845da2edb0
/usr/local/bin/node: ELF 32-bit LSB pie executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /lib/ld-musl-armhf.so.1, with debug_info, not stripped

And yet, uname -a reported aarch64:

Step 3/7 : RUN uname -a
 ---> Running in 6db379c53b1f
Linux f91ad8d0b1a4 5.4.58 #1 SMP PREEMPT Wed Dec 16 13:40:47 UTC 2020 aarch64 Linux

[pdcastro]

Additional information: The current CLI releases for Intel (e.g. CLI running on a NUC with balenaOS and balenaEngine, not using binfmt_misc), using QEMU to build ARM images, report either armv7l or aarch64 (and no other variant like ARM v6) in the output of uname:

• balena build -e -A rpi -d raspberry-pi: RUN uname -a reports armv7l
• balena build -e -A armv7hf -d raspberrypi3: RUN uname -a reports armv7l
• balena build -e -A aarch64 -d raspberrypi4-64: RUN uname -a reports aarch64

... even when using the respective base images in the Dockerfile: FROM balenalib/raspberry-pi-alpine-node FROM balenalib/raspberrypi3-alpine-node FROM balenalib/raspberrypi4-64-alpine-node

This may have something to do with the fact that we only produce two QEMU emulators, one for ARM 32 bits (selected by the CLI for both ARM v6 and v7) and one for ARM 64 bits (selected for ARM v8 / aarch64):

qemu-5.2.0.balena4-arm.tar.gz qemu-5.2.0.balena4-aarch64.tar.gz https://github.com/balena-io/qemu/releases/tag/v5.2.0%2Bbalena4

The results happen to be the same when running the CLI on my Mac laptop with Docker Desktop for Mac and binfmt_misc instead of QEMU.

[pdcastro]

Additional information:

It's also worth noting that setarch --32-bit or setarch armv7l doesn't resolve this problem, as it makes uname return armv8l, which pip detects as incompatible with armv7l and armv6l.

[pdcastro]

Additional information: Here's @jakogut's proposed (draft) implementation of libfakeuname.so

libfakeuname.c

/* RTLD_NEXT is not POSIX, indicate that we want GNU extensions */
#define _GNU_SOURCE

#include <dlfcn.h>
#include <errno.h>
#include <sys/utsname.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

/* The constructor will initialize this to the location of the original
 * implementation of uname
 */
int (*orig_uname)(struct utsname *name);

void inject_overrides(struct utsname *name)
{
    char *machine_override;
    machine_override = getenv("MACHINE_OVERRIDE");
    if (machine_override) {
        strncpy(name->machine, machine_override, sizeof(name->machine));
    }
}

int uname(struct utsname *name)
{
    int ret;
    ret = orig_uname(name);
    inject_overrides(name);
    return ret;
}

__attribute__((constructor))
void init(void)
{
    orig_uname = dlsym(RTLD_NEXT, "uname");
}

Makefile

CC ?= gcc
CFLAGS ?= -Wall -fPIC -Os

LIB_LDFLAGS += -shared

SRCS = libfakeuname.c
LIBS = libfakeuname.so

all: ${LIBS}

${LIBS}: ${SRCS} 
    $(CC) $(LIB_LDFLAGS) -o $@ $(CFLAGS) -ldl $<