pkg/build: minimal linux init/image

[dvyukov]

There is a long standing idea of providing own minimal init that will do just the things we need and no more, most likely in a form of a static Go binary. Advantages:

• portable across arches
• boots fast
• can test arm64 in qemu/emulation on GCE (stretch doesn't boot in 10 minutes)
• simpler maintenance
• no variance in setup (we've seen some things are not mounted or mounted in different locations in different images)
• will allow simpler CI setup (build own image, run in qemu)

Disadvantages:

• unclear if we want to get into this business and maintain this

I guess the verdict depends on how easy it will be to build this. Here are some relevant links:

• u-root is able to build a Go init on-the-fly (use example)
• gokrazy is a pure-Go userland for your Raspberry Pi 3 appliances (quickstart, use example)
• https://github.com/bradfitz/embiggen-disk/blob/master/integration_test.go does something similar

[dvyukov]

However, for arm64 it can make sense to check a minimal buildroot image first.

[dvyukov]

Some references for u-root from @hugelgupf:

>Is there a similar programmatic API by any chance?
Yes, see https://godoc.org/github.com/u-root/u-root/pkg/uroot

Honestly, the best use example for that API is the main file: https://github.com/u-root/u-root/blob/master/u-root.go

as a side note, we also have infra that runs go tests in QEMU VMs... I saw qemu being mentioned.

this is how you run them (you use go test ... but with some env vars set)
https://github.com/u-root/u-root/blob/master/integration/README.md 

example 1 test setup: https://github.com/u-root/u-root/blob/master/pkg/mount/mount_integration_test.go (if you want to add commands to call from the test by shelling out, see the more config version below, but I try to use the programmatic APIs for most our stuff)
example 1 go test run inside QEMU VM: https://github.com/u-root/u-root/blob/master/pkg/mount/mount_linux_test.go

example 2 test setup (more configuration): https://github.com/hugelgupf/p9/blob/master/fsimpl/localfs/localfs_integration_test.go
example 2 test run inside VM: https://github.com/hugelgupf/p9/blob/master/fsimpl/localfs/localfs_test.go

the programmatic API for this one is... improvement-worthy.

you can also forego using Go unit tests, and run random commands in a shell with expect-style scripting, example:
https://github.com/u-root/u-root/blob/master/integration/generic-tests/pxeboot_test.go#L63

[evdenis]

Init from wireguard tests https://git.zx2c4.com/wireguard-linux-compat/tree/src/tests/qemu/init.c With minimal vm https://git.zx2c4.com/wireguard-linux-compat/tree/src/tests/qemu

[dvyukov]

@evdenis nice minimalistic example, that's useful, thanks.

[dvyukov]

LinuxKit: A toolkit for building secure, portable and lean operating systems for containers

[oswalpalash]

I've been testing buildroot images for arm64 kernels (linux/raspberry pi 64) on a newer pi board. Buildroot works the best compared to a big set of images I've tested (with the least amount of crashes surrounding startup compared to stretch/others).

[dvyukov]

FTR, here is a script that builds arm64 image that can work with syzkaller. I used buildroot on 2020.08.x.

set -eux

make qemu_aarch64_virt_defconfig

cat >>.config <<EOF
BR2_TARGET_GENERIC_HOSTNAME="syzkaller"
BR2_TARGET_GENERIC_ISSUE="syzkaller"
BR2_cortex_a57=y
BR2_TOOLCHAIN_BUILDROOT_GLIBC=y
BR2_ROOTFS_DEVICE_CREATION_DYNAMIC_EUDEV=y
BR2_PACKAGE_DHCPCD=y
BR2_PACKAGE_OPENSSH=y
# BR2_PACKAGE_URANDOM_SCRIPTS is not set
BR2_TARGET_ROOTFS_EXT2_SIZE="1G"
BR2_ROOTFS_POST_FAKEROOT_SCRIPT="./rootfs_script.sh"
# BR2_LINUX_KERNEL is not set
EOF

cat >rootfs_script.sh <<'EOFEOF'
set -eux

# Mount debugfs for KCOV.
cat >>$1/etc/fstab <<EOF
debugfs /sys/kernel/debug debugfs defaults 0 0
securityfs /sys/kernel/security securityfs defaults 0 0
configfs /sys/kernel/config/ configfs defaults 0 0
binfmt_misc /proc/sys/fs/binfmt_misc binfmt_misc defaults 0 0
EOF

# Setup ssh without key/password.
cat >$1/etc/ssh/sshd_config <<EOF
PermitRootLogin yes
PasswordAuthentication yes
PermitEmptyPasswords yes
EOF

# Generate sshd host keys.
ssh-keygen -A -f $1
mkdir -p $1/var/db/dhcpcd

cat >$1/etc/udev/rules.d/50-syzkaller.rules <<EOF
ATTR{name}=="vim2m", SYMLINK+="vim2m"
SUBSYSTEMS=="pci", DRIVERS=="i915", SYMLINK+="i915"
EOF
for i in {0..31}; do
    echo "KERNEL==\"binder$i\", NAME=\"binder$i\", MODE=\"0666\"" >> $1/etc/udev/rules.d/50-syzkaller.rules
done
EOFEOF
chmod u+x rootfs_script.sh

make olddefconfig
make

And here is qemu command line to run it:

qemu-system-aarch64 \
        -machine virt,virtualization=true -cpu cortex-a57 -smp 4 \
        -m 4G,slots=4,maxmem=16G \
        -device virtio-blk-device,drive=hd0 \
        -drive if=none,format=raw,id=hd0,file=/buildroot/output/images/rootfs.ext2 \
        -kernel arch/arm64/boot/Image \
        -nographic \
        -device virtio-rng-pci \
        -net user,host=10.0.2.10,hostfwd=tcp::10022-:22 -net nic,model=virtio-net-pci \
        -append "root=/dev/vda earlyprintk=serial"

[dvyukov]

amd64 image also works with qemu_x86_64_defconfig and removing BR2_cortex_a57=y (which does not harm either).

[dvyukov]

There is an unpleasant issue with firmware.

Some drivers require separate firmware blobs:

[   67.739498][ T3000] usb 1-1: ath9k_htc: Firmware ath9k_htc/htc_9271-1.4.0.fw requested
[   68.397443][ T3000] usb 1-1: ath9k_htc: Transferred FW: ath9k_htc/htc_9271-1.4.0.fw, size: 51008

Without the firmware blob the driver won't work.

In the Debian Stretch image these blobs are located in /lib/firmware:

# ls -1 /lib/firmware/
ar3k
ar5523.bin
ar7010.fw
ar7010_1_1.fw
ar9271.fw
ath10k
ath3k-1.fw
ath6k
ath9k_htc
htc_7010.fw
htc_9271.fw
qca

If we use buildroot, (1) we need to install these blobs as well, (2) figure out how kernel locates it (how does it know about /lib/firmware).

FTR, ath9k firmware is available here: https://github.com/qca/open-ath9k-htc-firmware