Generate image with robust A/B update using mender.io.

[lueschem]

Goal

Provide an edi configuration similar to edi-pi (see also this blog post) with full support for mender.io.

The configuration can be used to generate a Debian image for the Raspberry Pi that allows robust A/B updates of a big fleet of devices. Furthermore the configuration shall also produce a cross compilation environment for faster development.

Sub Goals

• Allow a hybrid update strategy (mender for image based upgrades, apt for package based updates)
• The hybrid update strategy shall also allow a kernel update (based on edi-boot-shim) using apt
• Directly build the mender compliant image without making the detour over mender-convert (However: mender-convert is a really great tool!)
• Vanilla u-boot shall be used as bootloader - the mender specific patches shall get replaced by a specially crafted hush script (boot.scr).
• Mender client shall be installed as a Debian package (many thanks to Andreas Henriksson!)

Partition Layout

vfat boot partition

The following stuff is on the vfat partition:

• Pi bootloader
• u-boot (unpatched)
• uboot.env (persistent storage of environment variables, gets written on first boot)
• boot.scr: does the mender glue logic, loads the boot.scr of the active ext4 partition

1x ext4 data partition

The data partition is shared between the A and the B system. The partition has a fixed size.

2x ext4 system partition (A/B)

The system partitions have the following characteristics:

• Dynamically adjusted size on first boot to make use of the full disk.
• Contain a full Debian (currently stretch) root file system including kernel
• Contain a boot.scr that describes how to load the kernel/device-tree/initrd (the boot.scr gets updated by edi-boot-shim)

Design Reasons

• I would like to keep the possibility of apt based updates to a) speed up updates during development and to b) have reduced network traffic for security updates of an IoT fleet
• I would like to have an unmodified u-boot in order to get rid of maintaining a forked u-boot (I am a lazy guy where possible)
• I would like to directly generate mender compliant images to have feature parity with Yocto, shorter build cycles and more flexibility
• I would like to use Debian kernels directly (without moving around dtbs, kernels and initrds) (see also this blog post)

[lueschem]

It looks like Debian buster (arm64 container, amd64 host, qemu 3.1, systemd 240) currently can not be run within an LXD container (no network, systemd is eating 100% CPU). However, it works if there is no qemu user mode emulation. I will try to analyze this later.

For now, I will build a Debian stretch based image.

[mirzak]

Great initiative and overall looks good.

Note that supporting a vanilla U-boot might be difficult, even if most of the scripting can be moved to boot.scr, e.g this patch is required to fully support "robust" environment and to support "rollback" CONFIG_BOOTCOUNT_LIMIT is needed.

Note that the U-boot fork for RPi is maintained by Mender folks as it is used for mender-convert. If we can establish collaboration we might be able to ease the pain of maintaining that :)

[lueschem]

Many thanks for taking a look at my proposal! My assumption that CONFIG_BOOTCOUNT_LIMIT is a generic U-Boot feature was actually wrong. I will take a deeper look at the required U-Boot changes. Maybe the scripting that is done for RAUC could also point us to a "patch free" U-Boot. I will also try to find out what the people at SWUpdate do.

[lueschem]

Solution see: https://www.get-edi.io/Updating-a-Debian-Based-IoT-Fleet/

[mrbluecoat]

Is there a way to implement this on non-Raspberry Pi devices, like Orange Pi running Armbian? I'll take a look at https://docs.mender.io/system-updates-debian-family/convert-a-mender-debian-image and https://hub.mender.io/t/rockpro64-armbian/770

[lueschem]

Yes, absolutely. Here is an example for the Compulab IOT-GATE-iMX8. And there is some work in progress for the Variscite i.MX 8 Nano. However, I have not touched the Orange Pi yet. If there is already an Armbian setup then it should be easier to derive an edi setup from it. Another possibility would be to integrate the Mender setup into the Armbian setup.