search

silitics / rugpi

An open-source platform empowering you to build innovative devices around customized Linux distributions.
https://rugpi.io
Apache License 2.0
35 stars 1 forks source link

support older boards without the `tryboot` feature #4

Closed koehlma closed 9 months ago

koehlma commented 9 months ago

In principle, it should be possible to add support for older boards without the tryboot feature by using U-Boot as an intermediate bootloader. U-Boot would then be responsible for switching between A/B partitions. This would also open up the possibility to use Rugpi with other boards than Raspberry Pi. If you want to see this happen, please give this issue a 👍.

Known Caveats

Open Questions

reubenmiller commented 9 months ago

swupdate might also be worth evaluating as it does have the potential to stream the new images to partition (though I haven't used it personally).

Streaming the new image directly to the cold partition is attractive for devices with little available disk space. Mender supports streaming by default.

koehlma commented 9 months ago

I decided against using RAUC or SWUpdate as a basis (for now). Integrating with them would add a lot of complexity (both in terms of required configuration and the building pipeline). As it turns out, getting a U-Boot setup working was easier than anticipated. And, by doing this ourselves, we gain more flexibility going ahead and overall achieve greater consistency between the different boot flows.

Rugpi Bakery does now support an option boot_flow = "u-boot" to enable the U-Boot boot flow (this also disables the tryboot boot flow). I modeled the U-Boot boot flow after the tryboot boot flow such that they are more consistent (still need to update the docs). In particular, committing works by writing a file to the config partition.

I tested the image built with the U-Boot boot flow and the Bullseye-based release of Raspberry Pi OS on a Raspberry Pi 3. In principle, the same image should work for Zero 2W as the device tree loaded by the Raspberry Pi firmware is used (or at least that is what the U-Boot documentation claims).

@reubenmiller Could you please test, whether it works on the Zero 2W?

koehlma commented 9 months ago

Rugpi now also has experimental support for streaming images directly to the SD card by using --stream. You can use

rugpi-ctrl update install --stream -

to install an image streamed to stdin, e.g., via curl. This also allows using compressed images by piping them through xz (or gzip). Mid-term, streaming should also become the default.

reubenmiller commented 9 months ago

@koehlma I must be doing something wrong, as I can't get the image to boot on Raspberry PI 3 or Zero 2W.

I've set the boot flow in the rugpi-bakery.toml (see my project here: https://github.com/reubenmiller/rpi-tedge-image/tree/support-rp3), and I've made sure the latest docker image

boot_flow = "u-boot"

I've tried the following images (buster and bookworm):

And to confirm that there was nothing wrong with my sd card or my usage of the Raspberry Pi Imager, switching to the official images is able to boot both devices successfully.

koehlma commented 9 months ago

No, you are doing everything right. GitHub Actions does not include LFS files by default. The U-Boot binaries are stored in LFS. Instead of them, their stubs were then baked into the image. Commit 94fd6e7 should fix that. If you are building the Docker image yourself, please make sure that you have a working LFS setup.

Edit: The integrity of the binaries will now also be verified when building the Docker image (see 44f3f70).

reubenmiller commented 9 months ago

Perfect. I’ll give the images a try again in the next day.

reubenmiller commented 9 months ago

I've been able to successfully test initial bootstrapping and using the u-boot and everything works as expected 🥇

Procedure

  1. Build an image then flash the SD Card (using the boot_flow = "u-boot" option)
  2. Boot device for the first time with flashed SD Card
  3. Perform an update (using the same) image, going from A -> B partition

Devices under Test

✅ = PASS

reubenmiller commented 9 months ago

Any idea on the effort to support older 32 bit Pi models such as, Raspberry Pi 1, 2 and Zero? I would be happy to assist with testing (if that helps).

koehlma commented 9 months ago

That's great news!

For 32-bit support, first Rugpi Bakery needs to be made aware of the architecture (currently, it simply assumes arm64). As far as I know, there are three different architectures which we would have to support: arm64 (ARMv8), armhf (ARMv7), and armel (ARMv6). At least this is how the CPU families fit into Debian's architecture schema.

Raspberry Pi 2 B v1.2 is actually based on ARMv8 (same chip family as Pi 3) and should support 64-bit. However, I am not sure whether this is properly supported by the bootrom (first stage bootloader burned into the ROM of the chip). I can test the existing U-Boot boot flow with it later – maybe it just works. ;) I tried using the existing image and U-Boot binary and copied over the respective .dtbs but it did not work.

In any case, supporting armhf, i.e., the entire Raspberry Pi 2 family, should be rather easy. We would have to produce U-Boot binaries and adapt Rugpi Bakery to place them together with an appropriate config.txt in the image. Rugpi Ctrl gets build for armhf already. The recipe installing it simply needs to be made aware of the architecture. This should all be easily doable in an afternoon.

For Raspberry Pi 1 and Zero, I am not sure. Actually, it is still a bit unclear to me how there can just be one armhf image apparently working on all Pis. It seems like this involves some customization as it falls between the officially supported Debian architectures armhf and armel.^1 I guess, building Rugpi Ctrl for arm-unknown-linux-musleabihf instead of armv7-unknown-linux-musleabihf would do the trick – but I have to investigate this further.

Do you think that someone starting a new project with Rugpi today would like to use those older boards? Or, is this just so that existing setups can switch to Rugpi? I would like to understand the use case better before investing time into this.

reubenmiller commented 9 months ago

That's great news!

For 32-bit support, first Rugpi Bakery needs to be made aware of the architecture (currently, it simply assumes arm64). As far as I know, there are three different architectures which we would have to support: arm64 (ARMv8), armhf (ARMv7), and armel (ARMv6). At least this is how the CPU families fit into Debian's architecture schema.

Raspberry Pi 2 B v1.2 is actually based on ARMv8 (same chip family as Pi 3) and should support 64-bit. However, I am not sure whether this is properly supported by the bootrom (first stage bootloader burned into the ROM of the chip). ~I can test the existing U-Boot boot flow with it later – maybe it just works. ;)~ I tried using the existing image and U-Boot binary and copied over the respective .dtbs but it did not work.

In any case, supporting armhf, i.e., the entire Raspberry Pi 2 family, should be rather easy. We would have to produce U-Boot binaries and adapt Rugpi Bakery to place them together with an appropriate config.txt in the image. Rugpi Ctrl gets build for armhf already. The recipe installing it simply needs to be made aware of the architecture. This should all be easily doable in an afternoon.

For Raspberry Pi 1 and Zero, I am not sure. Actually, it is still a bit unclear to me how there can just be one armhf image apparently working on all Pis. It seems like this involves some customization as it falls between the officially supported Debian architectures armhf and armel.1 I guess, building Rugpi Ctrl for arm-unknown-linux-musleabihf instead of armv7-unknown-linux-musleabihf would do the trick – but I have to investigate this further.

Do you think that someone starting a new project with Rugpi today would like to use those older boards? Or, is this just so that existing setups can switch to Rugpi? I would like to understand the use case better before investing time into this.

Footnotes

  1. https://wiki.debian.org/RaspberryPi ↩

I was able to run the same image and do an image update on a Raspberry Pi 2 Model B Rev 1.2. I've done one successful update, so I'm happy to monitor future updates on this device to see if it could fall under the "supported" category.

Generally I think you can skip support for Raspberry 1 and Zero as these devices are very limited (only single core processors). I was just curious to see if it was an easy-win or not, as some customers are pushing for lower cost devices all the time and consider using older devices with lower specs just to save the per-unit cost....however I would still avoid those customers to avoid a single core cpu if possible.

koehlma commented 9 months ago

I just realized that what I believed to be a Pi 2 B v1.2 is actually a Pi 1 B+ v1.2. That is probably the reason why it did not work.

Yes, that makes sense. Let's see. Of course, it would also be nice if we can just claim that every Pi model is supported. At some point it may also make sense to look at other boards than Raspberry Pi.

koehlma commented 9 months ago

There may now be support for Pi 1, Pi 2, and Pi Zero. You need to use the following settings:

architecture = "armhf"
boot_flow = "u-boot"

They have to be used for customize and for bake.

And, of course, a 32-bit base image must be used (Bookworm worked for me).

I did test the resulting image on a Pi 1 B+ v1.2 and it seems to work.

@reubenmiller Can you please test whether this works on Pi Zero and Pi 2? The image should be identical for all those boards as the bootloader is selected by a filter in config.txt.

Took me a bit less than an hour to get that working.

reubenmiller commented 9 months ago

@koehlma I'm checking the images now, though I ran in to the classic error of forgetting to install the armhf using tonistiigi/binfmt, so should probably make it into the docs after we can confirm the image is working (should have confirmation within 12 hours)...

But for those reading this, if you are building for armhf, then make sure you run the following command first.

docker run --privileged --rm tonistiigi/binfmt --install armhf
reubenmiller commented 9 months ago

Sorry I'm distracted by other activities at the moment...but I should have something to report in the next few days.

reubenmiller commented 9 months ago

After doing some testing I can confirm that I have been able to successfully build and flash the image, and do one OTA update (using thin-edge.io to apply the update, see tedge-rugpi-image).

Note: The above devices take 15-25 minutes to update given that they only have a one-core CPU.

The testing took longer than expected due to complications in the build process introduced because of the differences in the actual CPU architecture being used during the creation of the image, and the target device. Because of this, the install scripts I was using in the build was wrongly installing the armv7l (armv7-unknown-linux-musleabihf) version rather than the armv6 version of thin-edge.io (arm-unknown-linux-musleabihf).

Below shows the differences in the architectures in the build and target device:

At build time

$ uname -m
armv7l

At runtime (on device, pi0 and pi 1B)

$ uname -m
armv6l

I'm not sure if there is a better way to handle as the special "armv6" is not widely supported outside of Raspberry PI 1 and Zero, and causes a lot of issues (as you can't use the public debian repositories as armhf usually refers to armv7l and not armv6l).

koehlma commented 9 months ago

Great, thanks!

Indeed, the whole situation with the different architectures is a bit of a mess. In general, the architecture of a binary is automatically determined by binfmt_misc and the binary is then run via QEMU. We are not doing anything special here. I guess, this is then also what uname reports. Rugpi Ctrl is also compiled for arm-unknown-linux-musleabihf. Unfortunately, I do not think that we can do anything about it, except mentioning it in the docs.

I will close this issue now. It was an easy win after all. ;)

I am now also confident that we can rather easily support other boards, should the need arise.

As the next step, I will focus on introducing a layer architecture similar to Docker and enabling delta updates that way.