metropolis: firmware loading issues

[lorenz]

We currently have all kernel modules built-in (for ease of building the image as well as security), the firmware is either built into the initramfs or the rootfs depending on what is used to build. This works fine for the initramfs, but is broken for rootfs as Linux does not wait for the rootfs to become available before trying to read firmware from it. This causes Metropolis to fail to initialize network adapters requiring firmware before they become available:

...
[    8.674518] ice 0000:41:00.0: Direct firmware load for intel/ice/ddp/ice.pkg failed with error -2
...
[   10.962346] erofs: (device dm-0): mounted with root inode @ nid 36.
[   10.974204] VFS: Mounted root (erofs filesystem) readonly on device 253:0.

Changes to this behavior have been discussed on LKML, Linus's opinion is however that this is just not something you should be doing, you should keep the firmware together with the modules. For us this is quite impractical as the firmware can be big and if we need to colocate it we need to include it in the kernel (https://www.kernel.org/doc/html/latest/driver-api/firmware/built-in-fw.html) we bloat the EFI payload significantly as well as incurring license shenanigans.

Another option would be to abandon building in everything and put the modules onto the rootfs, same as with the firmware. This has the advantage of only needing to ship the firmware once as well as improving boot performance as it will allow us to only load the modules necessary to boot. Disadvantage is more complexity in the kernel build process, needing to figure out load_pin/verity to make sure that the module loading interface is secure as well as have some small Go process which handles the dynamic module loading (like udev).

[q3k]

https://review.monogon.dev/c/monogon/+/1791

[lorenz]

Has now landed in c7b036bca213962a7e60f3edb47624606799d074, 46bf7d6c6437dfbf9dcc1e1d7d80fcc1c601f9b5 and 6c45434189e387b234109b68b1ed5a8f2cd5b439