search

siderolabs / pkgs

Mozilla Public License 2.0
31 stars 104 forks source link

The existing boot mechanism only supports RPI 4B 4GB USB boot, not RPI 4B 8GB #229

Closed ossfellow closed 1 year ago

ossfellow commented 3 years ago

I ran a number of tests to determine if the current boot solution does also support USB boot or not and here's what I've found so far:

  1. USB2 boot halts right after XFS mount (step 6 of installation); it doesn't provide much of a performance boost, so, this could be ignored, and just seen as an observation.
  2. USB3/3.1 (Gen1) boot works, with RPI4 firmware version 2020-09-03. I updated the EEPROM's config to boot from USB first, but this isn't an absolute necessity, if no SD card is inserted.
  3. The very first installation took around 20 minutes, but 2 subsequent installations took considerably less, which is what's expected given the much higher I/O bandwidth, compared to SD's.
  4. The attached file has rudimentary measurements of installation times, as well as boot times, and what was happening, for USB3.1 and SD card, plus the configuration files used (akimiski.yaml and talosconfig).

Issues encountered:

  1. providing --arch flag to talosctl gen config has no effect, as it always generated amd64 config files (I manually modified the init.yaml, even though it seems it's not required). I wonder if the flag is needed at all, given that the installer can, perhaps, detect the architecture and pull the corresponding images (there could be edge cases).
  2. Adding my internal CA's public key to the configuration file, based on the Talos guide would cause an installation crash loop. I tried it multiple times and the installation keeps failing and rebooting the node. Unfortunately, the crash happens in earlier stages of the installation, so, I could not query the server for any of its internal states or logs, to attach, but I could recreate the failure on VirtualBox and captured the video recording. Please note that on VirtualBox instead of rebooting it halts/sleeps, but the error could be seen, which happens in step 12 of installation. I've also attached the configuration file, which includes the CA public key (arctic.yaml).
  3. The biggest issue I encountered was that USB boot only works on RPI 4B 4GB. This means 8GB version is not supported. This isn't entirely surprising though given what's documented on Debian wiki, VL805's firmware patch, and on Raspberry Pi's firmware site. I have been able to boot Ubuntu 20.04.1 using a decompressed vmlinuz and a modified config.txt (which is attached).

As part of my research work, I run ML models on Kubernetes on RPI clusters, and to get the best o both worlds I'm willing to do a custom build using the decompressed kernel approach, if you could confirm that if I include the decompressed kernel, it wouldn't interfere with the Talos installation solution. If it's possible I'd post the results by early next week, for anyone, who might be interested. Talos.zip

andrewrynhard commented 3 years ago

@masoudbahar Really appreciate this. Excellent feedback. Let us chew on this for a day or two. Looks there is already a discussion in the slack too. We'd have to think through how to get Talos booting off USB and the 8GB model, but I have some ideas.

ossfellow commented 3 years ago

Thank you Andrew; that will be awesome. I haven't tried it personally, but I've read Ubuntu 20.10 supports USB boot out of the box, using u-boot.

smira commented 3 years ago

Debian mentions USB boot is broken with Linux <5.7, but we have latest kernel

ossfellow commented 3 years ago

That’s true, as apparently the fixes are in 5.8 kernel, but the decompressed approach works well. Having said that there seems to be a better of managing all of this (boot from SD, store state on USB connected disk, as Andrew has clarified). Once I get the hint about how to do it, I’ll try it, update this issue, and also submit a documentation update PR, so others would learn the approach too.

DWSR commented 3 years ago

@masoudbahar Were you ever able to document how you got USB support working on the Pi 4 8GB? I'm currently getting bitten by this and would love to know how you worked around it. I'm not so concerned with USB boot as just the ability to interface with USB disks as user disks in Talos.

AlekSi commented 3 years ago

That should be fixed at https://github.com/talos-systems/talos/releases/tag/v0.10.0-alpha.0

DWSR commented 1 year ago

Confirmed this is fixed as of a long time ago. This issue should be closed. @andrewrynhard

frezbo commented 1 year ago

Closing