search

siderolabs / talos

Talos Linux is a modern Linux distribution built for Kubernetes.
https://www.talos.dev
Mozilla Public License 2.0
6.95k stars 564 forks source link

Support for rpi_5 #7978

Open etenzy opened 1 year ago

etenzy commented 1 year ago

When i try to start the rpi_generic version i get the following output:

Raspberry Pi 5 - 8GB
bootloader: 30de0ba5 2023/10/30
update-ts: 1699955031

board: d04170 797932e d8:3a:dd:xx:xx:xx
boot: mode USB-MSD 4 order f41 retry 0/128 restart 0/-1
SD: card detected 00035344534433324785594c22760176
part: 0 mbr [Oxee:00000001 0x00:00000000 0x00:00000000 0x00:000000001
power: supply: USB-PD 3000 mA CC1 PHIC: reset normal 0x0 usb_over_current=0
net: down ip: 0.0.0.0 sn: 0.0.0.0 gw: 0.0.0.0
tftp: 0.0.0.0 00:00:00:00:00:00
display: DISPO: HDMI HPD=1 EDID=ok #2 DISP1: HPD=0 EDID=none #0

usb_max_current_enable default 0 max-current 3000
Device-tree file "bcm2712-rpi-5-b.dtb" not found.

The installed operating system (OS) does not indicate support for Raspberry Pi 5
Update the OS or set os_check=0 in config.txt to skip this check.
Trying partition: 0
type: 32 1ba: 2048 'mkfs.fat' ' EFI • clusters 201618 (1)
Trying partition: 0
type: 32 1ba: 2048 'mkfs.fat' • EFI • clusters 201618 (1)
Read config.txt bytes 576 hnd 0x19759
usb_max_current_enable default 0 max-current 3000
Device-tree file "bcm2712-rpi-5-b.dtb" not found.

The installed operating system (OS) does not indicate support for Raspberry Pi 5
Update the OS or set os_check=0 in config.txt to skip this check.
Boot mode: USB-MSD (04) order f
wurmr commented 11 months ago

In the latest beta build v1.6.0-beta.1 it doesn't even get this far.

On a 4GB Pi 5 it never gets to the step of reading config.txt and keeps looping forever on the line type: 32 1ba: 2048 'mkfs.fat' • EFI • clusters 201618 (1)

rothgar commented 9 months ago

I just booted 1.6.4 and get the same error about os_check=0.

Based on the post here it might have to wait for 6.8 kernel to be supported.

AlexanderSkrock commented 8 months ago

Hi there,

as the mentioned Linux version 6.8 is released now, I would like to take the opportunity to ask whether there are any plans to support the Rasberry Pi 5 in the near future. I would be excited to get my newly purchased Rasberry Pi 5 up and running using Talos.

Kind regards Alexander Skrock

frezbo commented 8 months ago

There is no u-boot support yet, so it seems way far ahead

Toasterson commented 6 months ago

Any updates since? I am going with k3s at the moment but would love to testdrive Talos once it's ready.

rothgar commented 6 months ago

Based on some email list conversations I read it looks like uboot 2024.04 adds support for raspberry pi 5 booting from SD card (no USB or nvme)

If someone wants to try it out you would have to update the uboot version here https://github.com/siderolabs/sbc-raspberrypi/blob/main/Pkgfile#L12-L14 and build a docker image and push it to a registry

Then you can follow these steps to use imager to create a custom image that you should be able to dd to an SD card. https://www.talos.dev/v1.7/talos-guides/install/boot-assets/#example-raspberry-pi-overlay-with-imager

I'm leaving for vacation soon so I don't have time to try it but would love to hear back from anyone that does.

githubcdr commented 5 months ago

I tried compiling u-boot but it didn't work, the nvme patches seem to be required for compiling.

rothgar commented 5 months ago

The uboot patch says it only supports sd card right now. Did you try that?

githubcdr commented 5 months ago

I tried compiling without any patch and with patches, both failed for me.

githubcdr commented 5 months ago 
38.13 make[1]: *** [scripts/Makefile.host:112: tools/image-sig-host.o] Segmentation fault (core dumped)
38.13 make[1]: *** Deleting file 'tools/image-sig-host.o'
38.13 make[1]: *** Waiting for unfinished jobs....
41.22 make[1]: *** [scripts/Makefile.host:112: tools/imx8image.o] Error 139
41.22 make[1]: *** Deleting file 'tools/imx8image.o'
41.22 make: *** [Makefile:1859: tools] Error 2
githubcdr commented 5 months ago

Good news! I have been able to boot Talos on rpi 5 using manually compiled uboot 2024.4, the ethernet port does not seem to be detected, probably some config.txt issues..

ruryx00 commented 5 months ago

Good news! I have been able to boot Talos on rpi 5 using manually compiled uboot 2024.4, the ethernet port does not seem to be detected, probably some config.txt issues..

I'm trying also could you explain better how you were able to do it. thanks

githubcdr commented 5 months ago

I first tried to build it via the bldr tool, but for some reason that gave me segfaults. What I did instead was manually compile uboot 2024.4 from source on a arm64 machine (rpi4) and then just place u-boot.bin file in the /boot partition :)

But as mentioned this will not work since the rpi firmware and the talos kernel must match if I understand correctly..

Maikusan commented 4 months ago

Are there any news regarding raspi 5 support?

pocket-pc commented 3 months ago

Should this be moved to siderolabs/sbc-raspberrypi for task tracking?

mooneydude commented 1 month ago

Good news! I have been able to boot Talos on rpi 5 using manually compiled uboot 2024.4, the ethernet port does not seem to be detected, probably some config.txt issues..

i also just got Talos to boot on Pi 5 but hangs on 'talos failed looking up time.cloudflare.com'? Any progress with the ethernet port?

philipgeraldtaylor commented 3 weeks ago

A couple of Kiwis with @mooneydude same issue bump this.

attilaolah commented 2 weeks ago

We were planning to bring up a small lab cluster with 3 Pi 5s and bumped into the same issue.

Boot hangs as it can't reach the NTP server, sad thing is it never goes further. There is no Ethernet signal so my best guess is that the Ethernet driver is not loaded at all, or if loaded then not configured, but our clunky setup with the Pi booting makes it hard to even capture dmesg.

I would invest a little time here maybe to try and push this a little further, for starters I think building a custom image with a modified config would be sufficient, what I'd be interested in is:

EDIT: There is also some discussion over at https://github.com/siderolabs/sbc-raspberrypi/issues/23.

rothgar commented 2 weeks ago

Has anyone tried adding the usb modem system extension and using a USB Ethernet adapter?

attilaolah commented 2 weeks ago

I'd be up to try that tomorrow when I have both the Pis and USB ethernet adapter at one place. It wouldn't be a long-term solution as I only have one adapter but if it gets us far enough to apply configs with a custom installer at least it might be easier to experiment or try loading the right ethernet module.

attilaolah commented 2 weeks ago

I started with the following schematic:

overlay:
  name: rpi_generic
  image: siderolabs/sbc-raspberrypi
customization:
  extraKernelArgs:
  - ip=:::::eth1:dhcp
  systemExtensions:
    officialExtensions:
    - siderolabs/usb-modem

…to get the schematic ID 3c84f43bf6579424d25b9c6239410028868946757ce84edc08378ed0f1726061, and tried running https://factory.talos.dev/image/3c84f43bf6579424d25b9c6239410028868946757ce84edc08378ed0f1726061/v1.9.0-alpha.2/metal-arm64.raw.xz on the Pi, with a USB Ethernet adapter that identifies itself as:

0b95:1790 ASIX Electronics Corp. AX88179 Gigabit Ethernet

I verified with Raspberry Pi OS that the USB enhernet adapter works and left it in the same USB slot. No luck, same output: Ethernet adapter is off, Talos stuck at NTP lookup failure.

Then I tried schematic 3e9f701c9714396a76061cf7069322f4188cd7428f9b1516c93a5c5bea47dada with the following congif:

overlay:
  image: siderolabs/sbc-raspberrypi
  name: rpi_generic
  options:
    configTxtAppend: |-
      dtoverlay=disable-bt-pi5
      dtoverlay=disable-wifi-pi5
customization:
  extraKernelArgs:
  - net.ifnames=0
  - ip=:::::eth1:dhcp
  systemExtensions:
    officialExtensions:
    - siderolabs/usb-modem-drivers

Same result with Talos 1.9.0-alpha.2.

I'm not sure what's the correct incantation of kernel args to get it to work. I also don't know what would be the predictable interface name, I have the adapter in the blue USB 3 port next to the built-in network adapter, in the one closest to the board (lower one), which identifies itself as "Bus 002 Device 005".

I'll play with it a little more but any suggestions would be highly appreciated.

EDIT: And I verified the µSD card contents, the kernel params are properly set in grub/grub.cfg, and config.txt is properly updated with additional two lines.

thielepaul commented 2 weeks ago

I think both the ethernet port as well as the usb connectors are behind the RP1 chip and are not supported in the mainline kernel 6.6 that talos is using. See also this thread: https://forums.raspberrypi.com/viewtopic.php?t=360653

I am wondering if it is possible to build a talos image using the kernel from raspberry pi instead of the mainline kernel? https://github.com/raspberrypi/linux

attilaolah commented 2 weeks ago

Well, I tried to build the sbc-raspberrypi image yesterday with the latest firmware which includes 6.6.60, but ~the patches could not be applied cleanly, and~ ultimately my build failed, so I gave up.

In the meantime, I discovered that the USB adapter uses the ax88179_178a module. To be honest I'd rather stick to the mainline kernel and apply the patches necessary to build network adapter… or even just have the adapter as an overlay.

EDIT: I think it is unlikely that the uboot patches have anything to do with my build failure; uboot seems to be working as-is. But even with a recent firmware, I think in the end you may be right, it's best if I try to build the Raspberry Pi kernel that you linked…

attilaolah commented 2 weeks ago

I also attempted to build the kernel using github.com/siderolabs/pkgs and running make kernel PLATFORM=linux/arm64, and I end up with a segfaulting gcc :/

What I did was:

I tried with the latest 6.6.60, compiler segfaults. I tried with the Raspberry Pi latest stable, 6.6.51, that then needs modifications to the hardening checker to ignore the two missing missing UBSAN configs which are apparently not present in that version yet. However, even after removing, the build fails.

My next attempt is going to be updating the Raspberry Pi os, loading configs, then extracting the config from the running kernel (which is the latest stable, 6.6.51), then import that config directly and try to build that in siderolabs/pkgs. The difference there is that Pi Os maintainers use a slightly different GCC but I'm hoping that wouldn't be much of an issue.

Apologies for spamming this ticket here, if I get it to work I'll make sure to write up the steps to reproduce; if not, at least someone might pick it up where I left off.

attilaolah commented 2 weeks ago

EDIT: Nevermind; I eventually worked around the segfaulting gcc by setting up a native remote builder for buildx. That seems to work now but is quite slow so it will take a while to get to a working kernel, but should be possible.

Old comment below.

By any chance, could you give me the Toml config you use for setting up the buildx builder, for building the kernel package? I may be paranoid, but I'm beginning to think that my Buildx builder is misconfigured. I'm using Buildkit 0.17.1 with Docker 27.3.1 using the containerd image store.

I tried to build the kernel with:

My thinking is, it shouldn't be that easy to get GCC to segfault, and I have plenty of free RAM and cores so the one thing I can think of is the build environment somehow being different.

I used the config from make help:

[worker.oci]
  gc = true
  gckeepstorage = 50000

  [[worker.oci.gcpolicy]]
    keepBytes = 10737418240
    keepDuration = 604800
    filters = [ "type==source.local", "type==exec.cachemount", "type==source.git.checkout"]
  [[worker.oci.gcpolicy]]
    all = true
    keepBytes = 53687091200
attilaolah commented 2 weeks ago

I ended up building the Raspberry Pi kernel version 6.6.60 using configs from the Talos repo, modified by make kernel-olddefconfig. This still includes all the hardening flags so the hardening checks pass. I also built a new firmware overlay containing matching firmware for 6.6.60. Finally, I build the SD card image like so:

docker run --rm -it --privileged=true \
    -v /dev:/dev \
    -v $PWD/out:/out \
    attilaolah/talos-imager:v1.9.0-alpha.2-5-g9a02ecc49 \
    rpi_generic \
    --arch arm64 \
    --overlay-image attilaolah/talos-sbc-raspberrypi:v0.1.0-beta.2-2-gf2a59cd@sha256:f15b0f12082dfce081484c3957d1e7b8719c9da75dca2596da6d413b1477d74e \
    --overlay-name=rpi_generic \
    --extra-kernel-arg=net.ifnames=0 \
    --extra-kernel-arg=-console \
    --extra-kernel-arg=console=ttyS1

However the Pi 5 ends up in an endless boot loop. It gets past the u-boot screen and four raspberries show up where the tuxes would normally be, then I get a red LED for a moment and a reboot.

My guess is that the Talos kernel configs are not really compatible with the Raspberry kernel, so the next attempt would be to try with a config based on the current stable 6.6.51, but that would (a) definitely not pass hardening checks, which is OK for a lab setup, but (b) potentially miss important configs like EBPF which would be required for Cilium, so further tweaking would be necessary.

The config diff between Talos and Raspberry kernel configs is well over 6000 settings flags so I see little hope of finding a combination that both boots and is still usable by Talos :(

attilaolah commented 2 weeks ago

So building the Pi kernel with the Pi config leads to… interesting results:

PXL_20241113_203738608~2

Aside from the weird yellow screen, at least the bootloader is able to start the kernel, the USB hubs are registered, and it gets so far as to start the init binary, which then reboots because, hugepages are not enabled :(

So now, enabling CONFIG_HUGETLBFS, fixing up any config issues with olddefconfig, rebuilding the entire kernel (painfully slow), building a new imager, generating new boot artifacts, preparing the SD card… a long journey ahead still, I fear.

thielepaul commented 2 weeks ago

Not sure if it helps, but here is a fork of the mainline kernel, where there seems to be active work ongoing to get the RP1 working: https://github.com/6by9/linux/commits/mainline_2712_integration/

Another project that might be interesting to test: https://github.com/worproject/rpi5-uefi It neither has ethernet support, but at least USB should be working, so a usb-ethernet adapter should be working

attilaolah commented 2 weeks ago

It is helpful, thanks! I knew about the rpi5-uefi project, but as far as I can see the onboard Ethernet adapter is not working there still; it would be an option with the USB Ethernet adapter, and the official Talos kernel + RPi overlay; I haven't tried that yet.

The other Kernel fork I haven't seen yet, I think that is also definitely worth a try. I'll give it a go when I have some more spare time.

In the end, I think that is the direction I need to take. Enabling CONFIG_HUGETLBFS did not get me any further; somehow I still end up with the same error.

So next up I'll try to flash the UEFI on one of the Pis, and maybe try that other kernel fork with another Pi.

For what it's worth, https://github.com/6by9/linux is building nicely so far, passes all hardening checks, but is a lot newer than both Talos and the Pi OS kernel apparently, currently identifying itself as 6.12-rc6.

attilaolah commented 2 weeks ago

I tried the kernel at https://github.com/6by9/linux/commits/mainline_2712_integration/, at commit 5c6ae15de0a27c7783818bd184233afac8bae011. Talos boots up and gets stuck at the usual NTP lookup error, neither the Ethernet port nor the USB port is loaded.

However, I haven't added any new modules to hack/modules-arm64.txt, I'm suspecting there is a module there that's needed to access these devices? Would it make sense to do an lsmod in Pi OS to try and figure out which one is responsible for the RP1, and then load that explicitly?

thielepaul commented 2 weeks ago

Unfortunately, I don't know the answers to your questions, but I guess 6by9 would be able to help you. You could ask him in the raspberrypi forum: https://forums.raspberrypi.com/viewtopic.php?p=2262592#p2262592

attilaolah commented 2 weeks ago

I asked on the forum for the configs that I might be missing. As for the modules, I took the sledgehammer approach and added them all to the image, and tried booting again — still no luck. Among others, ax88179_178a is now included in my image, which is needed for my USB Ethernet adapter.

I guess once I get the Ethernet to work in some way, I can fire up a privileged container and inspect all loaded modules to find out which ones are in use, and remove the unnecessary ones. But I'm guessing I'm also missing some configs to get there.

While waiting for the answer from the forum, I'm considering trying another build, taking the union of the Pi OS config and the Talos config, i.e. pick all configs that are set to whatever value in either one, and see if that gets me any further.

attilaolah commented 1 week ago

Apparently 6.12 has landed, with some more of the Pi 5 patches upstreamed — might be worth trying to rebuild from stock 6.12 and see how far that gets us. Although I expect some config-arm64 tweaking is still required.

In the meantime, response from the Pi forum was to try and merge with these configs:

I'm somewhat preoccupied these days but I'll give it another go soon. I've been using the scripts/kconfig/merge_config.sh script to merge configs with the ones that Talos used, in case anyone else wants to give it a try.

attilaolah commented 1 week ago

I tried to to merge the 16k pages config with both mainline 6.12 and the fork by 6by9, they both end up in a boot loop due to the missing loopback device, just like it was described in https://github.com/siderolabs/talos/issues/3621 — but I don't see how the issue was resolved in that case, I'm sure I'm missing a module but there's no loop.ko in the generated image. Which makes sense since CONFIG_BLK_DEV_LOOP is =y not =m, and I also have CONFIG_BLK_DEV_LOOP_MIN_COUNT=8. I wonder if something special is needed to enable the loopback devices in the newer kernel?

awillis commented 1 week ago

I'm completely new to Talos, trying to get up to speed and see where I can help.

I can build an SD card image

$ sudo podman run --rm -it \
-v $PWD/out:/secureboot:ro \
-v $PWD/out:/out \
-v /dev:/dev \
--privileged ghcr.io/siderolabs/imager:v1.8.3 metal \
--arch arm64 \
--extra-kernel-arg=net.ifnames=0  \
--extra-kernel-arg=-console  \
--extra-kernel-arg=console=ttyS1

I pulled the latest 6.12 kernel and the kernel configs for bcm2711 and 2712, merged them into .config and compiled.

$ ./scripts/kconfig/merge_config.sh arch/arm64/configs/bcm2711_defconfig arch/arm64/configs/bcm2712_defconfig

IIUC, I need to create an overlay that includes this kernel image + mods and config.txt changes? I see the configuration for rpi_generic above, but I don't fully grok how they are created and applied. I have a USB ethernet adapter from J5 to help me work around the NTP issue, works pretty well on linux and fbsd.

I'm going to peruse the kernel config and read through the sbc-raspberrypi repo.

@attilaolah Could you drop me a hint on next steps?

attilaolah commented 1 week ago

My setup is a bit clunky but it might still give you some pointers.

I start with my fork of https://github.com/siderolabs/pkgs, which has the kernel package.

If you modified the Kernel config (e.g. after merging), you can clean it up by running make kernel-olddefconfig in the repo root. When you merge configs, it makes sense to revert a few of them, like CONFIG_LOCALVERSION back to -talos, and CONFIG_CMDLINE, since the Pi config hard-codes the root fs there.

Then build the kernel. I use namespace.so since I don't have a decent host for building arm64 binaries, emulation takes forever (or just fails due to bad Qemu settings or whatever), plus the folks at Namespace Labs really like it when folks use their arm builders to compile the Linux kernel. It makes their day. :smile_cat:

So I set the registry:

export REGISTRY="$(nsc workspace describe --output=json | jq -r .registry_url)"

Make Docker buildx is configured to use Namespace:

nsc docker buildx setup --background --use

And finally compile the kernel:

make kernel REGISTRY=$REGISTRY PLATFORM=linux/arm64 PUSH=true

Obviously this takes forever, but at least the compilation isn't happening locally. When done, the image is pushed to the nscr.io registry. You can skip the Namespace part and build locally, the image will be left in the build cache if you omit PUSH=true, or you can just use Docker Hub or sign in to ghcr.io with your GitHub token (but then change USERNAME to your GitHub user since you can't push to siderolabs/…).

Once this is done, I can pull the image, alternatively tag it (but not necessary), then the next step is to build the imager. Do this from the https://github.com/siderolabs/talos repo.

For the imager, I first want to look at the kernel modules within the kernel container, so I do something like this:

crane export attilaolah/talos-kernel:v1.9.0-alpha.0-54-gcbb968d --platform=linux/arm64 | tar tf - | grep '^lib/modules/.*\.ko$' | sed 's:.*-talos/::' >> hack/modules-arm64.txt

Then I take a look at hack/modules-arm64.txt and remove the duplicates (:sort u in vim), check whether required modules are there, do a git diff maybe, and finally build the imager, referencing the kernel package from the previous step:

make imager REGISTRY="$REGISTRY" PKG_KERNEL=tag-fromskernel-image-from-previous-step PUSH=true INSTALLER_ARCH=arm64 PLATFORM=linux/arm64

Notice that I'm building from PLATFORM=linux/arm64? That's because I haven't figured out how to build an amd64 imager for only the linux/arm64 target. When I try to build the imager for amd64, it tries to pull the kernel for the linux/amd64 platform, which I don't have, since in the previous step I only compiled for linux/arm64.

Anyway, I build the imager and then, since it is an arm64 build, I SSH into my trusty Pi 5 running Pi OS, and, on the Pi itself, run the imager using Docker/Podman, similarly to your comment, but with these differences:

The only overlay I use is the sbc-raspberrypi, there I didn't change much, I just picked the one that was available at the time:

--overlay-image ghcr.io/siderolabs/sbc-raspberrypi:v0.1.0-beta.2@sha256:900bfd99ae1ed13e8fc1ba2d130c80d3c4730370fbc0f2c8330dab91471991bc
--overlay-name=rpi_generic

Then I decompress the generated compressed image (maybe there's a flag to skip compression? Dunno.) Then I dd the raw image to a second SD carad on the Pi, using a USB SD card adapter. Power off, swap cards, power on, cross fingers and wait.

There's probably more efficient ways of doing this but this works well enough.

pl4nty commented 1 week ago

Haven't touched this in over 6 months, but in case it's useful - here's the CI setup I had to build pkgs for a different SBC. At one point I built an amd64 imager targeting arm64, then discarded it because the patchset was too large for frequent rebases and I had an Ampere VM anyway. Nico has a pretty nice CI too, with image build/upload

pkgs with namespace.so only build kernel/U-Boot pkgs talos with namespace.so override kernel/U-Boot

awillis commented 1 week ago

Looks like the raspberry pi kernel is up to 6.6.62 as is the current siderolabs/pkgs repo as of 5 days ago

https://github.com/raspberrypi/linux/commit/c1036e4f14d03aba549cdd9b186148d331013056 https://github.com/siderolabs/pkgs/commit/a5530cf13d299e2722015a512cd4134db575bfc7

I'm going to point my pkgs repo at this version of the raspberry pi kernel and see what I get. The mainlined patches to 6.12.x do not include RP1 support, which is pretty critical for RPi5 support.

I have a SolidRun HoneyComb (lx2160a) box that I use for my kids' minecraft servers. Going to use it to build kernels while they are sleeping :o)