Open bhuism opened 1 year ago
Does the problematic pi work on other slots 1 2 4 or on the official carrier board?
@wenyi0421 yes, perfectly and the other 2 daughterboards with cm4 show the same symptoms, all work in all slots except slot 3. And none of them work in slot 3. But since this looks like a u-boot issue, maybe I should post in issue with them. I've wrote a howto here: https://github.com/bhuism/talos-tpi2 if you want to reproduce.
Please use the official image of the Raspberry Pi to test it, and consider replacing the CM4 Adapter V1.0 to try, this may be a hardware problem
I've replaced the adapter+cm4 already, I have 4 adapters and 3 cm4 modules, none of them work (with Talos using u-boot) in slot 3, all of them work in all other slots.
I'll try raspberry pi os
I can confirm this same issue, with the same error state (syndrome register value 0xbf000002) on any of my CM4 modules (8GB, Wifi, eMMC) connected to the node3 slot that boots using U-Boot. In my case the version packaged in the Fedora IoT 37 image.
Raspberry Pi OS lite boots just fine in node3 slot, I swapped cd4 and daughterboards now to get u-boot to work.
Maybe because node3 is connected to sata, uboot wants to start from sata, but it fails.This may require modifying the uboot startup items in the flashed OS
I'm afraid so
I gave this an attempt by building a custom u-boot image with modified boot-order and patched the pre-built talos image with it. While it managed to get into grub, it failed silently and reset anyway. I didn't find any obvious issues with the grub configuration, but i might look closer into it another day.
Maybe grub also probes SATA and fails?
At least interesting @Daedaluz , thanks for looking into it
Let me know if there is something to test. Happy to tinker around as well
I poked around a little bit again, and it looks like grub actually loads the kernel / initramfs properly, but the kernel itself makes it reset. I can manually load the kernel and the initramfs without issues, the reset happens after the boot command.
I kind of expected something from the kernel, like an oops or panic whatnot, but the only thing i get after boot command is:
EFI stub: Booting Linux Kernel...
EFI stub: Using DTB from configuration table
EFI stub: Exiting boot services...
Ideas?
Thanks @Daedaluz, this is too hard core kernel for me, btw did you know that slot/node 2 boots fine with a sata mini pcie card inserted, with the same chipset as on the tpiv2 board? namely asmedia asm 1061 chipset. Maybe this helps?
Thanks @Daedaluz, this is too hard core kernel for me, btw did you know that slot/node 2 boots fine with a sata mini pcie card inserted, with the same chipset as on the tpiv2 board? namely asmedia asm 1061 chipset. Maybe this helps?
Different for me, doesn't boot here.
I poked around a little bit again, and it looks like grub actually loads the kernel / initramfs properly, but the kernel itself makes it reset. I can manually load the kernel and the initramfs without issues, the reset happens after the boot command.
I kind of expected something from the kernel, like an oops or panic whatnot, but the only thing i get after boot command is:
EFI stub: Booting Linux Kernel... EFI stub: Using DTB from configuration table EFI stub: Exiting boot services...
Ideas?
Not much. is there something similar in a non uboot system we can compare to? Maybe report upstream at talos or uboot or both as well.
obviously, when rk1 comes out, we definitely needs talos to boot on it
obviously!
Working on this a little bit, conform: https://github.com/u-boot/u-boot/blob/master/doc/develop/crash_dumps.rst
I get:
$ echo 'Code: 350001f3 f94017e0 39400000 92401c00 (d5033fbf)' | CROSS_COMPILE=aarch64-linux-gnu- ARCH=arm64 scripts/decodecode
Code: 350001f3 f94017e0 39400000 92401c00 (d5033fbf)
All code
========
0: 350001f3 cbnz w19, 0x3c
4: f94017e0 ldr x0, [sp, #40]
8: 39400000 ldrb w0, [x0]
c: 92401c00 and x0, x0, #0xff
10:* d5033fbf dmb sy <-- trapping instruction
Code starting with the faulting instruction
===========================================
0: d5033fbf dmb sy
I hit the same issue by working on alpine based image powered by uboot. In my opinion the problem is related to uboot and sata controller. Node3 is not working because has native SATA controller connected. Both, Node1 and Node2 are working. When mpcie SATA controller is connected those nodes behave the same as Node3.
@SheGe My experience was not the same with talos, I booted talos fine with a satacontroller in the mpcie slot in node2 (and same chip a on the tpiv2 board on node 3) go figure
this issues is also reported a sidero here: https://github.com/siderolabs/talos/issues/7358
I make a custom rpi image with a logging/trace enabled u-boot, see log attached
new log and map:
I've got talos booted on node3 with a workaround, a custom u-boot.bin (and thus talos image) was needed, it's a hack though.
@bhuism I'm facing this same issue with eMMC CM4s, but also in slots 1 & 2 (as I have mini PCIe SATA cards installed there).
How did you get UART working there? Did it work out of the box? Or did you tinker with BOOT_UART
in EEPROM, config.txt
or something similar in the image flashed to eMMC?
In my case (RPI debug probe @ 115200
) output is empty.
@maxromanovsky uarts work out of the box, come to the discord chat, and search for serial debug, you can easily get serial to any node from the bmc command line
@bhuism thanks! What command do you use? I tried the following one on BMC, and the output is always empty:
# tpi --uart=get -n 1
{
"response": [{
"uart": ""
}]
}#
@maxromanovsky the serial ports of the cm4's are all connected to serial devices on the bmc, all 4. I've wrote something up here: https://github.com/bhuism/talos-tpi2#hardwired-bmc-serial-port-connections-to-nodes, you can use microcom or picocom of the bmc.
I've got talos booted on node3 with a workaround, a custom u-boot.bin (and thus talos image) was needed, it's a hack though.
Since you're already set up to build custom u-boot.bin
s, could you revert your workaround, confirm the problem still occurs, and then try this patch?
0001-pci-pcie-brcmstb-do-not-rely-on-CLKREQ-signal.patch
If it works for you, you may provide (at your option) an Acked-by:
/Reported-by:
/Tested-by:
that I will use to credit you on the patch when I submit it upstream, if you'd like.
@CFSworks will do asap
@CFSworks it does get past u-boot now and into grub, but gets stuck in booting the kernel:
Booting `A - Talos v1.4.7-dirty'
EFI stub: Booting Linux Kernel...
RPI Compute Module 4 (0xc03141)
PCIe BRCM: link up, 5.0 Gbps x1 (SSC)
PCI: Failed autoconfig bar 10
PCI: Failed autoconfig bar 14
PCI: Failed autoconfig bar 18
PCI: Failed autoconfig bar 1c
PCI: Failed autoconfig bar 20
after this boot loops
this log is not clean btw, I use picotom from the bmc (I ssh into bmc) and the lines that come back often look garbled
I tried u-boot development branche and the exact u-boot version (2023.1) talos 1.4.7 is using, incl their patches, both same result, I was using development in my patch.
(this talos image with ur patch boots fine on a normal rpi4b btw)
The pasted output is all (presumably) normal output from U-Boot.
Could you log into the BMC and have this running:
microcom -s 115200 /dev/ttyS4 | tee node3.log
...and try a boot? This should capture all of the characters into node3.log
, so that later terminal shenanigans don't overwrite earlier output.
I just reproduced this boot loop on my own hardware. I'll spend some time today seeing if this new problem is a shortcoming in my U-Boot patch or a problem in a different component of Talos.
Editing the GRUB boot entry with e
and adding the following to the kernel cmdline: earlycon=pl011,0xfe201000,115200
...allows kernel boot log output. The kernel is failing to boot, with:
[ 2.226887] pci_bus 0000:00: root bus resource [bus 00-ff]
[ 2.232470] pci_bus 0000:00: root bus resource [mem 0x600000000-0x63fffffff] (bus address [0xc0000000-0xffffffff])
[ 2.243009] pci 0000:00:00.0: [14e4:2711] type 01 class 0x060400
[ 2.249199] pci 0000:00:00.0: PME# supported from D0 D3hot
[ 2.258616] pci_bus 0000:01: supply vpcie3v3 not found, using dummy regulator
[ 2.266062] pci_bus 0000:01: supply vpcie3v3aux not found, using dummy regulator
[ 2.273635] pci_bus 0000:01: supply vpcie12v not found, using dummy regulator
[ 2.334384] brcm-pcie fd500000.pcie: link up, 5.0 GT/s PCIe x1 (SSC)
[ 2.389016] SError Interrupt on CPU1, code 0x00000000bf000002 -- SError
[ 2.389033] CPU: 1 PID: 1 Comm: swapper/0 Not tainted 6.1.44-talos #1
[ 2.389046] Hardware name: Unknown Unknown Product/Unknown Product, BIOS 2023.07-00970-gd74fa80c0a 07/01/2023
[ 2.389053] pstate: 204000c5 (nzCv daIF +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[ 2.389064] pc : pci_generic_config_read+0x64/0xf0
[ 2.389093] lr : pci_generic_config_read+0x4c/0xf0
[ 2.389107] sp : ffff80000802b7d0
[ 2.389111] x29: ffff80000802b7d0 x28: ffff3828409cf800 x27: 0000000000000001
[ 2.389128] x26: 0000000000000000 x25: 0000000000000000 x24: 0000000000000000
[ 2.389141] x23: 0000000000000000 x22: 0000000000000000 x21: ffff80000802b854
[ 2.389153] x20: 0000000000000004 x19: ffff3828409cf800 x18: 0000000000000000
[ 2.389165] x17: 6f74616c75676572 x16: 0000000000000107 x15: 072f075407470720
[ 2.389178] x14: 0730072e07350720 x13: 072f075407470720 x12: 0730072e07350720
[ 2.389191] x11: 0720072007200720 x10: 0720072007200720 x9 : ffffc99190cc7ebc
[ 2.389203] x8 : ffff80000802b578 x7 : 000000000002ffe8 x6 : 00000000000affa8
[ 2.389215] x5 : ffffc99190cc7e70 x4 : ffff80000802b854 x3 : 000000000000000b
[ 2.389227] x2 : ffff800008bc9000 x1 : 00000000deaddead x0 : ffff800008bc8000
[ 2.389242] Kernel panic - not syncing: Asynchronous SError Interrupt
[ 2.389247] CPU: 1 PID: 1 Comm: swapper/0 Not tainted 6.1.44-talos #1
[ 2.389256] Hardware name: Unknown Unknown Product/Unknown Product, BIOS 2023.07-00970-gd74fa80c0a 07/01/2023
[ 2.389262] Call trace:
[ 2.389267] dump_backtrace.part.0+0xec/0x100
[ 2.389282] show_stack+0x30/0x40
[ 2.389291] dump_stack_lvl+0x64/0x80
[ 2.389310] dump_stack+0x18/0x34
[ 2.389321] panic+0x180/0x35c
[ 2.389334] nmi_panic+0xbc/0xc0
[ 2.389345] arm64_serror_panic+0x78/0x84
[ 2.389355] do_serror+0x30/0x7c
[ 2.389365] el1h_64_error_handler+0x3c/0x70
[ 2.389379] el1h_64_error+0x78/0x7c
[ 2.389387] pci_generic_config_read+0x64/0xf0
[ 2.389400] pci_bus_read_config_dword+0xa0/0x160
[ 2.389414] pci_bus_generic_read_dev_vendor_id+0x40/0x180
[ 2.389431] pci_scan_single_device+0xb4/0x120
[ 2.389447] pci_scan_slot+0x6c/0x200
[ 2.389461] pci_scan_child_bus_extend+0x48/0x240
[ 2.389478] pci_scan_bridge_extend+0x158/0x580
[ 2.389494] pci_scan_child_bus_extend+0xd0/0x240
[ 2.389509] pci_scan_root_bus_bridge+0x6c/0xe0
[ 2.389525] pci_host_probe+0x24/0xd0
[ 2.389533] brcm_pcie_probe+0x258/0x630
[ 2.389545] platform_probe+0x70/0xcc
[ 2.389563] really_probe+0xc8/0x2e4
[ 2.389576] __driver_probe_device+0x80/0x11c
[ 2.389589] driver_probe_device+0x4c/0x120
[ 2.389601] __driver_attach+0xa4/0x170
[ 2.389614] bus_for_each_dev+0x84/0xdc
[ 2.389624] driver_attach+0x34/0x44
[ 2.389636] bus_add_driver+0x15c/0x210
[ 2.389648] driver_register+0x7c/0x13c
[ 2.389661] __platform_driver_register+0x38/0x4c
[ 2.389677] brcm_pcie_driver_init+0x30/0x64
[ 2.389689] do_one_initcall+0x60/0x270
[ 2.389700] kernel_init_freeable+0x478/0x554
[ 2.389709] kernel_init+0x30/0x140
[ 2.389723] ret_from_fork+0x10/0x20
[ 2.389738] SMP: stopping secondary CPUs
[ 2.389748] Kernel Offset: 0x499188380000 from 0xffff800008000000
[ 2.389754] PHYS_OFFSET: 0xffffc7d8c0000000
[ 2.389758] CPU features: 0x40000,2013c080,0000421b
[ 2.389765] Memory Limit: none
...which is this panic tracked upstream in the Linux kernel bug database.
This might be exacerbated by the timing of U-Boot using the PCIe RC for a while and then shutting it down later in the boot when EFI boot services are exited, but is not itself the fault of U-Boot, so I'm going to send that patch upstream now.
serial log: