buildroot Incorrect selection of kernel headers: expected 5.16.x, got 5.14.x (was: ULX3S -- kernel panic on boot)

[sebeaumont]

--============= Liftoff! ===============--

OpenSBI v0.8-1-gecf7701
   ____                    _____ ____ _____
  / __ \                  / ____|  _ \_   _|
 | |  | |_ __   ___ _ __ | (___ | |_) || |
 | |  | | '_ \ / _ \ '_ \ \___ \|  _ < | |
 | |__| | |_) |  __/ | | |____) | |_) || |_
  \____/| .__/ \___|_| |_|_____/|____/_____|
        | |
        |_|

Platform Name       : LiteX / VexRiscv-SMP
Platform Features   : timer,mfdeleg
Platform HART Count : 8
Boot HART ID        : 0
Boot HART ISA       : rv32imas
BOOT HART Features  : time
BOOT HART PMP Count : 0
Firmware Base       : 0x40f00000
Firmware Size       : 124 KB
Runtime SBI Version : 0.2

MIDELEG : 0x00000222
MEDELEG : 0x0000b101
[    0.000000] Linux version 5.14.0 (florent@panda) (riscv32-buildroot-linux-gnu-gcc.br_real (Buildroot 2021.08-381-g279167ee8d) 10.3.0, GNU ld (GNU Binutils) 2.36.1) #1 SMP Tue Sep 21 12:57:31 CEST 2021
[    0.000000] earlycon: liteuart0 at I/O port 0x0 (options '')
[    0.000000] Malformed early option 'console'
[    0.000000] earlycon: liteuart0 at MMIO 0xf0001000 (options '')
[    0.000000] printk: bootconsole [liteuart0] enabled
[    0.000000] kobject ((ptrval)): tried to init an initialized object, something is seriously wrong.
[    0.000000] CPU: 0 PID: 0 Comm: swapper Not tainted 5.14.0 #1
[    0.000000] Call Trace:
[    0.000000] [<c0003b64>] dump_backtrace+0x2c/0x3c
[    0.000000] Unable to handle kernel paging request at virtual address ffffffff
[    0.000000] Oops [#1]
[    0.000000] CPU: 0 PID: 0 Comm: swapper Not tainted 5.14.0 #1
[    0.000000] epc : unflatten_dt_nodes+0x374/0x568
[    0.000000]  ra : unflatten_dt_nodes+0x36c/0x568
[    0.000000] epc : c03fbde0 ra : c03fbdd8 sp : c06a3db0
[    0.000000]  gp : c072e8d8 tp : c06a8740 t0 : 00000000
[    0.000000]  t1 : 000000cf t2 : 00000000 s0 : c06a3f30
[    0.000000]  s1 : c3ffeb20 a0 : ffffffff a1 : c0ef003c
[    0.000000]  a2 : 00000000 a3 : 00000001 a4 : c3ffeb50
[    0.000000]  a5 : ffffffff a6 : 000000ec a7 : 00000011
[    0.000000]  s2 : 00000000 s3 : c0ef003c s4 : c3ffea50
[    0.000000]  s5 : c3ffeafc s6 : c0ef005c s7 : c0ef0000
[    0.000000]  s8 : c0673c58 s9 : 00000000 s10: c3ffea50
[    0.000000]  s11: c3ffeb51 t3 : 00000000 t4 : 00000000
[    0.000000]  t5 : 00000001 t6 : ffffffff
[    0.000000] status: 00000100 badaddr: ffffffff cause: 0000000f
[    0.000000] [<c03fbde0>] unflatten_dt_nodes+0x374/0x568
[    0.000000] [<c03fc09c>] __unflatten_device_tree+0xc8/0x184
[    0.000000] [<c05aaa20>] unflatten_device_tree+0x3c/0x5c
[    0.000000] [<c0591c48>] setup_arch+0x100/0x558
[    0.000000] [<c058eee8>] start_kernel+0x8c/0x63c
[    0.000000] random: get_random_bytes called from oops_exit+0x48/0x94 with crng_init=0
[    0.000000] ---[ end trace 0000000000000000 ]---
[    0.000000] Kernel panic - not syncing: Attempted to kill the idle task!
[    0.000000] ---[ end Kernel panic - not syncing: Attempted to kill the idle task! ]---

Any hints? I'm using freshly built gateware nothing fancy ./make.py --build --board ulx3s --device LFE5U-85F but with binary blob of images. I'll try with a fresh build of buildroot

[sebeaumont]

Same ballgame -- indeed same gateware I stuck on sd card -- boots ok when I use my trusty raspberry pi4...

[   76.946099] mmc0: error -22 whilst initialising MMC card
[   76.969776] litex-mmc f0004800.mmc: Requested clk_freq=0: set to 195312 via div=256
Starting syslogd: OK
Starting klogd: OK
Running sysctl: OK
Saving random seed: [  184.725613] random: dd: uninitialized urandom read (512 bytes read)
OK
Starting network: OK

Welcome to Buildroot
buildroot login: root
                   __   _
                  / /  (_)__  __ ____ __
                 / /__/ / _ \/ // /\ \ /
                /____/_/_//_/\_,_//_\_\
                      / _ \/ _ \
   __   _ __      _  _\___/_//_/         ___  _
  / /  (_) /____ | |/_/__| | / /____ __ / _ \(_)__ _____  __
 / /__/ / __/ -_)>  </___/ |/ / -_) \ // , _/ (_-</ __/ |/ /
/____/_/\__/\__/_/|_|____|___/\__/_\_\/_/|_/_/___/\__/|___/
                  / __/  |/  / _ \
                 _\ \/ /|_/ / ___/
                /___/_/  /_/_/
  32-bit RISC-V Linux running on LiteX / VexRiscv-SMP.

login[72]: root login on 'console'
root@buildroot:~# 

😀

[sebeaumont]

BTW: building buildroot according to README.md I'm getting Incorrect selection of kernel headers: expected 5.16.x, got 5.14.x not sure where to adjust this.

[Dolu1990]

BR2_PACKAGE_HOST_LINUX_HEADERS_CUSTOM_5_14=y in https://github.com/litex-hub/linux-on-litex-vexriscv/tree/master/buildroot/configs :)

If that works, can you open a PR ?

[sebeaumont]

@Dolu1990 If that means starting over with the build... I maybe some time...

[Dolu1990]

Yes ^^ likely the best is to restart the build, they maybe ways to avoid it, but for sanity reasons, best to take the safe long road first XD

[sebeaumont]

• It builds ok -- so I guess that's a patch. I'll leave it until tomorrow to see if it boots ;-)
• I've not forked the repo yet do you want to do the PR?
• Is it worth considering an update to a later kernel?

[Dolu1990]

Thanks for the PR ^^

Is it worth considering an update to a later kernel?

I'm not sure who is handeling the litex linux kernel fork. @mithro what is the situation on that side ?

[geertu]

https://github.com/litex-hub/linux/tree/litex-rebase is at v5.18-rc3, and working fine on my OrangeCrab.

[Dolu1990]

Then i guess it would be fine to update the kernel.

In some ways, myself, i'm avoiding any unecessary update as things may break secretly, and then it is literaly hell to figure out where the issue come from XD

Developping a CPU and getting it into a stable state can easly turn into madness / paranoia ^^

[enjoy-digital]

Thanks @sebeaumont, @Dolu1990 for looking at this. We can stick to v5.14 for now. I'll do a test with v5.18 and if working, will switch the project to it. (Otherwise @gsomlo has been maintaining the Linux Kernel fork).

[gsomlo]

On Mon, May 02, 2022 at 03:53:43AM -0700, enjoy-digital wrote:

(Otherwise @gsomlo has been maintaining the Linux Kernel fork).

I've been "curating" a list of LiteX specific out-of-tree kernel patches at https://github.com/litex-hub/linux/tree/litex-rebase

I'm testing (about once per day) the subset of those applicable to 64-bit Rocket configurations of LiteX.

If the tree contains an obvious bug that manifests on vexriscv (or any other LiteX-supported CPU model besides Rocket) I'm happy to apply additional patches tested by other members of the LiteX community.

[Dolu1990]

@gsomlo ok thanks :) By the way, will you be in https://open-src-soc.org/2022-05/ ?

[enjoy-digital]

The initial issue has been fixed thanks to @sebeaumont's PRs, we can probably close this.