Kernel panic on mpfs-dev-cli-icicle-kit-es image (SD card)

Hello,

I have only added a layer (https://github.com/NobuoTsukamoto/meta-tensorflow-lite) and added some packages to the image. Other than that, the build is still original. H/W: Icicle Kit rev 1.0 with the 2021.11 reference design.

I get the following kernel log:

[5.459010] HSS_Boot_PMPSetupHandler(): Hart1 setup complete

U-Boot 2021.07 (Jul 05 2021 - 15:11:28 +0000)

CPU:   rv64imafdc
Model: Microchip PolarFire-SoC Icicle Kit
DRAM:  1.7 GiB
MMC:   mmc@20008000: 0
Loading Environment from nowhere... OK
In:    serial@20100000
Out:   serial@20100000
Err:   serial@20100000
Net:   eth0: ethernet@20112000
Hit any key to stop autoboot:  0
switch to partitions #0, OK
mmc0 is current device
Scanning mmc 0:1...
Found U-Boot script /boot.scr.uimg
1263 bytes read in 20 ms (61.5 KiB/s)
## Executing script at 88100000
' - try 'help'd '
6541208 bytes read in 300 ms (20.8 MiB/s)
' - try 'help'd '
## Copying 'fdt-microchip_microchip-mpfs-icicle-kit.dtb' subimage from FIT image at 88300000 ...
sha256+    Loading part 253 ... OK
## Loading kernel from FIT Image at 88300000 ...
   Using 'conf-microchip_microchip-mpfs-icicle-kit.dtb' configuration
   Trying 'kernel-1' kernel subimage
     Description:  Linux kernel
     Type:         Kernel Image
     Compression:  gzip compressed
     Data Start:   0x883000fc
     Data Size:    6525827 Bytes = 6.2 MiB
     Architecture: RISC-V
     OS:           Linux
     Load Address: 0x80200000
     Entry Point:  0x80200000
     Hash algo:    sha256
     Hash value:   a6d72b57aa3205e31251baba09fc131f8783c62e1c1f8e16537639f35b194693
   Verifying Hash Integrity ... sha256+ OK
## Flattened Device Tree blob at 88000000
   Booting using the fdt blob at 0x88000000
   Uncompressing Kernel Image
   Loading Device Tree to 000000008fff9000, end 000000008ffff426 ... OK

Starting kernel ...

[    0.696100] printk: console [ttyS0] enabled
[    0.700452] printk: bootconsole [sbi0] disabled
[    0.706521] 20102000.serial: ttyS1 at MMIO 0x20102000 (irq = 17, base_baud = 9375000) is a 16550A
[    0.717168] 20104000.serial: ttyS2 at MMIO 0x20104000 (irq = 18, base_baud = 9375000) is a 16550A
[    0.727534] 20106000.serial: ttyS3 at MMIO 0x20106000 (irq = 19, base_baud = 9375000) is a 16550A
[    0.737729] [drm] radeon kernel modesetting enabled.
[    0.760666] loop: module loaded
[    0.947954] u-dma-buf udmabuf-ddr-c0: driver version = 3.2.4
[    0.953729] u-dma-buf udmabuf-ddr-c0: major number   = 249
[    0.959298] u-dma-buf udmabuf-ddr-c0: minor number   = 0
[    0.964676] u-dma-buf udmabuf-ddr-c0: phys address   = 0x00000000ae000000
[    0.971531] u-dma-buf udmabuf-ddr-c0: buffer size    = 33554432
[    0.977529] u-dma-buf udmabuf0: driver installed.
[    4.000586] u-dma-buf udmabuf-ddr-nc0: driver version = 3.2.4
[    4.006407] u-dma-buf udmabuf-ddr-nc0: major number   = 249
[    4.012057] u-dma-buf udmabuf-ddr-nc0: minor number   = 1
[    4.017515] u-dma-buf udmabuf-ddr-nc0: phys address   = 0x00000000c0000000
[    4.024482] u-dma-buf udmabuf-ddr-nc0: buffer size    = 134217728
[    4.030630] u-dma-buf udmabuf1: driver installed.
[    4.102346] u-dma-buf udmabuf-ddr-nc-wcb0: driver version = 3.2.4
[    4.108510] u-dma-buf udmabuf-ddr-nc-wcb0: major number   = 249
[    4.114531] u-dma-buf udmabuf-ddr-nc-wcb0: minor number   = 2
[    4.120331] u-dma-buf udmabuf-ddr-nc-wcb0: phys address   = 0x00000000d8000000
[    4.127635] u-dma-buf udmabuf-ddr-nc-wcb0: buffer size    = 134217728
[    4.134127] u-dma-buf udmabuf2: driver installed.
[    4.142218] microsemi-mss-spi 20108000.spi: Microsemi SPI Controller 0 up
[    4.150005] microsemi-mss-spi 20109000.spi: Microsemi SPI Controller 1 up
[    4.157953] microsemi-mss-qspi 21000000.qspi: Microsemi QSPI Controller 2 up
[    4.166122] libphy: Fixed MDIO Bus: probed
[    4.199074] libphy: MACB_mii_bus: probed
[    4.204185] macb 20110000.ethernet eth0: Cadence GEM rev 0x0107010c at 0x20110000 irq 28 (00:04:a3:b1:b2:3e)
[    4.215600] libphy: MACB_mii_bus: probed
[    4.222478] macb 20112000.ethernet eth1: Cadence GEM rev 0x0107010c at 0x20112000 irq 34 (00:04:a3:b1:b2:3d)
[    4.234057] can-uio 2010c000.can: registered device
[    4.239320] pdma-uio 3000000.pdma: Running Probe
[    4.246176] pdma-uio 3000000.pdma: Registered 8 devices
[    4.251740] mss-dma-uio 60020000.fpgadma: Running Probe
[    4.257503] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[    4.264145] ehci-pci: EHCI PCI platform driver
[    4.268730] ehci-platform: EHCI generic platform driver
[    4.274217] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
[    4.280490] ohci-pci: OHCI PCI platform driver
[    4.285061] ohci-platform: OHCI generic platform driver
[    4.291076] usbcore: registered new interface driver cdc_acm
[    4.296810] cdc_acm: USB Abstract Control Model driver for USB modems and ISDN adapters
[    4.304986] usbcore: registered new interface driver usb-storage
[    4.311415] microchip-musb 20201000.usb: Registered MPFS MUSB driver
[    4.318343] musb-hdrc musb-hdrc.1.auto: MUSB HDRC host driver
[    4.324196] musb-hdrc musb-hdrc.1.auto: new USB bus registered, assigned bus number 1
[    4.333088] hub 1-0:1.0: USB hub found
[    4.336984] hub 1-0:1.0: 1 port detected
[    4.341884] mousedev: PS/2 mouse device common for all mice
[    4.348071] mpfs_rtc 20124000.rtc: Microchip Polarfire SoC RTC
[    4.354428] mpfs_rtc 20124000.rtc: registered as rtc0
[    4.359630] mpfs_rtc 20124000.rtc: setting system clock to 2000-01-01T00:00:00 UTC (946684800)
[    4.368570] i2c /dev entries driver
[    4.373209] microchip-mpfs-i2c 44000000.i2c: Microchip I2C Probe Complete
[    4.380607] microchip-mpfs-i2c 2010a000.i2c: Microchip I2C Probe Complete
[    4.388261] microchip-mpfs-i2c 2010b000.i2c: Microchip I2C Probe Complete
[    4.395434] usbcore: registered new interface driver uvcvideo
[    4.401577] sdhci: Secure Digital Host Controller Interface driver
[    4.407857] sdhci: Copyright(c) Pierre Ossman
[    4.412324] sdhci-pltfm: SDHCI platform and OF driver helper
[    4.418412] usbcore: registered new interface driver usbhid
[    4.424093] usbhid: USB HID core driver
[    4.428222] mpfs-mailbox 37020000.mailbox: Registered MPFS mailbox controller driver
[    4.436976] pac193x 2-0010: Chip revision: 0xFF
[    4.455634] mmc0: SDHCI controller on 20008000.mmc [20008000.mmc] using ADMA 64-bit
[    4.571828] pac193x 2-0010: :pac193x_prep_iio_channels: Channel 0 active
[    4.578621] pac193x 2-0010: :pac193x_prep_iio_channels: Channel 1 active
[    4.585389] pac193x 2-0010: :pac193x_prep_iio_channels: Channel 2 active
[    4.592145] pac193x 2-0010: :pac193x_prep_iio_channels: Channel 3 active
[    4.598906] pac193x 2-0010: :pac193x_prep_iio_channels: Active chip channels: 25
[    4.616024] NET: Registered protocol family 10
[    4.622776] Segment Routing with IPv6
[    4.626669] sit: IPv6, IPv4 and MPLS over IPv4 tunneling driver
[    4.633702] NET: Registered protocol family 17
[    4.638389] Key type dns_resolver registered
[    4.643402] mpfs-sys-controller soc:syscontroller: Registered MPFS system controller driver
[    4.652490] mpfs-generic-service soc:sysserv: Successfully registered mpfs generic_service driver
[    4.662919] mpfs-rng soc:hwrandom: Successfully registered HW RNG
[    4.670684] Waiting for root device /dev/mmcblk0p3...
[    4.749213] mmc0: new ultra high speed DDR50 SDHC card at address 0001
[    4.757867] mmcblk0: mmc0:0001 0016G 15.2 GiB
[    4.771056]  mmcblk0: p1 p2 p3
[    4.797085] JBD2: no valid journal superblock found
[    4.802066] EXT4-fs (mmcblk0p3): error loading journal
[    4.809102] random: fast init done
[    4.812599] exFAT-fs (mmcblk0p3): invalid boot record signature
[    4.818630] exFAT-fs (mmcblk0p3): failed to read boot sector
[    4.824372] exFAT-fs (mmcblk0p3): failed to recognize exfat type
[    4.830512] List of all partitions:
[    4.834100] b300        15917056 mmcblk0
[    4.834118]  driver: mmcblk
[    4.841003]   b301           21624 mmcblk0p1 ec856a63-fe9d-4d2b-ac3f-ec9255e396ed
[    4.841019]
[    4.850046]   b302            8192 mmcblk0p2 c5340137-c876-4a1b-bc88-ec345895435a
[    4.850061]
[    4.859080]   b303         8610695 mmcblk0p3 c2501d4f-1d42-4e60-bcde-964618e4862d
[    4.859095]
[    4.868152] No filesystem could mount root, tried:
[    4.868162]  ext3
[    4.873094]  ext2
[    4.875026]  ext4
[    4.876977]  vfat
[    4.878906]  msdos
[    4.880852]  exfat
[    4.882874]
[    4.886412] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(179,3)
[    4.894917] CPU: 1 PID: 1 Comm: swapper/0 Not tainted 5.12.19 #1
[    4.900956] Hardware name: Microchip PolarFire-SoC Icicle Kit (DT)
[    4.907180] Call Trace:
[    4.909652] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000940
[    4.918513] Oops [#1]
[    4.920798] Modules linked in:
[    4.923879] CPU: 1 PID: 1 Comm: swapper/0 Not tainted 5.12.19 #1
[    4.929918] Hardware name: Microchip PolarFire-SoC Icicle Kit (DT)
[    4.936143] epc : walk_stackframe+0xb0/0xc6
[    4.940387]  ra : dump_backtrace+0x1c/0x24
[    4.944527] epc : ffffffe0000047bc ra : ffffffe0000047ee sp : ffffffef7fe5fc10
[    4.951829]  gp : ffffffe0012f2ec8 tp : ffffffef7fe68000 t0 : ffffffe001301097
[    4.959127]  t1 : ffffffe001301088 t2 : 0000000000000000 s0 : ffffffef7fe5fc60
[    4.966427]  s1 : ffffffe000ed1f78 a0 : 0000000000001000 a1 : 0000000000000000
[    4.973726]  a2 : ffffffe0007bc61c a3 : ffffffe000ed1f78 a4 : 0000000000000000
[    4.981028]  a5 : 0000000000001000 a6 : 0000000000000005 a7 : ffffffffffffffff
[    4.988328]  s2 : 0000000000000000 s3 : 0000000000000000 s4 : 0000000000000000
[    4.995628]  s5 : ffffffe000ed1f78 s6 : ffffffe0007bc61c s7 : ffffffe000e59ce0
[    5.002926]  s8 : ffffffe0012f4980 s9 : ffffffe0012f40f8 s10: ffffffe0012f4100
[    5.010225]  s11: ffffffffffffffea t3 : 0000000000000043 t4 : ffffffffffffffff
[    5.017525]  t5 : 0000000000000000 t6 : ffffffef7fe5f9c8
[    5.022872] status: 0000000200000100 badaddr: 0000000000000940 cause: 000000000000000d
[    5.030867] [<ffffffe0000047bc>] walk_stackframe+0xb0/0xc6
[    5.036405] [<ffffffe0000047ee>] dump_backtrace+0x1c/0x24
[    5.041840] [<ffffffe0007bc66c>] show_stack+0x2c/0x38
[    5.046936] [<ffffffe0007bee8e>] dump_stack+0x72/0x8c
[    5.052020] [<ffffffe0007bc83a>] panic+0xf0/0x2aa
[    5.056752] [<ffffffe0008012d6>] mount_block_root+0x340/0x35e
[    5.062531] [<ffffffe000801372>] mount_root+0x7e/0x88
[    5.067611] [<ffffffe0008014be>] prepare_namespace+0x142/0x184
[    5.073494] [<ffffffe000800e30>] kernel_init_freeable+0x200/0x21c
[    5.079625] [<ffffffe0007c9126>] kernel_init+0x12/0xf8
[    5.084814] [<ffffffe000003286>] ret_from_exception+0x0/0xc
[    5.090473] ---[ end trace 5cf6b712a0d62d8b ]---
[    5.095163] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b
[    5.102900] SMP: stopping secondary CPUs
[    5.106870] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b ]---

And the HSS log:

[3.925186] HSS_E51_Banner(): PolarFire(R) SoC Hart Software Services (HSS) - version 0.99.26-dev-build
MPFS HAL version 1.8.131 / Mi-V IHC version 0.1.0(c) Copyright 2017-2021 Microchip FPGA Embedded Systems Solutions.

[3.946365] HSS_E51_Banner(): incorporating OpenSBI - version 0.9
(c) Copyright 2019-2021 Western Digital Corporation.

[3.958777] HSS_PrintBuildId(): Build ID: a1a221886cfa1687d3f76a2e83a13ae485a0456e
[3.967804] HSS_PrintToolVersions(): Built with the following tools:
 - riscv64-unknown-elf-gcc (xPack GNU RISC-V Embedded GCC (Microsemi SoftConsole build), 64-bit) 8.3.0
 - GNU ld (xPack GNU RISC-V Embedded GCC (Microsemi SoftConsole build), 64-bit) 2.32

[3.992282] HSS_E51_Banner(): NOTICE: Running from L2LIM

[3.999226] Device_Serial_Number_Init(): Serial Number:
3db2b1ccea54d27f82d1335a63c9bb7900000000000000000000000000000000000000000000000000000000000000000000
[4.14850] HSS_DDRPrintSegConfig(): Segment Configuration:
        Cached: SEG0_0: offset 0x0080000000, physical DDR 0x00000000
        Cached: SEG0_1: offset 0x1000000000, physical DDR 0x30000000
    Non-cached: SEG1_2: offset 0x00c0000000, physical DDR 0x70000000
Non-cached WCB: SEG1_4: offset 0x00d0000000, physical DDR 0x70000000
[4.46098] HSS_DDRPrintL2CacheWaysConfig(): L2 Cache Configuration:
    L2-Scratchpad:  4 ways (512 KiB)
         L2-Cache:  8 ways (1024 KiB)
           L2-LIM:  4 ways (512 KiB)
[4.63805] HSS_MemTestDDRFast(): DDR-Lo size is 768 MiB
[4.70228] HSS_MemTestDDRFast(): DDR-Hi size is 1 GiB
[4.76478] HSS_BoardLateInit(): Please ensure that jumpers J34/J43 are correct for 1.8V MMC voltage...
[4.87328] HSS_MMCInit(): Attempting to select SDCARD ... Passed
Press a key to enter CLI, ESC to skip
Timeout in 1 second
..
[5.228800] HSS_TinyCLI_Parser(): CLI boot interrupt timeout
[5.235657] HSS_IHCInit(): Initializing Mi-V IHC
[5.241473] IPI_QueuesInit(): Initializing IPI Queues (6056 bytes @ 0x8035708)...
[5.250153] HSS_PMP_Init(): Initializing PMPs
[5.255708] HSS_BootInit(): Initializing Boot Image..
[5.261958] getBootImageFromMMC_(): Preparing to copy from MMC to DDR ...
[5.270178] GPT_ValidateHeader(): Validated GPT Header ...
[5.294221] GPT_ValidatePartitionEntries(): Validated GPT Partition Entries ...
[5.302866] getBootImageFromMMC_(): Boot Partition found at index 1
[5.310331] getBootImageFromMMC_(): Attempting to read image header (1552 bytes) ...
[5.320539] copyBootImageToDDR_(): Copying 495816 bytes to 0xa0000000
[5.349547] HSS_BootInit(): Boot image passed CRC
[5.355710] HSS_BootInit(): Boot image set name: "PolarFire-SoC-HSS::U-Boot"
[5.363956] HSS_BootInit(): Boot Image registered...
[5.370119] HSS_Boot_RestartCore(): called for all harts
[5.376629] HSS_Wdog_MonitorHart(): monitoring U54_1 U54_2 U54_3 U54_4
[5.384354] RunStateMachine(): ipi_poll_service :: Init -> Monitoring
[5.392253] RunStateMachine(): boot_service(u54_1) :: Init -> SetupPMP
[5.400239] RunStateMachine(): boot_service(u54_2) :: Init -> SetupPMP
[5.408224] RunStateMachine(): boot_service(u54_3) :: Init -> SetupPMP
[5.416210] RunStateMachine(): boot_service(u54_4) :: Init -> SetupPMP
[5.424195] RunStateMachine(): usbdmsc_service :: Init -> Idle
[5.431487] RunStateMachine(): beu_service :: init -> monitoring
[5.438951] RunStateMachines():  loop 1 took 32758012 ticks (max 32758012 ticks)
[5.447805] boot_setup_pmp_onEntry(): boot_service(u54_1)::Registering domain "u-boot.bin" (hart mask 0x1e)
[5.458846] RunStateMachine(): boot_service(u54_1) :: SetupPMP -> SetupPMPComplete
[5.470755] RunStateMachine(): boot_service(u54_2) :: SetupPMP -> SetupPMPComplete
[5.480841] RunStateMachine(): boot_service(u54_3) :: SetupPMP -> SetupPMPComplete
[5.489868] RunStateMachine(): boot_service(u54_4) :: SetupPMP -> SetupPMPComplete
>> [5.499156] RunStateMachine(): boot_service(u54_1) :: SetupPMPComplete -> ZeroInit
[5.508183] RunStateMachine(): boot_service(u54_2) :: SetupPMPComplete -> ZeroInit
[5.517210] RunStateMachine(): boot_service(u54_3) :: SetupPMPComplete -> ZeroInit
[5.526238] RunStateMachine(): boot_service(u54_4) :: SetupPMPComplete -> ZeroInit
[5.535265] RunStateMachine(): boot_service(u54_1) :: ZeroInit -> Download
[5.543598] RunStateMachine(): boot_service(u54_2) :: ZeroInit -> Download
[5.551930] RunStateMachine(): boot_service(u54_3) :: ZeroInit -> Download
[5.560263] RunStateMachine(): boot_service(u54_4) :: ZeroInit -> Download
[5.568596] boot_download_chunks_onEntry(): boot_service(u54_1)::Processing boot image: "u-boot.bin"
[5.578925] RunStateMachine(): boot_service(u54_2) :: Download -> Idle
[5.586911] RunStateMachine(): boot_service(u54_3) :: Download -> Idle
[5.594896] RunStateMachine(): boot_service(u54_4) :: Download -> Idle
[5.646881] RunStateMachine(): boot_service(u54_1) :: Download -> OpenSBIInit
[5.655474] boot_opensbi_init_handler(): boot_service(u54_1)::u54_2:sbi_init 0x80200000
[5.664675] boot_opensbi_init_handler(): boot_service(u54_1)::u54_3:sbi_init 0x80200000
[5.673876] boot_opensbi_init_handler(): boot_service(u54_1)::u54_4:sbi_init 0x80200000
[5.683077] RunStateMachine(): boot_service(u54_1) :: OpenSBIInit -> Wait
[5.691323] boot_opensbi_init_onExit(): boot_service(u54_1)::u54_1:sbi_init 0x80200000
[5.700437] boot_wait_onEntry(): boot_service(u54_1)::Checking for IPI ACKs: - -
[5.709030] boot_wait_handler(): boot_service(u54_1)::Checking for IPI ACKs: ACK/IDLE ACK
[5.718404] RunStateMachine(): boot_service(u54_1) :: Wait -> Idle

Thanks, Nassim

polarfire-soc / polarfire-soc-documentation

Kernel panic on mpfs-dev-cli-icicle-kit-es image (SD card) #12