Panic while booting BL32 after reboot

[olekslemberg]

I am trying to run the image on stm32mp157f-dk2, and Everything works well on the first boot, but after the power cycle or reset button pressed, I constantly get the exception every boot:

NOTICE:  CPU: STM32MP157FAC Rev.Z
NOTICE:  Model: STMicroelectronics STM32MP157F-DK2 Discovery Board
NOTICE:  Board: MB1272 Var4.0 Rev.C-02
INFO:    Reset reason (0x14):
INFO:      Pad Reset from NRST
INFO:    PMIC version = 0x20
INFO:    FCONF: Reading TB_FW firmware configuration file from: 0x2ffe2000
INFO:    FCONF: Reading firmware configuration information for: stm32mp_io
INFO:    Using SDMMC
INFO:      Instance 1
INFO:    Boot used partition fsbl1
NOTICE:  BL2: v2.4-r2.2(debug):reboot-16103-g09ee81d829-dirty
NOTICE:  BL2: Built : 15:56:58, Feb 22 2023
INFO:    BL2: Doing platform setup
INFO:    RAM: DDR3-DDR3L 16bits 533000kHz
INFO:    Memory size = 0x20000000 (512 MB)
INFO:    BL2: Loading image id 31
INFO:    Loading image id=31 at address 0x2ffff000
INFO:    Image id=31 loaded: 0x2ffff000 - 0x2ffff226
INFO:    FCONF: Reading FW_CONFIG firmware configuration file from: 0x2ffff000
INFO:    FCONF: Reading firmware configuration information for: dyn_cfg
INFO:    FCONF: Reading firmware configuration information for: stm32mp1_firewall
INFO:    BL2: Loading image id 4
INFO:    Loading image id=4 at address 0x2ffc5000
INFO:    Image id=4 loaded: 0x2ffc5000 - 0x2ffd97ec
WARNING: Not a known TEE, use default loading options.
INFO:    BL2: Skip loading image id 21
INFO:    BL2: Skip loading image id 22
INFO:    BL2: Loading image id 23
INFO:    Loading image id=23 at address 0xc0500000
INFO:    Image id=23 loaded: 0xc0500000 - 0xc051d39c
INFO:    BL2: Loading image id 26
INFO:    Loading image id=26 at address 0x2ffc0000
INFO:    Image id=26 loaded: 0x2ffc0000 - 0x2ffc4761
INFO:    BL2: Loading image id 5
INFO:    Loading image id=5 at address 0xc0100000
INFO:    Image id=5 loaded: 0xc0100000 - 0xc01cf870
NOTICE:  BL2: Booting BL32
INFO:    Entry point address = 0x2ffc5000
INFO:    SPSR = 0x1d3
PANIC at PC : 0x2ffcd5a3

Exception mode=0x00000016 at: 0x2ffcd5a3

After some investigations, I found that the kernel or some other module/application is changing the 'fip' partition (where the SP-min and u-boot are stored) around addresses 0x174800 - 0x1767f0 (at the bin). If I try to boot the u-boot console and then reset, there is no exception; but if I let it boot the kernel and reset, it hangs up.

If I rewrite only 'fip' partition on the sd card (after exception), the devkit will boot up one more time.

[dhdhxji]

(It's my second account) May be any thoughs here? I can test anything and ofc will help in debuging=)

[olekslemberg]

Update: I unpacked the FIP binary and noticed that the only tos-fw-config.bin is changed, where is device tree stored used by SP-MIN (BL32) Even more, when the FIP from build image contains valid device tree blob, the dumped one has invalid format and can't be converted with dtc -I dtb -O dts tos-fw-config.bin

[namezis]

Thanks for reporting this issue. I will try to replicate it.

[olekslemberg]

UPD2: It looks like the device tree is overwritten while executing the u-boot boot command (I did not allow the kernel to load this time). Also, the device tree is replaced with u-boot environment

diff <(hexdump -C build_fip/tos-fw-config.bin) <(hexdump -C dump_fip/tos-fw-config.bin) > tos-fw-config.diff

...
< 000028b0  00 00 00 06 00 00 00 02  00 00 00 01 6e 61 6e 64  |............nand|
< 000028c0  5f 6f 74 70 40 32 34 00  00 00 00 03 00 00 00 08  |_otp@24.........|
< 000028d0  00 00 00 38 00 00 00 24  00 00 00 04 00 00 00 03  |...8...$........|
---
> 000008e0  00 00 58 ce 5a b5 01 61  6c 74 62 6f 6f 74 63 6d  |..X.Z..altbootcm|
> 000008f0  64 3d 72 75 6e 20 62 6f  6f 74 63 6d 64 00 61 72  |d=run bootcmd.ar|
> 00000900  63 68 3d 61 72 6d 00 61  75 74 6f 6c 6f 61 64 3d  |ch=arm.autoload=|
> 00000910  6e 6f 00 62 61 75 64 72  61 74 65 3d 31 31 35 32  |no.baudrate=1152|
> 00000920  30 30 00 62 6f 61 72 64  3d 73 74 6d 33 32 6d 70  |00.board=stm32mp|
> 00000930  31 00 62 6f 61 72 64 5f  69 64 3d 30 78 31 32 37  |1.board_id=0x127|
> 00000940  32 00 62 6f 61 72 64 5f  6e 61 6d 65 3d 73 74 6d  |2.board_name=stm|
> 00000950  33 32 6d 70 31 35 37 66  2d 64 6b 32 00 62 6f 61  |32mp157f-dk2.boa|
...

Also, noticed the following error, which may be related to this bug (after u-boot boot command)

Kernel image @ 0xc2000000 [ 0x000000 - 0x63ec90 ]
## Flattened Device Tree blob at c4000000
   Booting using the fdt blob at 0xc4000000
ERROR: reserving fdt memory region failed (addr=da000000 size=4000000 flags=4)
   Loading Device Tree to cffe3000, end cffffd0b ... OK

Starting kernel ...

[olekslemberg]

One more update. The saveenv command in the u-boot brokes everything; it writes the environment to the wrong place. I will look into it later this week (I searched briefly, but currently not found the "entry point" of it yet)

[olekslemberg]

Probably the CONFIG_ENV_OFFSET address is wrong (int the u-boot/configs/stm32mp15_trusted_defconfig), so I see the following options here: debug it and change this offset address or even disable the storing environment

[namezis]

Yes, uboot env is not configured correctly.

I will reconfigure it to use EXT4 /boot partition. Let's see how it goes

[namezis]

Ok, so it is now adjusted.

The uboot env is located on the EXT4 MMC 0:4 in the directory /boot.

On the first boot, the environment file is created.

I did not want to create any more partitions.

Tested with initial condition no env, fw_printenv from OpenWRT, fw_setenv from OpenWRT, listing env in u-boot, and OpenWRT reboots.