OP-TEE / optee_os

Trusted side of the TEE
Other
1.6k stars 1.07k forks source link

OP-TEE developer build v3.22.0 fails to boot on new RPi 3B+ board #6284

Open RossPorter506 opened 1 year ago

RossPorter506 commented 1 year ago

The OP-TEE developer build fails to boot on my newly purchased RPi 3B+ board. I believe I've traced the issue down to a recent board revision (late 2022) that is incompatible with the old (~2019) raspberry pi firmware used in the OP-TEE build.

I used release v3.22.0 and followed the build instructions from here: https://optee.readthedocs.io/en/latest/building/devices/rpi3.html The ACT LED flashes 4 times slowly, then 7 times fast. Serial terminal shows no output. As a sanity check I flashed a recent Raspbian release (2023-05-03) which boots fine, however an older Raspbian release (2018-03-13) fails and gives the same LED pattern. I searched around and according to this thread https://forums.raspberrypi.com/viewtopic.php?t=349340 the recent RPi 3B+ rev1.4 board has hardware changes which make it incompatible with older firmware. I ran the following to check my board revision:

$ grep -e Revision -e Model /proc/cpuinfo
Revision        : a020d4
Model           : Raspberry Pi 3 Model B Plus Rev 1.4

The above thread mentions that replacing the start.elf, fixup.dat and bootcode.bin files in the boot partition with those from a newer (2021+) Raspbian release makes older Raspbian versions bootable. Unfortunately the OP-TEE developer build does not appear to be quite so easy to fix. Bumping the raspberry pi firmware repo to 1.20230405 in the manifest is enough to get further into the boot process, including actual serial output, but u-boot complains about the lack of a valid device tree (see below).

I'm not sure where to go from here. Any advice would be appreciated.

NOTICE:  Booting Trusted Firmware
NOTICE:  BL1: v2.6(debug):v2.6
NOTICE:  BL1: Built : 10:40:45, Aug 30 2023
INFO:    BL1: RAM 0x100ee000 - 0x100f7000
INFO:    BL1: cortex_a53: CPU workaround for 843419 was applied
INFO:    BL1: cortex_a53: CPU workaround for 855873 was applied
WARNING: BL1: cortex_a53: CPU workaround for 1530924 was missing!
ERROR:   rpi3: Unknown board revision 0x00a020d4
NOTICE:  rpi3: Detected: Unknown (Unknown) [0x00a020d4]
INFO:    BL1: Loading BL2
INFO:    Loading image id=1 at address 0x100b4000
INFO:    Image id=1 loaded: 0x100b4000 - 0x100bc419
NOTICE:  BL1: Booting BL2
INFO:    Entry point address = 0x100b4000
INFO:    SPSR = 0x3c5
ERROR:   rpi3_sdhost: transfer FIFO word 0: 0x21
INFO:    rpi3_sdhost: HC_COMMAND:        0x00004073
INFO:    rpi3_sdhost: HC_ARGUMENT:       0x00000000
INFO:    rpi3_sdhost: HC_TIMEOUTCOUNTER: 0x00f00000
INFO:    rpi3_sdhost: HC_CLOCKDIVISOR:   0x0000027b
INFO:    rpi3_sdhost: HC_RESPONSE_0:     0x00000920
INFO:    rpi3_sdhost: HC_RESPONSE_1:     0x00003348
INFO:    rpi3_sdhost: HC_RESPONSE_2:     0xffffffff
INFO:    rpi3_sdhost: HC_RESPONSE_3:     0x0002480c
INFO:    rpi3_sdhost: HC_HOSTSTATUS:     0x00000021
INFO:    rpi3_sdhost: HC_POWER:          0x00000001
INFO:    rpi3_sdhost: HC_DEBUG:          0x00010871
INFO:    rpi3_sdhost: HC_HOSTCONFIG:     0x00000400
INFO:    rpi3_sdhost: HC_BLOCKSIZE:      0x00000008
INFO:    rpi3_sdhost: HC_BLOCKCOUNT:     0x00000001
NOTICE:  BL2: v2.6(debug):v2.6
NOTICE:  BL2: Built : 10:40:46, Aug 30 2023
INFO:    BL2: Doing platform setup
INFO:    BL2: Loading image id 3
INFO:    Loading image id=3 at address 0x100e0000
INFO:    Image id=3 loaded: 0x100e0000 - 0x100ea073
INFO:    BL2: Loading image id 4
INFO:    Loading image id=4 at address 0x10100000
INFO:    Image id=4 loaded: 0x10100000 - 0x1010001c
INFO:    OPTEE ep=0x10100000
INFO:    OPTEE header info:
INFO:          magic=0x4554504f
INFO:          version=0x2
INFO:          arch=0x1
INFO:          flags=0x0
INFO:          nb_images=0x1
INFO:    BL2: Loading image id 21
INFO:    Loading image id=21 at address 0x10100000
INFO:    Image id=21 loaded: 0x10100000 - 0x10179338
INFO:    BL2: Skip loading image id 22
INFO:    BL2: Loading image id 5
INFO:    Loading image id=5 at address 0x11000000
INFO:    Image id=5 loaded: 0x11000000 - 0x11084b90
NOTICE:  BL1: Booting BL31
INFO:    Entry point address = 0x100e0000
INFO:    SPSR = 0x3cd
NOTICE:  BL31: v2.6(debug):v2.6
NOTICE:  BL31: Built : 10:40:46, Aug 30 2023
INFO:    rpi3: Checking DTB...
INFO:    BL31: Initializing runtime services
INFO:    BL31: cortex_a53: CPU workaround for 843419 was applied
INFO:    BL31: cortex_a53: CPU workaround for 855873 was applied
WARNING: BL31: cortex_a53: CPU workaround for 1530924 was missing!
INFO:    BL31: Initializing BL32
D/TC:0   add_phys_mem:665 VCORE_UNPG_RX_PA type TEE_RAM_RX 0x10100000 size 0x00079000
D/TC:0   add_phys_mem:665 VCORE_UNPG_RW_PA type TEE_RAM_RW 0x10179000 size 0x00287000
D/TC:0   add_phys_mem:665 ta_base type TA_RAM 0x10400000 size 0x00c00000
D/TC:0   add_phys_mem:665 CONSOLE_UART_BASE type IO_NSEC 0x3f200000 size 0x00200000
D/TC:0   add_phys_mem:665 TEE_SHMEM_START type NSEC_SHM 0x08000000 size 0x00400000
D/TC:0   add_va_space:705 type RES_VASPACE size 0x00a00000
D/TC:0   add_va_space:705 type SHM_VASPACE size 0x02000000
D/TC:0   dump_mmap_table:831 type TEE_RAM_RX   va 0x10100000..0x10178fff pa 0x10100000..0x10178fff size 0x00079000 (smallpg)
D/TC:0   dump_mmap_table:831 type TEE_RAM_RW   va 0x10179000..0x103fffff pa 0x10179000..0x103fffff size 0x00287000 (smallpg)
D/TC:0   dump_mmap_table:831 type RES_VASPACE  va 0x10400000..0x10dfffff pa 0x00000000..0x009fffff size 0x00a00000 (pgdir)
D/TC:0   dump_mmap_table:831 type SHM_VASPACE  va 0x10e00000..0x12dfffff pa 0x00000000..0x01ffffff size 0x02000000 (pgdir)
D/TC:0   dump_mmap_table:831 type NSEC_SHM     va 0x12e00000..0x131fffff pa 0x08000000..0x083fffff size 0x00400000 (pgdir)
D/TC:0   dump_mmap_table:831 type TA_RAM       va 0x13200000..0x13dfffff pa 0x10400000..0x10ffffff size 0x00c00000 (pgdir)
D/TC:0   dump_mmap_table:831 type IO_NSEC      va 0x13e00000..0x13ffffff pa 0x3f200000..0x3f3fffff size 0x00200000 (pgdir)
D/TC:0   core_mmu_xlat_table_alloc:526 xlat tables used 1 / 5
D/TC:0   core_mmu_xlat_table_alloc:526 xlat tables used 2 / 5
I/TC: 
I/TC: OP-TEE version: 3.22.0 (gcc version 11.3.1 20220712 (Arm GNU Toolchain 11.3.Rel1)) #1 Tue Aug 29 22:39:47 UTC 2023 aarch64
I/TC: WARNING: This OP-TEE configuration might be insecure!
I/TC: WARNING: Please check https://optee.readthedocs.io/en/latest/architecture/porting_guidelines.html
I/TC: Primary CPU initializing
D/TC:0 0 call_preinitcalls:21 level 2 mobj_mapped_shm_init()
D/TC:0 0 mobj_mapped_shm_init:470 Shared memory address range: 10e00000, 12e00000
D/TC:0 0 call_initcalls:40 level 1 register_time_source()
D/TC:0 0 call_initcalls:40 level 1 teecore_init_pub_ram()
D/TC:0 0 call_initcalls:40 level 3 check_ta_store()
D/TC:0 0 check_ta_store:417 TA store: "early TA"
D/TC:0 0 check_ta_store:417 TA store: "Secure Storage TA"
D/TC:0 0 check_ta_store:417 TA store: "REE"
D/TC:0 0 call_initcalls:40 level 3 early_ta_init()
D/TC:0 0 early_ta_init:56 Early TA f04a0fe7-1f5d-4b9b-abf7-619b85b4ce8c size 47637 (compressed, uncompressed 109256)
D/TC:0 0 call_initcalls:40 level 3 verify_pseudo_tas_conformance()
D/TC:0 0 call_initcalls:40 level 3 tee_cryp_init()
D/TC:0 0 call_initcalls:40 level 4 tee_fs_init_key_manager()
D/TC:0 0 call_initcalls:40 level 6 mobj_init()
D/TC:0 0 call_initcalls:40 level 6 default_mobj_init()
D/TC:0 0 call_initcalls:40 level 6 ftmn_boot_tests()
D/TC:0 0 ftmn_boot_tests:198 Calling simple_call()
D/TC:0 0 ftmn_boot_tests:198 Return from simple_call()
D/TC:0 0 ftmn_boot_tests:199 Calling two_level_call()
D/TC:0 0 ftmn_boot_tests:199 Return from two_level_call()
D/TC:0 0 ftmn_boot_tests:200 Calling chained_calls()
D/TC:0 0 ftmn_boot_tests:200 Return from chained_calls()
D/TC:0 0 ftmn_boot_tests:202 *************************************************
D/TC:0 0 ftmn_boot_tests:203 **************  Tests complete  *****************
D/TC:0 0 ftmn_boot_tests:204 *************************************************
I/TC: Primary CPU switching to normal world boot
INFO:    BL31: Preparing for EL3 exit to normal world
INFO:    Entry point address = 0x11000000
INFO:    SPSR = 0x3c9

U-Boot 2021.10 (Aug 30 2023 - 10:59:07 +1200)

DRAM:  948 MiB
RPI 3 Model B+ (0xa020d4)
MMC:   mmc@7e202000: 0, sdhci@7e300000: 1
Loading Environment from FAT... OK
In:    serial
Out:   serial
Err:   serial
Net:   No ethernet found.
starting USB...
Bus usb@7e980000: USB DWC2
scanning bus usb@7e980000 for devices... 
Error: lan78xx_eth address not set.
3 USB Device(s) found
       scanning usb for storage devices... 0 Storage Device(s) found
Hit any key to stop autoboot:  0 
20869632 bytes read in 866 ms (23 MiB/s)
ERROR: Did not find a cmdline Flattened Device Tree
Could not find a valid device tree
U-Boot> 
Chae0510 commented 1 year ago

I have also encountered the same issue and have not yet been able to resolve it.

jbech-linaro commented 1 year ago

Thanks for bring this to my attention. I do have an older RPi3B+ myself, which I use when we're making the OP-TEE releases. I cannot say why it doesn't boot when bumping the firmware. Perhaps there is a need to use another DT blob or alternative perhaps some addresses etc might have changed. Unfortunately I don't know if and when I'd have time to look into this.

Chae0510 commented 1 year ago

Thank you for responding. I was curious if you had managed to resolve it, but it seems like there's still a roadblock. 😔 I'll try some other methods myself! If you come across a solution, could you please let me know? Thanks!

Message ID: @.***>

jbech-linaro commented 1 year ago

I'll try some other methods myself! If you come across a solution, could you please let me know? Thanks!

Absolutely, I'll respond to this issue/thread. I'll mark this as a "bug" which will force the issue to stay open.

RossPorter506 commented 1 year ago

I've managed to get the system booting with some manual intervention. Autoboot still fails, but when you are kicked back to the u-boot command prompt you can manually load the device tree file that's in the boot partition, and then try booting again like so:

setenv fdtdevice mmc 0:1
setenv fdtfile bcm2710-rpi-3-b-plus.dtb
setenv loadaddr 0x01000000
load ${fdtdevice} ${loadaddr} ${fdtfile}
fdt addr ${loadaddr}
boot

I used 0x01000000 as it's the device_tree_address found in /boot/config.txt, so I'm presuming it should be here. Is this correct? Anyway, after (re)loading the FDT to 0x01000000 and booting it then proceeds through the Linux kernel scroll and ends on a login screen. Logging in and running xtest results in all tests passing:

27040 subtests of which 0 failed
104 test cases of which 0 failed
0 test cases were skipped
TEE test application done!

If I try to point u-boot to 0x01000000 using fdt addr 0x01000000 before (re)loading the FDT I get libfdt fdt_check_header(): FDT_ERR_BADMAGIC, so perhaps the FDT address has changed? How can I check?

RossPorter506 commented 1 year ago

I'm still not sure about a root cause, but a band-aid fix is to integrate the above commands into u-boot's autoboot process. In build/rpi3/firmware/uboot.env.txt make the following modifications:

# bootcmd & bootargs configuration
preboot=usb start
bootcmd=run mmcboot
load_kernel=fatload mmc 0:1 ${kernel_addr_r} kernel8.img
+ load_fdt=fatload mmc 0:1 ${fdt_addr_r} bcm2710-rpi-3-b-plus.dtb
- mmcboot=run load_kernel; run set_bootargs_tty set_bootargs_mmc set_common_args; run boot_it
- nfsboot=run load_kernel; run set_bootargs_tty set_bootargs_nfs set_common_args; run boot_it
+ mmcboot=run load_fdt; run load_kernel; run set_bootargs_tty set_bootargs_mmc set_common_args; run boot_it
+ nfsboot=run load_fdt; run load_kernel; run set_bootargs_tty set_bootargs_nfs set_common_args; run boot_it
mackskaren commented 1 year ago

@RossPorter506 did you change the revision in the /.repo/manifests/rpi3.xml file? I'm running into the exact same problem with the same rpi 3B+ board (same revision and model number). Also are the changes to the uboot.env.txt file and the manifest file implemented before or after running make?

RossPorter506 commented 1 year ago

@RossPorter506 did you change the revision in the /.repo/manifests/rpi3.xml file?

Yes.

Are the changes to the uboot.env.txt file and the manifest file implemented before or after running make?

Before. To clarify, first update the RPi firmware version in the .repo/manifests/rpi3.xml file from 1.20190401 to 1.20230405, then do a repo sync -m rpi3.xml -j4 --no-clone-bundle to pull the new firmware. Modify uboot.env.txt as above, and then run make as usual.

Chae0510 commented 1 year ago

Thank you for your sharing. 🥲

  1. Locate the string 1.20190401 and change it to 1.20230405

  2. repo sync -m rpi3.xml -j4 --no-clone-bundle

3.. Edit the uboot.env.txt File

After going through this process, typing make img-help instructs to enter sudo dd if=/path/to/../out/rpi-sdcard.img of=/dev/ bs=1024k conv=fsync status=progress, but entering this results in /out/rpi-sdcard.img: No such file or directory being displayed. I'm curious if you also encounter this error?

RossPorter506 commented 1 year ago

@Chae0510 I don't think it's related to this issue, but I have encountered it before. Silly question, but is there actually a rpi-sdcard.img file at that filepath? If not, then it sounds like the build failed. If you use parallel compilation (by using make -j `nproc` rather than make) sometimes build errors can be hidden by the other compilation threads. Build it with only one thread (if you aren't already) and double check.

iitKD commented 4 months ago

Any help on this ? i am also stuck at same point

jjsshh3116 commented 4 months ago

Any help on this ? i am also stuck at same point

Hello! My name is Se-hyun Jung, and I am an undergraduate research assistant working on research related to OP-TEE. I also encountered a related issue and happened to solve it, so I’ve organized the solution in a Notion link below. The post is written in Korean, but I believe you can understand it using a web page translator. (https://bead-bassoon-d51.notion.site/raspberry-Pi3-B-OP-TEE-ae055579747247138e5ac4d47ec91ae6)

To give a brief overview:

1.  Use Buildroot to create a booting SD card for the Raspberry Pi 3B+ and extract the start.elf file.
2.  Use sudo dd in OP-TEE to make an SD card, remove the existing start.elf file, and copy the start.elf file created with Buildroot onto the SD card.
3.  Insert the SD card into the Raspberry Pi 3B+ board and attempt to boot.
4.  This will take you to manual U-boot booting; entering the U-boot commands listed in Notion in sequence will successfully boot the system.

Please try this out, and if you have any further questions, leave a comment. I will help as much as I can. Sending support for your enthusiasm, Se-hyun Jung.

e-mail address: jjsshh3116@gmail.com