Closed ricardosalveti closed 3 years ago
Changes in meta-lmp to enable OP-TEE on uz3eg-iocc:
diff --git a/meta-lmp-bsp/recipes-bsp/arm-trusted-firmware/arm-trusted-firmware_2020.1.bbappend b/meta-lmp-bsp/recipes-bsp/arm-trusted-firmware/arm-trusted-firmware_2020.1.bbappend
index b856881..bf4aa65 100644
--- a/meta-lmp-bsp/recipes-bsp/arm-trusted-firmware/arm-trusted-firmware_2020.1.bbappend
+++ b/meta-lmp-bsp/recipes-bsp/arm-trusted-firmware/arm-trusted-firmware_2020.1.bbappend
@@ -2,4 +2,4 @@
PROVIDES += "virtual/trusted-firmware-a"
# Enable opteed as the main SPD provider (required for optee)
-#EXTRA_OEMAKE_append_zynqmp = " SPD=opteed"
+EXTRA_OEMAKE_append_zynqmp = " SPD=opteed"
diff --git a/meta-lmp-bsp/recipes-bsp/u-boot/u-boot-ostree-scr-fit/uz/boot.cmd b/meta-lmp-bsp/recipes-bsp/u-boot/u-boot-ostree-scr-fit/uz/boot.cmd
index 7b2ab39..c38db48 100644
--- a/meta-lmp-bsp/recipes-bsp/u-boot/u-boot-ostree-scr-fit/uz/boot.cmd
+++ b/meta-lmp-bsp/recipes-bsp/u-boot/u-boot-ostree-scr-fit/uz/boot.cmd
@@ -9,11 +9,12 @@ setenv devnum 0
setenv loadaddr 0x10000000
setenv fdt_addr 0x1f000000
+setenv fdt_high 0x7e000000
setenv optee_ovl_addr 0x22000000
setenv bootcmd_load_f 'ext4load ${devtype} ${devnum}:2 ${loadaddr} "/boot"${kernel_image}'
-setenv bootcmd_tee_ovy 'imxtract ${loadaddr}#conf@${fdtfile} fdt@${fdt_file} ${fdt_addr}; fdt addr ${fdt_addr}; fdt resize 0x1000;'
-#setenv bootcmd_tee_ovy 'imxtract ${loadaddr}#conf@${fdtfile} fdt@${fdt_file} ${fdt_addr}; fdt addr ${fdt_addr}; fdt resize 0x1000; fdt apply ${optee_ovl_addr}'
+#setenv bootcmd_tee_ovy 'imxtract ${loadaddr}#conf@${fdtfile} fdt@${fdt_file} ${fdt_addr}; fdt addr ${fdt_addr}; fdt resize 0x1000;'
+setenv bootcmd_tee_ovy 'imxtract ${loadaddr}#conf@${fdtfile} fdt@${fdt_file} ${fdt_addr}; fdt addr ${fdt_addr}; fdt resize 0x1000; fdt apply ${optee_ovl_addr}'
setenv bootcmd_run 'bootm ${loadaddr}#conf@${fdt_file} ${loadaddr}#conf@${fdt_file} ${fdt_addr}'
setenv bootcmd_rollbackenv 'setenv kernel_image ${kernel_image2}; setenv bootargs ${bootargs2}'
setenv bootcmd_set_rollback 'if test ! "${rollback}" = "1"; then setenv rollback 1; setenv upgrade_available 0; saveenv; fi'
This is using meta-xilinx https://github.com/Xilinx/meta-xilinx/commit/fa72812585023bba487502d89b16656246ce6086 and meta-xilinx-tools https://github.com/Xilinx/meta-xilinx-tools/commit/d7b24fdfaa7627b90bf8e94eddd00f2a74cb585a.
To reproduce just write https://ci.foundries.io/projects/lmp/builds/1069/other-uz3eg-iocc/artifacts/lmp-base-console-image-uz3eg-iocc.wic.gz on a sdcard and copy over boot.bin boot.itb u-boot.itb from http://rsalveti.net/tmp/xilinx/218/ into the first (vfat) partition, replacing the older files.
Kernel config available at https://ci.foundries.io/projects/lmp/builds/1069/other-uz3eg-iocc/artifacts/other/kernel-config.
DTB available at https://ci.foundries.io/projects/lmp/builds/1069/other-uz3eg-iocc/artifacts/other/system.dtb.
Early boot log diff:
## Checking hash(es) for Image uboot ... sha256+ OK
## Checking hash(es) for Image ubootfdt ... sha256+ OK
## Checking hash(es) for Image optee ... sha256+ OK
-NOTICE: ATF running on XCZU3EG/silicon v4/RTL5.1 at 0xfffea000
+NOTICE: ATF running on XCZU3EG/silicon v4/RTL5.1 at 0x1000
NOTICE: BL31: v2.2(release):xilinx_rebase_v2.2_2020.1
NOTICE: BL31: Built : 00:00:00, Jan 1 1970
+I/TC:
+I/TC: Non-secure external DT found
+I/TC: OP-TEE version: 3.10.0-52-g9393a879 (gcc version 10.2.0 (GCC)) #1 Thu Jan 1 00:00:00 UTC 1970 aarch64
+I/TC: Primary CPU initializing
+I/TC: Primary CPU switching to normal world boot
One interesting piece here is that ATF gets loaded into a different offset when built with SPD or DEBUG.
atf/plat/xilinx/zynqmp/include/platform_def.h:
/*******************************************************************************
* BL31 specific defines.
******************************************************************************/
/*
* Put BL31 at the top of the Trusted SRAM (just below the shared memory, if
* present). BL31_BASE is calculated using the current BL31 debug size plus a
* little space for growth.
*/
#ifndef ZYNQMP_ATF_MEM_BASE
#if !DEBUG && defined(SPD_none)
# define BL31_BASE 0xfffea000
# define BL31_LIMIT 0xffffffff
#else
# define BL31_BASE 0x1000
# define BL31_LIMIT 0x7ffff
#endif
#else
# define BL31_BASE (ZYNQMP_ATF_MEM_BASE)
# define BL31_LIMIT (ZYNQMP_ATF_MEM_BASE + ZYNQMP_ATF_MEM_SIZE - 1)
# ifdef ZYNQMP_ATF_MEM_PROGBITS_SIZE
# define BL31_PROGBITS_LIMIT (ZYNQMP_ATF_MEM_BASE + ZYNQMP_ATF_MEM_PROGBITS_SIZE - 1)
# endif
#endif
Can confirm this is not really related to OP-TEE, can reproduce the same problem without OP-TEE by just building ATF in debug mode (which also forces it to a different offset, outside trusted sram).
So this looks like a bug in ATF itself.
Changed the base and offset to still get it to fit at the trusted SRAM and I'm able to confirm that it all works with OP-TEE as long it is there.
So the actual bug is getting all of this to work when ATF is allocated at dram instead.
Used a different offset and size for ATF (without conflicting with SPL) and everything works fine now.
ATF_MEM_BASE = "0xfffe5000"
ATF_MEM_SIZE = "0x1affe"
(see https://github.com/foundriesio/meta-lmp/issues/218#issuecomment-706319382, the issue here is ATF/trusted ram related instead)
uz3eg-iocc build https://ci.foundries.io/projects/lmp/builds/1069/other-uz3eg-iocc/artifacts/lmp-base-console-image-uz3eg-iocc.wic.gz works correctly, and the kernel is able to boot without any issues. This build uses SPL -> ATF -> U-Boot proper -> kernel/initramfs fitImage.
Once OP-TEE is enabled (without really doing much), the kernel fails to boot fully as the system stalls, giving a ton of 'rcu_sched detected stalls' messages. This behavior seems similar to https://forums.xilinx.com/t5/Embedded-Linux/zynqmp-rcu-INFO-rcu-sched-detected-stalls-on-CPUs-tasks/td-p/939934, but in that case OP-TEE doesn't seems to be enabled, so while similar it might not be the same.
Boot log without OP-TEE:
Boot log with OP-TEE enabled: