Configuring the Linux kernel so that it recognises OP-TEE

[huntc]

Hi, I'm attempting to configure OP-TEE for my custom Microchip SAMA5D27-WLSOM1 board.

I see some other questions relating to OP-TEE and the Microchip boards, but a difference here is that at91bootstrap3 launches the Linux kernel directly i.e. there is no u-boot involved.

I’m able to use buildroot to configure and build OP-TEE such that at91bootstrap3 is able to invoke it. But I’m unsure what to configure with the Linux kernel. The Microchip documentation doesn’t appear to have a recipe for this, only some examples for other boards.

Here are some log file snippets:

AT91Bootstrap 4.0.6 (2024-06-22 23:47:29)

SD/MMC: Image: Read file zImage to 0x26f00000
MMC: ADMA supported
SD: Card Capacity: High or Extended
SD: Specification Version 3.0X
SD/MMC: dt blob: Read file sama5d27_wlsom1_cuprous_gw.dtb to 0x21000000

KERNEL: Booting zImage ...
DT: Using device tree in place at 0x21000000

KERNEL: Starting linux kernel ..., machid: 0xffffffff

SD/MMC: Done to load image
SD/MMC: Image: Read file tee.bin to 0x20000000
Starting OP-TEE, Run at 0x20000000, Non-secure entry at 0x26f00000
…
I/TC: Embedded DTB found
I/TC: OP-TEE version: Unknown_4.2 (gcc version 13.2.1 20231009 (Arm GNU Toolchain 13.2.rel1 (Build arm-13.7))) #1 Sat Jun 22 13:01:44 UTC 2024 arm
I/TC: WARNING: This OP-TEE configuration might be insecure!
I/TC: WARNING: Please check https://optee.readthedocs.io/en/latest/architecture/porting_guidelines.html
I/TC: Primary CPU initializing
...
I/TC: Primary CPU switching to normal world boot

…and then that’s it i.e. the kernel doesn’t appear to boot. I have a working buildroot config without OP-TEE.

I have enabled kernel support for OP-TEE, but I'm wondering what else must be done. I'm suspecting that something device-tree related may need to be done.

Thanks for any help or pointers.

[jenswi-linaro]

So you load both the kernel and OP-TEE using AT91Bootstrap, and when OP-TEE exits to the normal world it enters the kernel directly instead of a normal world boot loader? You must make sure that r0-r3 holds the values needed by the kernel. This is something that a bootloader normally prepares, but now AT91Bootstrap must tell OP-TEE these values. Compare with that core/arch/arm/plat-ti/main.c does with struct plat_nsec_ctx.

[huntc]

At91bootstrap normally invokes the kernel for me directly instead of invoking uboot. Are you saying I must do something to op-tee so that it can invoke the kernel?

[jenswi-linaro]

Yes, and preferably in a way that OP-TEE only passes on values received from the loader. @nodeax, do you have any advice?

[nodeax]

I have not tried op-tee without u-boot. I don't have any advice off the top of my head but I will try dig into the at91bootstrap hand-off code and get back later this week.

@clementleger for possible suggestions.

[huntc]

I've had an initial look at at91bootstrap and it does appear to set up r0, r1 and r2, which I think is the requirement. This appears to be the relevant bit of code. The optee_init_nw_params structure is set up with r0, r1 and r2, and then used later by optee_load. optee_load appears to set up the registers and then jump into OP-TEE. So, it appears that OP-TEE gets these registers set up at least.

[huntc]

After spending a few hours on this today, I'm of the opinion that OP-TEE must be informed to pass on r0, r1 and r2, and that this isn't a requirement for when invoking U-Boot. As it is valid for at91bootstrap to bypass U-Boot then I think this should be supported by OP-TEE. I'd like to suggest a PR for this with some further guidance. However, at this stage, I'm not sure where these registers would require setting up, but I'm thinking it'd be around an overload ofinit_sec_mon, along with some assembly? Is the security monitor referred to in this context the Linux kernel? Thanks.

[clementleger]

@tprrt, Any idea ? (I'm not working anymore on this project)

[tprrt]

Hi,

I'm facing the same issue, since a few days, when I switched from u-boot-at91 to u-boot-mch.

I advised Microchip about it, I shall also investigate this issue.

Kind regards, Thomas

[huntc]

I'm facing the same issue, since a few days, when I switched from u-boot-at91 to u-boot-mch.

Am I right in thinking that jumping to u-boot shouldn't be an issue though i.e. it doesn't require those registers being set up?

Thanks for having followed up with Microchip.

[huntc]

Hey @tprrt - just wondering if you were able to find anything suspicious in your investigations.

Having studied the OP-TEE code today, I'm not seeing why calling into Linux should be different from calling U-Boot - the registers appear ok. My understanding is that the smc call here should switch to the monitor, which should then switch to the OS given the register setup.

I'm assuming that there's no special configuration required on the Linux kernel itself; at least to demonstrate that it can be jumped into from OP-TEE. Is that right?

Any other thoughts or ideas? Thanks.

[tprrt]

Hello @huntc,

I think it is an U-Boot issue, because on my setup no issue occurs, when I'm swapping the U-Boot binary with a binary built with an old U-boot-at91-devel branch.

Kind regards, Thomas

[huntc]

@tprrt - But I'm not using U-Boot.... I'm trying to launch the Linux kernel directly from OP-TEE. :-)

[tprrt]

Hello @huntc,

@tprrt - But I'm not using U-Boot.... I'm trying to launch the Linux kernel directly from OP-TEE. :-)

Sorry, I did not see. Moreover, I never test to do that.

Have you adapted the dts of your board to use scmi, psci and devices drived by OP-TEE ? Something like that: arch/arm/boot/dts/microchip/sama5d2-optee.dtsi

On wlsom1 and som1 evk, this dtsi is included in dts to override clk nodes and to disable device nodes managed by optee:

diff --git a/arch/arm/boot/dts/microchip/at91-sama5d27_som1_ek.dts b/arch/arm/boot/dts/microchip/at91-sama5d27_som1_ek.dts                                                                                                                    
index 08f0d4b995ff..5136139bcca7 100644                                                                                                                                                                                                       
--- a/arch/arm/boot/dts/microchip/at91-sama5d27_som1_ek.dts                                                                                                                                                                                   
+++ b/arch/arm/boot/dts/microchip/at91-sama5d27_som1_ek.dts                                                                                                                                                                                   
@@ -9,6 +9,7 @@                                                                                                                                                                                                                               
  */                                                                                                                                                                                                                                          
 /dts-v1/;                                                                                                                                                                                                                                    
 #include "at91-sama5d27_som1.dtsi"                                                                                                                                                                                                           
+#include "sama5d2-optee.dtsi"                                                                                                                                                                                                                
 #include <dt-bindings/mfd/atmel-flexcom.h>                                                                                                                                                                                                   
 #include <dt-bindings/gpio/gpio.h>                                                                                                                                                                                                           
 #include <dt-bindings/input/input.h>                                                                                                                                                                                                         
--   

Kind regards, Thomas

[huntc]

@tprrt - But I'm not using U-Boot.... I'm trying to launch the Linux kernel directly from OP-TEE. :-)

Sorry, I did not see. Moreover, I never test to do that.

Thanks for the reply. I like to pioneer the way. ;-) As having at91bootstrap boot the Linux kernel directly in place of U-Boot is a valid thing to do, it'd be great to understand how it can be achieved. I'm sure I'm not the first person to have the problem, but I'm happy to put in what effort is required to see this through. U-Boot adds considerably to the boot time of devices, and isn't necessary for our embedded scenario.

Have you adapted the dts of your board to use scmi, psci and devices drived by OP-TEE ? Something like that: arch/arm/boot/dts/microchip/sama5d2-optee.dtsi

I have done this - and modifed the Linux kernel kconfig to apply settings per the Microchip examples. Here's my dtb that's used by at91bootstrap, and (I think) available to OP-TEE:

$ dtc -I dtb output/images/sama5d27_wlsom1_cuprous_gw.dtb | grep optee
<stdout>: Warning (simple_bus_reg): /ahb/apb/ptc@fc060000: simple-bus unit address format error, expected "800000"
<stdout>: Warning (interrupt_provider): /ahb/apb/interrupt-controller@fc020000: Missing #address-cells in interrupt provider
<stdout>: Warning (interrupt_provider): /ahb/apb/pinctrl@fc038000: Missing #address-cells in interrupt provider
        optee_core@20000000 {
        optee_shm@21000000 {
        optee {
            compatible = "linaro,optee-tz";

...and here's evidence of that dtb being picked up:

RomBOOT
Backup mode enabled

AT91Bootstrap 4.0.6 (2024-06-28 16:56:34)

SD/MMC: Image: Read file zImage to 0x26f00000
MMC: ADMA supported
SD: Card Capacity: High or Extended
SD: Specification Version 3.0X
SD/MMC: dt blob: Read file sama5d27_wlsom1_cuprous_gw.dtb to 0x21000000    <<< HERE'S MY DTB

KERNEL: Booting zImage ...
DT: Using device tree in place at 0x21000000

KERNEL: Starting linux kernel ..., machid: 0xffffffff

SD/MMC: Done to load image
SD/MMC: Image: Read file tee.bin to 0x20000000
Starting OP-TEE, Run at 0x20000000, Non-secure entry at 0x26f00000
I/TC: 
I/TC: Embedded DTB found
I/TC: OP-TEE version: Unknown_4.2 (gcc version 13.2.1 20231009 (Arm GNU Toolchain 13.2.rel1 (Build arm-13.7))) #1 Fri Jun 28 07:24:45 UTC 2024 arm
I/TC: WARNING: This OP-TEE configuration might be insecure!
I/TC: WARNING: Please check https://optee.readthedocs.io/en/latest/architecture/porting_guidelines.html
I/TC: Primary CPU initializing
I/TC: Primary CPU switching to normal world boot

Even without that though, I'd expect to some evidence of Linux booting. I'm obviously doing something fundamentally wrong. Do you have any further recommendations re. debug?

Also, did Microchip respond? I can also raise a support case with them, but I suspect this is the best place to have my OP-TEE issues resolved, and for me to be able to contribute any fixes required.

Thanks for the help so far.

[huntc]

I don't know if this helps, but the Microchip example has CONFIG_ATMEL_PM disabled. However, I don't seem to be able to disable that unless CONFIG_PM is also disabled (which I don't think it is with the Microchip example).