trusted-key-tee: "keyctl add trusted" returns "add_key: No such device"

[johndoe31415]

Hello again. I have yet another issue that I've been chasing all day and don't know why it is not working. Maybe it's a stupid thing I'm doing wrong. I'm on 3.16.0 with a 5.15.24 kernel on STM32MP1. PKCS#11 works beautifully. Now I want key encapsulation and use the OP-TEE to seal/unseal keys and push them directly into the kernel keyring. Easy, I thought, but:

keyctl add trusted kmk "new 32" @u
add_key: No such device

No information in dmesg. strace shows the respective system call fails:

add_key("trusted", "kmk", "new 32", 6, KEY_SPEC_USER_KEYRING) = -1 ENODEV (No such device)

Fair enough. I've chased down where the respective kernel module is and it's security/keys/trusted-keys/trusted_tee.c.

In shotgun approach I sprinkeled printk all over that module, especially trusted_key_probe() and trusted_tee_init(). They are never called. The TA this references

    {UUID_INIT(0xf04a0fe7, 0x1f5d, 0x4b9b,
           0xab, 0xf7, 0x61, 0x9b, 0x85, 0xb4, 0xce, 0x8c)},

is present on my system

# ls /lib/optee_armtz/f04*
-rw------- 1 root root 51K   18.01.2023 10:29:32 /lib/optee_armtz/f04a0fe7-1f5d-4b9b-abf7-619b85b4ce8c.ta

I've searched a bunch but really found very few hints. There is one bug about trusted keys not working without CFG_CORE_DYN_SHM (https://github.com/OP-TEE/optee_os/issues/5632), but I have it enabled (it's included by default).

It is my understanding that the driver is always built (and I can verify by finding my debug strings in the kernel I built) whenever CONFIG_TEE is set, which it obviously is:

# cat /proc/config.gz | gunzip | grep CONFIG_TEE
CONFIG_TEE_REMOTEPROC=y
CONFIG_TEE=y

via security/keys/trusted-keys/Makefile it therefore is included in the kernel.

The only reason I could think why the trusted-key-tee driver is not initialized would be that the device tree tells the kernel not to. This is guesswork terretory for me, however. Extracting the device tree from the itb and decompiling it shows:

        firmware {

                optee {
                        compatible = "linaro,optee-tz";
                        method = "smc";
                };

                scmi {
                        compatible = "linaro,scmi-optee";
                        #address-cells = <0x01>;
                        #size-cells = <0x00>;
                        linaro,optee-channel-id = <0x00>;
                        shmem = <0x08>;

                        protocol@14 {
                                reg = <0x14>;
                                #clock-cells = <0x01>;
                                phandle = <0x01>;
                        };

                        protocol@16 {
                                reg = <0x16>;
                                #reset-cells = <0x01>;
                                phandle = <0x3f>;
                        };
                };
        };

        psci {
                compatible = "arm,psci-1.0";
                method = "smc";
        };

Which are both standard in the STM32MP1 dtsi as well.

Is there anything that I am missing that would not register this driver in my system? Anything I need to do? The respective trusted application does not ever seem to be loaded. Otherwise I would think the keyctl would hang if there is no tee-supplicant running, but it doesn't -- it just fails with the error code as shown above.

If you have any clues, I'd be super grateful. All the best!

[etienne-lms]

IIUC the trusted key TA is not seen by linux kenrel at inits. I think you need to embed trusted key TA as an early TA (a.k.a TA bin image is stored in a OP-TEE core rodata section and locally installed on demand). Trsuted-key TA is flagged TA_FLAG_DEVICE_ENUM so when it's a built-in/early TA, OP-TEE can enumerate it to optee Linux driver which in turn can init the trusted key etc...

There is a shortkey config in optee to embed an in-source-tree TA as an early TA, here: CFG_IN_TREE_EARLY_TAS=trusted_keys/f04a0fe7-1f5d-4b9b-abf7-619b85b4ce8c

[jforissier]

Yes @etienne-lms I think you're right. A few related links:

• The place where CFG_IN_TREE_EARLY_TAS is set in the OP-TEE development/testing environment: https://github.com/OP-TEE/build/blob/3.19.0/common.mk#L527
• The trusted keys tests: https://github.com/OP-TEE/build/blob/master/trusted-keys.exp
• These tests can be run alone in the QEMU environment with: make check CHECK_TESTS=trusted-keys. More details on how to build can be found in the OP-TEE documentation.

[johndoe31415]

IIUC the trusted key TA is not seen by linux kenrel at inits. I think you need to embed trusted key TA as an early TA (a.k.a TA bin image is stored in a OP-TEE core rodata section and locally installed on demand). Trsuted-key TA is flagged TA_FLAG_DEVICE_ENUM so when it's a built-in/early TA, OP-TEE can enumerate it to optee Linux driver which in turn can init the trusted key etc...

There is a shortkey config in optee to embed an in-source-tree TA as an early TA, here: CFG_IN_TREE_EARLY_TAS=trusted_keys/f04a0fe7-1f5d-4b9b-abf7-619b85b4ce8c

Ah, thanks for the pointer! I've tried it and compiled the trusted_keys TA as an early TA, but I still get the same behavior unfortunately. To verify that I've actually done the right thing in building OP-TEE, I also deliberately tried to misspell the name on the command line, and, sure enough that causes an error:

  CHK     out/arm-plat-stm32mp1/conf.mk
  UPD     out/arm-plat-stm32mp1/conf.mk
make: *** No rule to make target 'out/arm-plat-stm32mp1/ta/xxusted_keys/f04a0fe7-1f5d-4b9b-abf7-619b85b4ce8c.stripped.elf', needed by 'out/arm-plat-stm32mp1/core/early_ta_f04a0fe7-1f5d-4b9b-abf7-619b85b4ce8c.c'.  Stop.
make: *** Waiting for unfinished jobs....

When naming it correctly, it builds fine, so I assume that the early TA is present. I don't know how to check this, however. It does not appear in /sys (I'm not sure if it should):

# ls -1 /sys/devices/ | grep optee
optee-ta-94cf71ad-80e6-40b5-a7c6-3dc501eb2803
optee-ta-a8cfe406-d4f5-4a2e-9f8d-a25dc754c099
optee-ta-ab7a617c-b8e7-4d8f-8301-d09b61036b64
optee-ta-d96a5b40-c3e5-21e3-8794-1002a5d5c61b

The respective init routines of the driver are still not called on bootup (I left the printk calls in for now). Any ideas what could still be missing?

[johndoe31415]

Yes @etienne-lms I think you're right. A few related links:

• The place where CFG_IN_TREE_EARLY_TAS is set in the OP-TEE development/testing environment: https://github.com/OP-TEE/build/blob/3.19.0/common.mk#L527
• The trusted keys tests: https://github.com/OP-TEE/build/blob/master/trusted-keys.exp
• These tests can be run alone in the QEMU environment with: make check CHECK_TESTS=trusted-keys. More details on how to build can be found in the OP-TEE documentation.

Yes I did not see this in the Makefile. Regarding the documentation, I am familiar with it but quite a few of the intricacies of getting it to run on actual hardware are not mentioned. I had to piece it a bit together from knowledge of PKCS#11 with other hardware security modules. I believe it could be helpful to have some documentation of usage on the actual applications as well and could write a few words if you'd consider it helpful for inclusion as documentation? Are you accepting PRs for this and would you think this to be a good addition to the existing documentation?

[etienne-lms]

Are you accepting PRs for this and would you think this to be a good addition to the existing documentation?

OP-TEE docs really need improvement at many places. Efforts were made but we all lacking of time... So any help to enhance the docs is really welcome. Just have to create a pull request in https://github.com/OP-TEE/optee_docs.

[jforissier]

When naming it correctly, it builds fine, so I assume that the early TA is present. I don't know how to check this, however. It does not appear in /sys (I'm not sure if it should):

Yes it should... With QEMU:

# ls -l /sys/devices | grep optee
drwxr-xr-x    3 root     root             0 Jan 19 13:59 optee-ta-d96a5b40-c3e5-21e3-8794-1002a5d5c61b
drwxr-xr-x    3 root     root             0 Jan 19 13:59 optee-ta-f04a0fe7-1f5d-4b9b-abf7-619b85b4ce8c

(the other one is a test Pseudo-TA -- PTA_INVOKE_TESTS_UUID)

The OP-TEE boot log should show something like that (with CFG_TEE_CORE_LOG_LEVEL=3):

I/TC: OP-TEE version: 3.20.0-rc1-3-gb831e57b3 (gcc version 10.2.1 20201103 (GNU Toolchain for the A-profile Architecture 10.2-2020.11 (arm-10.16))) #2 Tue Jan 17 22:35:59 UTC 2023 aarch64
...
D/TC:0 0 call_initcalls:40 level 3 early_ta_init()
D/TC:0 0 early_ta_init:56 Early TA f04a0fe7-1f5d-4b9b-abf7-619b85b4ce8c size 38123 (compressed, uncompressed 63352)

[johndoe31415]

Are you accepting PRs for this and would you think this to be a good addition to the existing documentation?

OP-TEE docs really need improvement at many places. Efforts were made but we all lacking of time... So any help to enhance the docs is really welcome. Just have to create a pull request in https://github.com/OP-TEE/optee_docs.

I think that is what I will do. Many of the questions I have are fairly typical usecases (e.g., setting up client certificate keys inside OP-TEE) and I believe it would be helpful if a few examples would be put together. I'll give that a shot.

[johndoe31415]

When naming it correctly, it builds fine, so I assume that the early TA is present. I don't know how to check this, however. It does not appear in /sys (I'm not sure if it should):

Yes it should... With QEMU:

# ls -l /sys/devices | grep optee
drwxr-xr-x    3 root     root             0 Jan 19 13:59 optee-ta-d96a5b40-c3e5-21e3-8794-1002a5d5c61b
drwxr-xr-x    3 root     root             0 Jan 19 13:59 optee-ta-f04a0fe7-1f5d-4b9b-abf7-619b85b4ce8c

(the other one is a test Pseudo-TA -- PTA_INVOKE_TESTS_UUID)

Hmmm. Something seems to be definiely wrong, but I cannot figure out what it is. Firstly, do you know of an easy way to build a specific tag using the QEMU build system (as described in the docs, using the repo tool)? What I'm doing right now is:

$ repo forall -c "git checkout 3.16.0"

After the repo sync step, but it leads to a failing build:

$ make QEMU_VIRTFS_ENABLE=y QEMU_USERNET_ENABLE=y run
[...]
table='/tmp/optee/out-br/build/buildroot-fs/full_devices_table.txt'
make[1]: Leaving directory '/tmp/optee/out-br'
make -C /tmp/optee/build/../linux LOCALVERSION= CROSS_COMPILE=" /tmp/optee/build/../toolchains/aarch32/bin/arm-linux-gnueabihf-" ARCH=arm zImage
make[1]: Entering directory '/tmp/optee/linux'
  CALL    scripts/checksyscalls.sh
  Kernel: arch/arm/boot/Image is ready
  Kernel: arch/arm/boot/zImage is ready
make[1]: Leaving directory '/tmp/optee/linux'
mkdir -p /tmp/optee/build/../out/bin
ln -sf /tmp/optee/build/../linux/arch/arm/boot/zImage /tmp/optee/build/../out/bin
make -C /tmp/optee/build/../qemu
make[1]: Entering directory '/tmp/optee/qemu'
changing dir to build for make ""...
make[2]: Entering directory '/tmp/optee/qemu/build'
config-host.mak is out-of-date, running configure
bash: line 4: ./config.status: No such file or directory
make[2]: *** No rule to make target 'config-host.mak', needed by 'meson.stamp'.  Stop.
make[2]: Leaving directory '/tmp/optee/qemu/build'
make[1]: *** [GNUmakefile:11: all] Error 2
make[1]: Leaving directory '/tmp/optee/qemu'
make: *** [Makefile:89: qemu] Error 2

Note that I can build master just fine using this tool, but I would like to test with the version I'm also running on my target to not chase ghost bugs. I've also seen that repo itself has a branch functionality but no branches seem to be present there. That said, I'm not familiar with "repo", in fact it's the first time I've used it for the OP-TEE project.

To the non-presence of the early TA in my build, right now I have no idea what is going on. The parameter I'm passing (exactly as you've described in the bug) does seem to be used, because if I misspell it on purpose I see the build fails. So it's not ignoring it. Then, to further advance my descent into madness, I tried to replicate this:

The OP-TEE boot log should show something like that (with CFG_TEE_CORE_LOG_LEVEL=3):

I/TC: OP-TEE version: 3.20.0-rc1-3-gb831e57b3 (gcc version 10.2.1 20201103 (GNU Toolchain for the A-profile Architecture 10.2-2020.11 (arm-10.16))) #2 Tue Jan 17 22:35:59 UTC 2023 aarch64
...
D/TC:0 0 call_initcalls:40 level 3 early_ta_init()
D/TC:0 0 early_ta_init:56 Early TA f04a0fe7-1f5d-4b9b-abf7-619b85b4ce8c size 38123 (compressed, uncompressed 63352)

And I absolutely cannot see any change, even with CFG_TEE_CORE_LOG_LEVEL=4 on my target. I can see the TA log just fine (still have that enabled) but I'm not seeing the log messages you're seeing.

Something seems to be off with my build and I think I need to find my inner zen again and probably fix a really stupid oversight. Let me get my bearings and I'll hopefully figure it out.

Thanks again for your tremendous help. It is truly appreciated. I am absolutely amazed.

[jforissier]

Yes, you can checkout a particular version with -b , for example:

$ cd /tmp/optee
$ repo init -u https://github.com/OP-TEE/manifest.git -m qemu_v8.xml -b 3.16.0

And I absolutely cannot see any change, even with CFG_TEE_CORE_LOG_LEVEL=4 on my target. I can see the TA log just fine (still have that enabled) but I'm not seeing the log messages you're seeing.

This clearly has to be understood. First of all, can you see debug messages from the TEE core when you set CFG_TEE_CORE_LOG_LEVEL=4? (3 is enough for debug but 4 is fine too). That is, messages with the D/TC prefix. They are logged on the secure console, may or may not be a different serial port than the default console, depending on the platform, for the STM32MP1 I don't know. If you don't see them... then could it be that you didn't re-flash the proper TEE binary?

Once you see the TEE core (kernel) debug messages, we need to understand why you would not have the D/TC ... Early ta ... line. Some possibly useful commands:

optee_os $ find out/ -name early_ta\*
out/arm/core/early_ta_f04a0fe7-1f5d-4b9b-abf7-619b85b4ce8c.o
out/arm/core/early_ta_f04a0fe7-1f5d-4b9b-abf7-619b85b4ce8c.c
out/arm/core/kernel/early_ta.o
optee_os $ find out/ -name tee.map | xargs grep -H ts_bin_f04a0fe7
out/arm/core/tee.map: .rodata.ts_bin_f04a0fe71f5d4b9babf7619b85b4ce8c
out/arm/core/tee.map:                0x000000000e16d7e3                ts_bin_f04a0fe71f5d4b9babf7619b85b4ce8c

[etienne-lms]

... I would like to test with the version I'm also running on my target to not chase ghost bugs. ...

I use the OP-TEE manifest (following build instructions). As @jforissier pointer, you can select a manifest for a specific OP-TEE tag (at least > 3.10.0), I repo sync once then I checkout some of optee_os, linux, etc.. to the git revision i want. Then "make all" from build/, It worked fine so far, but for linux kernel that needs tweak in BR config, and that "make clean" sometime fail, repalced with rm -rf ../optee_os/out ../out-br ../out and like. But maybe you're building from another build env.

And I absolutely cannot see any change, even with CFG_TEE_CORE_LOG_LEVEL=4 on my target. I can see the TA log just fine (still have that enabled) but I'm not seeing the log messages you're seeing.

Once built, check the build output filetree. In something like out/arm/conf.mk (in optee_os build dir) is the config effectively use to compile and generate OP-TEE core image and the TA devkit. Check CFG_TEE_CORE_LOG_LEVEL assigned value. If 4 then you don't program the image you generated. If 3 then your log level 4 config is overriden somewhere in your build env.

To the non-presence of the early TA in my build, ...

Looking into the repos, I saw trusted-keys is built as an early TA by build.git. Since 3.18.0, see build.git commit 45cf44c5d). I rebuilt the optee_os source for my platforms and extracted the build traces below, they show the TA is compiled & embeded as an early TA:

  (...)
  CC      out/arm/ta/trusted_keys/entry.o
  CC      out/arm/ta/trusted_keys/user_ta_header.o
  AS      out/arm/ta/trusted_keys/ta_entry_a32.o
  (...)
  CPP     out/arm/ta/trusted_keys/ta.lds
  (...)
  LD      out/arm/ta/trusted_keys/f04a0fe7-1f5d-4b9b-abf7-619b85b4ce8c.elf
  (...)
  OBJCOPY out/arm/ta/trusted_keys/f04a0fe7-1f5d-4b9b-abf7-619b85b4ce8c.stripped.elf
  OBJDUMP out/arm/ta/trusted_keys/f04a0fe7-1f5d-4b9b-abf7-619b85b4ce8c.dmp
  GEN     out/arm/core/early_ta_f04a0fe7-1f5d-4b9b-abf7-619b85b4ce8c.c
  SIGN    out/arm/ta/trusted_keys/f04a0fe7-1f5d-4b9b-abf7-619b85b4ce8c.ta
  (...)
  CC      out/arm/core/early_ta_f04a0fe7-1f5d-4b9b-abf7-619b85b4ce8c.o
  (...)
  LD      out/arm/core/tee.elf
  OBJDUMP out/arm/core/tee.dmp
  GEN     out/arm/core/tee.bin
  (...)

[johndoe31415]

Okay, so I've had a good night's sleep and retraced my steps again, using the hints provided by @jforissier and @etienne-lms in order to cross-validate what I am doing. This is ridiculous.

First, I made sure that the early TA is built (it was):

$ ls ls out/arm-plat-stm32mp1/core/early_ta_*
-rw-r--r-- 1 joe joe 138K   21.01.2023 00:10:53 out/arm-plat-stm32mp1/core/early_ta_f04a0fe7-1f5d-4b9b-abf7-619b85b4ce8c.o

And that it's linked (it also is):

$ cat out/arm-plat-stm32mp1/core/tee.map | grep ts_bin
 .rodata.ts_bin_f04a0fe71f5d4b9babf7619b85b4ce8c
                0x000000003000b50b                ts_bin_f04a0fe71f5d4b9babf7619b85b4ce8c

I also checked that I had logging enabled:

$ cat out/arm-plat-stm32mp1/conf.mk | grep LOG
CFG_CORE_TPM_EVENT_LOG=n
CFG_TEE_CORE_LOG_LEVEL=4
CFG_TEE_TA_LOG_LEVEL=3

I have to recall my earlier claim and am actually seeing some OP-TEE messages, but really few, at a very early stage in the boot process and only ever at INFO level. Something was defiinitely off. So I also modified core/arch/arm/kernel/boot.c by this:

        IMSG("OP-TEE version: %s", core_v_str);
+       IMSG("Yep, this is definitely the correct build.");

To make absolutely sure I'm not doing something stupid (like flashing the wrong partition). Not only to convide you, but also to convince myself. At boot time: no change. The message is not there. I am clearly booting the wrong version.

But this CANNOT be, I thought. Because, clever me, I am using a mechanism to flash the absolutely right partition that works both in-system and out-of-system (when I brick it, I can recover quickly using an external SD card reader). In system I have /dev/mmcblk, but out-of-system it's /dev/sd. But GPT allows me to uniquely identify partitions via their PARTUUID. That's the mechanism I used when flashing.

Unfortunately:

/dev/mmcblk0p1: PARTLABEL="fsbl1" PARTUUID="891a994e-9885-4c39-be9e-5373ab4479d5"
/dev/mmcblk1p1: PARTLABEL="fsbl1" PARTUUID="891a994e-9885-4c39-be9e-5373ab4479d5" 

When I initially flashed my SD card, I did it with the exact same stock image that also was used to flash the internal eMMC. The GPTs are identical, so are the PARTUUIDs. I don't know what udev does, but it certainly (at least sometimes) pointed to the wrong one. I feel incredibly stupid. This is the one thing I thought for sure was guaranteed and didn't double check.

Unfortunately, no happy ending just yet. Realizing my mistake I obviously now am flashing the correct partition. However, with debugging enabled, the image seems to have become too big (some debug output added to show what is happening):

NOTICE:  Bootrom authentication failed
INFO:    Reset reason (0x54):
INFO:      System reset generated by MPU (MPSYSRST)
INFO:    FCONF: Reading TB_FW firmware configuration file from: 0x2ffdd000
INFO:    FCONF: Reading firmware configuration information for: tbbr
INFO:    FCONF: Reading firmware configuration information for: stm32mp_io
INFO:    Using SDMMC
INFO:      Instance 1
INFO:    Boot used partition fsbl1
NOTICE:  BL2: v2.6-stm32mp1-r1.0(release):f33a265
NOTICE:  BL2: Built : 00:38:36, Jan 21 2023
INFO:    Using crypto library 'stm32_crypto_lib'
INFO:    BL2: Doing platform setup
INFO:    RAM: DDR3-DDR3L 16bits 533000kHz
INFO:    Memory size = 0x20000000 (512 MB)
INFO:    BL2: Loading image id 31
INFO:    Loading image id=6 at address 0x2ffff000
INFO:    Image id=6 loaded: 0x2ffff000 - 0x2ffff2ca
NOTICE:  ROTPK is not deployed on platform. Skipping ROTPK verification.
INFO:    Loading image id=31 at address 0x2ffff000
INFO:    Image id=31 loaded: 0x2ffff000 - 0x2ffff1fa
INFO:    FCONF: Reading FW_CONFIG firmware configuration file from: 0x2ffff000
INFO:    FCONF: Reading firmware configuration information for: dyn_cfg
INFO:    FCONF: Reading firmware configuration information for: stm32mp1_firewall
INFO:    BL2: Loading image id 4
INFO:    Loading image id=7 at address 0x2ffc0000
INFO:    Image id=7 loaded: 0x2ffc0000 - 0x2ffc0284
NOTICE:  ROTPK is not deployed on platform. Skipping ROTPK verification.
INFO:    Loading image id=10 at address 0x2ffc0000
INFO:    Image id=10 loaded: 0x2ffc0000 - 0x2ffc022b
INFO:    Loading image id=14 at address 0x2ffc0000
INFO:    Image id=14 loaded: 0x2ffc0000 - 0x2ffc02e3
INFO:    Loading image id=4 at address 0x2ffc0000
INFO:    Image id=4 loaded: 0x2ffc0000 - 0x2ffc002c
INFO:    We should see this
INFO:    OPTEE ep=0x2ffc0000
INFO:    OPTEE header info:
INFO:          magic=0x4554504f
INFO:          version=0x2
INFO:          arch=0x0
INFO:          flags=0x0
INFO:          nb_images=0x2
INFO:    parsing pageR image
INFO:       image_info->image_base: 0x2ffc0000
INFO:       image_info->image_max_size: 0x1d000
INFO:       image->load_addr_hi: 0x0
INFO:       image->load_addr_lo: 0x2ffc0000
INFO:       image->image_id: 0x0
INFO:       image->size: 0x20850
INFO:       sizeof init_load_addr: 4
INFO:       free_end: 0x2ffdcfff
INFO:       requested_end: 0x2ffe084f
WARNING: The load address in optee header 0x2ffc0000 - 0x2ffe0850 is not in reserved area: 0x2ffc0000 - 0x2ffdd000.
INFO:    pageR image parse result: -1
ERROR:   OPTEE header parse error.
PANIC at PC : 0x2ffe9c2f

It seems the image is now simply too big, with a max size of 0x1d000 and its size being 0x20850, it's roughly 14 kiB too heavy. So I disabled all logging, reflashed and got it to boot. Now unfortunately without any insightful messages. But I do see the early TA in /sys now:

$ ls /sys/devices/ | grep optee
drwxr-xr-x  3 root root 0   01.01.2000 01:00:12 optee-ta-94cf71ad-80e6-40b5-a7c6-3dc501eb2803/
drwxr-xr-x  3 root root 0   01.01.2000 01:00:12 optee-ta-a8cfe406-d4f5-4a2e-9f8d-a25dc754c099/
drwxr-xr-x  3 root root 0   01.01.2000 01:00:12 optee-ta-ab7a617c-b8e7-4d8f-8301-d09b61036b64/
drwxr-xr-x  3 root root 0   01.01.2000 01:00:12 optee-ta-d96a5b40-c3e5-21e3-8794-1002a5d5c61b/
drwxr-xr-x  3 root root 0   01.01.2000 01:00:12 optee-ta-f04a0fe7-1f5d-4b9b-abf7-619b85b4ce8c/

However, the trusted keys driver is still neither probed nor loaded.

Is there any way to give OP-TEE more memory? I have the pager enabled, but I think this only affects memory at runtime. I'm still attaching my out/arm-plat-stm32mp1/conf.mk to this ticket, maybe there's an easy way to get logging and get it to boot as well.

I'll call it a night now ;-) Cheers!

conf.mk.txt

[johndoe31415]

Hmm. I've enabled early TA compression, disabled all algorithms I don't seem to need, reduced the CORE log level to 3, but it's still insufficient:

INFO:    OPTEE ep=0x2ffc0000
INFO:    OPTEE header info:
INFO:          magic=0x4554504f
INFO:          version=0x2
INFO:          arch=0x0
INFO:          flags=0x0
INFO:          nb_images=0x2
INFO:    parsing pageR image
INFO:       image_info->image_base: 0x2ffc0000
INFO:       image_info->image_max_size: 0x1d000
INFO:       image->load_addr_hi: 0x0
INFO:       image->load_addr_lo: 0x2ffc0000
INFO:       image->image_id: 0x0
INFO:       image->size: 0x1f2a8
INFO:       sizeof init_load_addr: 4
INFO:       free_end: 0x2ffdcfff
INFO:       requested_end: 0x2ffdf2a7
WARNING: The load address in optee header 0x2ffc0000 - 0x2ffdf2a8 is not in reserved area: 0x2ffc0000 - 0x2ffdd000.
INFO:    pageR image parse result: -1
ERROR:   OPTEE header parse error.
PANIC at PC : 0x2ffe9c2f

The STM32MP157C should bring more than enough SRAM to get this done. So I'm assiuming the 0x1d000 (116 kiB) are the result of an non-ideal memory layout. I'll try to see what settings I have to fiddle with that and get it a bit more space to get going.

[jforissier]

Did you make sure the Linux kernel has the proper driver? CONFIG_TRUSTED_KEYS=y and maybe also CONFIG_ENCRYPTED_KEYS=y

[etienne-lms]

Regarding enumeration, try something like the below patch to see what's enumerated:

--- a/core/pta/device.c
+++ b/core/pta/device.c
@@ -29,6 +29,9 @@ static void add_ta(uint32_t flags, const TEE_UUID *uuid, uint8_t *buf,
                return;
        }

+       IMSG(PTA_NAME ": enumerate TA %pUl, %ssuppplicant dependency", uuid,
+            flags & TA_FLAG_DEVICE_ENUM_SUPP ? "" : "no ");
+
        if (flags & rflags) {
                if (*pos + sizeof(*uuid) <= blen)
                        tee_uuid_to_octets(buf + *pos, uuid);

Regarding OP-TEE memory footprint in stm32mp15, the boot issue is not related to OP-TEE embedding early TAs or their size. Early TAs content is fully pageable content, not resident. The issue you have is the pager resisdent+init sections are too big for the platform. Try level 1 (EMSG()) or 2 (IMSG()) if 3 bring too many traces in pager resident memory. Levels 1 and 2 consumes about a 4kB page of resisdent RAM which is likely affordable.

[johndoe31415]

Hey there. I've not had a lot of luck in the meantime. Here's what happened:

Did you make sure the Linux kernel has the proper driver? CONFIG_TRUSTED_KEYS=y and maybe also CONFIG_ENCRYPTED_KEYS=y

Yes, both are in there:

$ cat /proc/config.gz | gunzip | grep CONFIG_TRUSTED
# CONFIG_TRUSTED_FOUNDATIONS is not set
CONFIG_TRUSTED_KEYS=y

$ cat /proc/config.gz | gunzip | grep CONFIG_ENCRYPT
CONFIG_ENCRYPTED_KEYS=y

CORE_LOG_LEVEL=2 is still too much in my case:

INFO:    parsing pageR image
INFO:       image_info->image_base: 0x2ffc0000
INFO:       image_info->image_max_size: 0x1d000
INFO:       image->load_addr_hi: 0x0
INFO:       image->load_addr_lo: 0x2ffc0000
INFO:       image->image_id: 0x0
INFO:       image->size: 0x1e008
INFO:       sizeof init_load_addr: 4
INFO:       free_end: 0x2ffdcfff
INFO:       requested_end: 0x2ffde007
WARNING: The load address in optee header 0x2ffc0000 - 0x2ffde008 is not in reserved area: 0x2ffc0000 - 0x2ffdd000.
INFO:    pageR image parse result: -1
ERROR:   OPTEE header parse error.
PANIC at PC : 0x2ffe9c2f

I've recompiled with CORE_LOG_LEVEL=1 and changed the IMSG in your code, @etienne-lms, to EMSG so that I would be able to see it. Unfortunately, and I cannot believe this, it seems to be exactly 8 bytes too large:

INFO:    parsing pageR image
INFO:       image_info->image_base: 0x2ffc0000
INFO:       image_info->image_max_size: 0x1d000
INFO:       image->load_addr_hi: 0x0
INFO:       image->load_addr_lo: 0x2ffc0000
INFO:       image->image_id: 0x0
INFO:       image->size: 0x1d008
INFO:       sizeof init_load_addr: 4
INFO:       free_end: 0x2ffdcfff
INFO:       requested_end: 0x2ffdd007
WARNING: The load address in optee header 0x2ffc0000 - 0x2ffdd008 is not in reserved area: 0x2ffc0000 - 0x2ffdd000.
INFO:    pageR image parse result: -1
ERROR:   OPTEE header parse error.
PANIC at PC : 0x2ffe9c2f

I then disabled further algorithms (DSS, RSA-PSS, DH, etc.), left core debugging level at 1. This time it passed the pageR image, but then it hangs just a few steps further:

INFO:    parsing pageR image
INFO:       image_info->image_base: 0x2ffc0000
INFO:       image_info->image_max_size: 0x1d000
INFO:       image->load_addr_hi: 0x0
INFO:       image->load_addr_lo: 0x2ffc0000
INFO:       image->image_id: 0x0
INFO:       image->size: 0x1cfd0
INFO:       sizeof init_load_addr: 4
INFO:       free_end: 0x2ffdcfff
INFO:       requested_end: 0x2ffdcfcf
INFO:    pageR image parse result: 0
INFO:    parsing pageD image
INFO:       image_info->image_base: 0xde200000
INFO:       image_info->image_max_size: 0x1e00000
INFO:       image->load_addr_hi: 0xffffffff
INFO:       image->load_addr_lo: 0xffffffff
INFO:       image->image_id: 0x1
INFO:       image->size: 0x4d000
INFO:       sizeof init_load_addr: 4
INFO:       loader decided init_load_addr: 0xde200000
INFO:       free_end: 0xdfffffff
INFO:       requested_end: 0xde24cfff
INFO:    pageD image parse result: 0
INFO:    BL2: Loading image id 21
INFO:    Loading image id=21 at address 0x2ffc0000
INFO:    Image id=21 loaded: 0x2ffc0000 - 0x2ffdcfd0
ERROR:   BL2: Failed to load image id 21 (-80)

Does this seem right to you? That any enabling of core logging (even level 1) renders the image unusable? Seems like something is wrong there, possibly? Only without debugging do I get successful boot:

INFO:    parsing pageR image
INFO:       image_info->image_base: 0x2ffc0000
INFO:       image_info->image_max_size: 0x1d000
INFO:       image->load_addr_hi: 0x0
INFO:       image->load_addr_lo: 0x2ffc0000
INFO:       image->image_id: 0x0
INFO:       image->size: 0x19ec8
INFO:       sizeof init_load_addr: 4
INFO:       free_end: 0x2ffdcfff
INFO:       requested_end: 0x2ffd9ec7
INFO:    pageR image parse result: 0
INFO:    parsing pageD image
INFO:       image_info->image_base: 0xde200000
INFO:       image_info->image_max_size: 0x1e00000
INFO:       image->load_addr_hi: 0xffffffff
INFO:       image->load_addr_lo: 0xffffffff
INFO:       image->image_id: 0x1
INFO:       image->size: 0x4a000
INFO:       sizeof init_load_addr: 4
INFO:       loader decided init_load_addr: 0xde200000
INFO:       free_end: 0xdfffffff
INFO:       requested_end: 0xde249fff
INFO:    pageD image parse result: 0
INFO:    BL2: Loading image id 21
INFO:    Loading image id=21 at address 0x2ffc0000
INFO:    Image id=21 loaded: 0x2ffc0000 - 0x2ffd9ec8
INFO:    BL2: Loading image id 22
INFO:    Loading image id=22 at address 0xde200000
INFO:    Image id=22 loaded: 0xde200000 - 0xde24a000
INFO:    BL2: Loading image id 23
INFO:    Loading image id=23 at address 0xc0500000
INFO:    Image id=23 loaded: 0xc0500000 - 0xc0516fb0
INFO:    BL2: Skip loading image id 26
INFO:    BL2: Loading image id 5
INFO:    Loading image id=11 at address 0xc0100000
INFO:    Image id=11 loaded: 0xc0100000 - 0xc010022c
INFO:    Loading image id=15 at address 0xc0100000
INFO:    Image id=15 loaded: 0xc0100000 - 0xc0100254
INFO:    Loading image id=5 at address 0xc0100000
INFO:    Image id=5 loaded: 0xc0100000 - 0xc01bc404
NOTICE:  BL2: Booting BL32
INFO:    Entry point address = 0x2ffc0000
INFO:    SPSR = 0x1d3

So the problem still exists :-( This is quite frustrating. One idea that I had was: How does the kernel know which is the "trusted" device to use? In other words, does it choose (randomly?) the first one? What if there was OP-TEE and a TPM module, for example -- there isn't, there's only OP-TEE in my case, but how would I tell the kernel to actually use OP-TEE for the trusted key storage? In other words, is there something to let the kernel know that it should use OP-TEE?

Any ideas, even longshots, are really appreciated. I'll try out everything ;-)

[b49020]

How does the kernel know which is the "trusted" device to use? In other words, does it choose (randomly?) the first one? What if there was OP-TEE and a TPM module, for example -- there isn't, there's only OP-TEE in my case, but how would I tell the kernel to actually use OP-TEE for the trusted key storage? In other words, is there something to let the kernel know that it should use OP-TEE?

Yeah there is a kernel command line parameter for this, have a look here: https://elixir.bootlin.com/linux/v6.2-rc5/source/Documentation/admin-guide/kernel-parameters.txt#L6331

BTW, can you share kernel boot logs along with any additional prints you have added? Also, do share devices and drivers listed within /sys/bus/tee/.

[johndoe31415]

@b49020 there's a smoking gun in dmesg, but I have overlooked it!

[    3.243210] trusted_tee_init
[    3.243299] trusted_key_probe
[    3.243323] trusted_key_probe got CTX
[    3.410583] trusted-key-tee optee-ta-f04a0fe7-1f5d-4b9b-abf7-619b85b4ce8c: tee_client_open_session failed, err: ffff0000
[    3.410657] trusted-key-tee: probe of optee-ta-f04a0fe7-1f5d-4b9b-abf7-619b85b4ce8c failed with error -22
[    3.410815] Key type encrypted registered

The "0xffff0000" is TEE_ERROR_GENERIC. I will try surrounding the code with a bitof debugging to see what is going on there. I believe errno 22 is "invalid argument".

Here's my /sys/bus/tee:

$ find /sys/bus/tee/
/sys/bus/tee/
/sys/bus/tee/drivers_probe
/sys/bus/tee/devices
/sys/bus/tee/devices/optee-ta-ab7a617c-b8e7-4d8f-8301-d09b61036b64
/sys/bus/tee/devices/optee-ta-d96a5b40-c3e5-21e3-8794-1002a5d5c61b
/sys/bus/tee/devices/optee-ta-a8cfe406-d4f5-4a2e-9f8d-a25dc754c099
/sys/bus/tee/devices/optee-ta-f04a0fe7-1f5d-4b9b-abf7-619b85b4ce8c
/sys/bus/tee/devices/optee-ta-94cf71ad-80e6-40b5-a7c6-3dc501eb2803
/sys/bus/tee/uevent
/sys/bus/tee/drivers
/sys/bus/tee/drivers/tee_remoteproc
/sys/bus/tee/drivers/tee_remoteproc/bind
/sys/bus/tee/drivers/tee_remoteproc/unbind
/sys/bus/tee/drivers/tee_remoteproc/uevent
/sys/bus/tee/drivers/optee-scmi-agent
/sys/bus/tee/drivers/optee-scmi-agent/bind
/sys/bus/tee/drivers/optee-scmi-agent/unbind
/sys/bus/tee/drivers/optee-scmi-agent/optee-ta-a8cfe406-d4f5-4a2e-9f8d-a25dc754c099
/sys/bus/tee/drivers/optee-scmi-agent/uevent
/sys/bus/tee/drivers/optee-rng
/sys/bus/tee/drivers/optee-rng/optee-ta-ab7a617c-b8e7-4d8f-8301-d09b61036b64
/sys/bus/tee/drivers/optee-rng/bind
/sys/bus/tee/drivers/optee-rng/unbind
/sys/bus/tee/drivers/optee-rng/uevent
/sys/bus/tee/drivers/stm32-bsec-pta
/sys/bus/tee/drivers/stm32-bsec-pta/bind
/sys/bus/tee/drivers/stm32-bsec-pta/unbind
/sys/bus/tee/drivers/stm32-bsec-pta/uevent
/sys/bus/tee/drivers/stm32-bsec-pta/optee-ta-94cf71ad-80e6-40b5-a7c6-3dc501eb2803
/sys/bus/tee/drivers/trusted-key-tee
/sys/bus/tee/drivers/trusted-key-tee/bind
/sys/bus/tee/drivers/trusted-key-tee/unbind
/sys/bus/tee/drivers/trusted-key-tee/uevent
/sys/bus/tee/drivers_autoprobe

[b49020]

[ 3.410583] trusted-key-tee optee-ta-f04a0fe7-1f5d-4b9b-abf7-619b85b4ce8c: tee_client_open_session failed, err: ffff0000

It looks like OP-TEE isn't able to load trusted keys early TA, can you share corresponding OP-TEE logs?

[johndoe31415]

Unfortunately, every time I enable logs, even CFG_TEE_CORE_LOG_LEVEL=1, the image size becomes so big that it becomes unbootable (see above). Otherwise I would happly provide the logs :-(

Is there any way to reduce the image size to just the bare minimum so I can get logging to work?

[b49020]

Is there any way to reduce the image size to just the bare minimum so I can get logging to work?

It looks like the image size constraints you are facing are related to pager support. Can you try to disable pager support CFG_WITH_PAGER=n and debug trusted keys problem? Once you fix that then you can re-enable pager support.

[johndoe31415]

Is there any way to reduce the image size to just the bare minimum so I can get logging to work?

It looks like the image size constraints you are facing are related to pager support. Can you try to disable pager support CFG_WITH_PAGER=n and debug trusted keys problem? Once you fix that then you can re-enable pager support.

Unfortunately, this does not help. CFG_WITH_PAGER=n CFG_TEE_CORE_LOG_LEVEL=3:

INFO:    OPTEE ep=0x2ffc0000
INFO:    OPTEE header info:
INFO:          magic=0x4554504f
INFO:          version=0x2
INFO:          arch=0x0
INFO:          flags=0x0
INFO:          nb_images=0x1
INFO:    parsing pageR image
INFO:       image_info->image_base: 0x2ffc0000
INFO:       image_info->image_max_size: 0x1d000
INFO:       image->load_addr_hi: 0x0
INFO:       image->load_addr_lo: 0xde200000
INFO:       image->image_id: 0x0
INFO:       image->size: 0x74ae8
INFO:       sizeof init_load_addr: 4
INFO:       free_end: 0x2ffdcfff
INFO:       requested_end: 0xde274ae7
WARNING: The load address in optee header 0xde200000 - 0xde274ae8 is not in reserved area: 0x2ffc0000 - 0x2ffdd000.
INFO:    pageR image parse result: -1
ERROR:   OPTEE header parse error.
PANIC at PC : 0x2ffe9c2f

Then tried CFG_WITH_PAGER=n CFG_TEE_CORE_LOG_LEVEL=2:

INFO:    Image id=14 loaded: 0x2ffc0000 - 0x2ffc02e3
INFO:    Loading image id=4 at address 0x2ffc0000
INFO:    Image id=4 loaded: 0x2ffc0000 - 0x2ffc001c
INFO:    We should see this
INFO:    OPTEE ep=0x2ffc0000
INFO:    OPTEE header info:
INFO:          magic=0x4554504f
INFO:          version=0x2
INFO:          arch=0x0
INFO:          flags=0x0
INFO:          nb_images=0x1
INFO:    parsing pageR image
INFO:       image_info->image_base: 0x2ffc0000
INFO:       image_info->image_max_size: 0x1d000
INFO:       image->load_addr_hi: 0x0
INFO:       image->load_addr_lo: 0xde200000
INFO:       image->image_id: 0x0
INFO:       image->size: 0x6f8a0
INFO:       sizeof init_load_addr: 4
INFO:       free_end: 0x2ffdcfff
INFO:       requested_end: 0xde26f89f
WARNING: The load address in optee header 0xde200000 - 0xde26f8a0 is not in reserved area: 0x2ffc0000 - 0x2ffdd000.
INFO:    pageR image parse result: -1
ERROR:   OPTEE header parse error.
PANIC at PC : 0x2ffe9c2f

Finally tried CFG_WITH_PAGER=n CFG_TEE_CORE_LOG_LEVEL=1:

INFO:    OPTEE ep=0x2ffc0000
INFO:    OPTEE header info:
INFO:          magic=0x4554504f
INFO:          version=0x2
INFO:          arch=0x0
INFO:          flags=0x0
INFO:          nb_images=0x1
INFO:    parsing pageR image
INFO:       image_info->image_base: 0x2ffc0000
INFO:       image_info->image_max_size: 0x1d000
INFO:       image->load_addr_hi: 0x0
INFO:       image->load_addr_lo: 0xde200000
INFO:       image->image_id: 0x0
INFO:       image->size: 0x6e8a0
INFO:       sizeof init_load_addr: 4
INFO:       free_end: 0x2ffdcfff
INFO:       requested_end: 0xde26e89f
WARNING: The load address in optee header 0xde200000 - 0xde26e8a0 is not in reserved area: 0x2ffc0000 - 0x2ffdd000.
INFO:    pageR image parse result: -1
ERROR:   OPTEE header parse error.
PANIC at PC : 0x2ffe9c2f

I seem to be fully unable to build OP-TEE with any core logging :-(

[jforissier]

I seem to be fully unable to build OP-TEE with any core logging :-(

You could try to selectively convert a few DMSG() into IMSG(). For example the one in core/kernel/early_ta.c.

[johndoe31415]

I seem to be fully unable to build OP-TEE with any core logging :-(

You could try to selectively convert a few DMSG() into IMSG(). For example the one in core/kernel/early_ta.c.

Ah, I thought that IMSG was only shown at CFG_TEE_CORE_LOG_LEVEL=1 or upwards. That's a great idea, I'll then convert a few and compile with CFG_TEE_CORE_LOG_LEVEL=0.

[jforissier]

I seem to be fully unable to build OP-TEE with any core logging :-(

You could try to selectively convert a few DMSG() into IMSG(). For example the one in core/kernel/early_ta.c.

Ah, I thought that IMSG was only shown at CFG_TEE_CORE_LOG_LEVEL=1 or upwards. That's a great idea, I'll then convert a few and compile with CFG_TEE_CORE_LOG_LEVEL=0.

Huh, no, level 0 does silence everything. For IMSG you need 2 or higher. Didn't you say you can see the info messages (I/TC)?

[etienne-lms]

You could run OP-TEE on stm32mp1 without pager (CFG_WITH_PAGER=n) for test purpose. Yet, i'll need a patch in TF-A to state that OP-TEE is to be loaded in DDR, not internal secure RAM as stated in the warning trace:

WARNING: The load address in optee header 0xde200000 - 0xde274ae8 is not in reserved area: 0x2ffc0000 - 0x2ffdd000.

The change depends on your TF-A version. Try something like the below:

--- a/fdts/stm32mp15-fw-config.dtsi
+++ b/fdts/stm32mp15-fw-config.dtsi
@@ -45,8 +45,8 @@

 #ifdef AARCH32_SP_OPTEE
                tos_fw {
-                       load-address = <0x0 STM32MP_OPTEE_BASE>;
-                       max-size = <STM32MP_OPTEE_SIZE>;
+                       load-address = <0x0 DDR_SEC_BASE>;
+                       max-size = <DDR_SEC_SIZE>;
                        id = <BL32_IMAGE_ID>;
                };
 #else

When CFG_WITH_PAGER=y, I find it strange that your core pager image is that big it can't be loaded. Embedding early TAs or not should not change its size. Did you enable core debug (CFG_TEE_CORE_DEBUG=y)? Or maybe your have private changes in your optee_os that overloads pager resisdent memory.

[johndoe31415]

Huh, no, level 0 does silence everything. For IMSG you need 2 or higher. Didn't you say you can see the info messages (I/TC)?

Indeed, previously this worked and I saw some log messages. That was, however, a different version of OP-TEE. With 3.16. I have never been able to get logging to work and I've tried all kinds of different combinations. With and without pager, with and without CORE_DEBUG and with different core log levels (only 0 leads to a bootable firmware). I don't know what's going on.

[johndoe31415]

When CFG_WITH_PAGER=y, I find it strange that your core pager image is that big it can't be loaded. Embedding early TAs or not should not change its size. Did you enable core debug (CFG_TEE_CORE_DEBUG=y)?

I have tried both with and without CFG_TEE_CORE_DEBUG, but it does not make a difference.

Or maybe your have private changes in your optee_os that overloads pager resisdent memory.

You mean patched the OS and changed memory addressed? Absolutely not... I've put some debugging messages here and there to try to figure out what is going on, such as:

--- a/core/kernel/early_ta.c
+++ b/core/kernel/early_ta.c
@@ -53,7 +53,7 @@ static TEE_Result early_ta_init(void)
                                 ta->uncompressed_size);
                else
                        msg[0] = '\0';
-               DMSG("Early TA %pUl size %u%s", (void *)&ta->uuid, ta->size,
+               IMSG("Early TA %pUl size %u%s", (void *)&ta->uuid, ta->size,

or

--- a/core/arch/arm/kernel/boot.c
+++ b/core/arch/arm/kernel/boot.c
@@ -1293,6 +1293,7 @@ void __weak boot_init_primary_late(unsigned long fdt)
        configure_console_from_dt();

        IMSG("OP-TEE version: %s", core_v_str);
+       IMSG("Yep, this is definitely the correct build.");

But that's about it. I'm essentially on 4bbca1689f6854f660a80327a0d2959537bea222 with some custom buildscripts/BSP for my hardware.

I've also tried the patch that is supposed to load OP-TEE into DDR, which absolutely should work. But it does not:

INFO:    OPTEE ep=0xde200000
INFO:    OPTEE header info:
INFO:          magic=0x4554504f
INFO:          version=0x2
INFO:          arch=0x0
INFO:          flags=0x0
INFO:          nb_images=0x2
INFO:    parsing pageR image
INFO:       image_info->image_base: 0xde200000
INFO:       image_info->image_max_size: 0x1e00000
INFO:       image->load_addr_hi: 0x0
INFO:       image->load_addr_lo: 0x2ffc0000
INFO:       image->image_id: 0x0
INFO:       image->size: 0x15270
INFO:       sizeof init_load_addr: 4
INFO:       free_end: 0xdfffffff
INFO:       requested_end: 0x2ffd526f
WARNING: The load address in optee header 0x2ffc0000 - 0x2ffd5270 is not in reserved area: 0xde200000 - 0xe0000000.
INFO:    pageR image parse result: -1
ERROR:   OPTEE header parse error.
PANIC at PC : 0x2ffe9c2f

Did I understand you correctly? I changed fdts/stm32mp15-fw-config.dtsi with your patch from https://github.com/OP-TEE/optee_os/issues/5778#issuecomment-1403862189, then recompiled with

CFG_TEE_CORE_DEBUG=n \
CFG_TEE_CORE_LOG_LEVEL=1 \
CFG_TEE_TA_LOG_LEVEL=0 \
CFG_EMBED_DTB_SOURCE_FILE=stm32mp157c-${hwrev}.dts \
CFG_IN_TREE_EARLY_TAS=trusted_keys/f04a0fe7-1f5d-4b9b-abf7-619b85b4ce8c \
CFG_WITH_PAGER=y \
CFG_EARLY_TA_COMPRESS=y \

But I still got the error message from above. Maybe I misunderstood?

[etienne-lms]

But that's about it. I'm essentially on https://github.com/OP-TEE/optee_os/commit/4bbca1689f6854f660a80327a0d2959537bea222 with some custom buildscripts/BSP for my hardware.

That is a fork of OP-TEE OS repository. This may not be the best place to get accurate answers. That said, I would expect with that fork CFG_TEE_CORE_DEBUG=n CFG_TEE_CORE_LOG_LEVEL=2 fits in boot ram. In OP-TEE, for this platform, from no logs (level 0) to log levels 1 or 2 consumes about 6kB (~2 pages) of pager resident memory (there ain't much IMSGs). log level 3 adds 8kB of resident memory. By the way, debug (CFG_TEE_CORE_DEBUG=y) consumes between 20 and 24kB of extra pager resisdent memory. You can check these numbers using ./script/mem_usage.py /path/to/tee.elf on various config builds.

Did I understand you correctly? I changed fdts/stm32mp15-fw-config.dtsi with your patch (...) But I still got the error message from above. Maybe I misunderstood?

My patch applies to TF-A, make sure you rebuild TF-A and the FIP images, if you're using a FIP image.

By the way, you should use CFG_EARLY_TA_COMPRESS=n, see 29dd59cfcaa56193b279b0d6c8dd0d337476d3d1.

[johndoe31415]

Hi everyone, just a small update. Good news first: it is finally working:

[    2.851526] Initialise system trusted keyrings
[    3.431101] trusted_tee_init
[    3.431202] trusted_key_probe
[    3.431228] trusted_key_probe got CTX
[    3.644277] trusted_key_probe got session
[    3.644292] Key type trusted registered
[    3.644306] trusted_key_probe all OK

and:

# time keyctl add trusted kek "load `cat kmk.sealed`" @s
141389696

real    0m0,213s
user    0m0,002s
sys 0m0,210s

Everything else that I'm using the kernel keyring falls into place. It all just works. There are kinks to work out, but this was the major blocker. I know I can get everything working now. That's a great relief.

The bad news is, I am not at all sure what it is that caused it not to work. What I did is redo everything from scratch. Started over, completely. Step by step. Then I noticed that it would not work at all with OP-TEE in DRAM (caused the error message every time). Reverted that as well, went back to LOGLVL 0. I believe that also early TA compression broke things in my case. Took that out as well.

And then, it... just worked. I am absolutely certain that I have tried this exact configuration setting, this exact kernel version before, because my written notes say exactly this. The only explanation I have for now is that I must have made some stupid mistake over and over again. I still don't know what that is, but I have kept all the Git history so hopefully I can retrace my steps and find out what exactly I broke and how I did.

I'll make a writeup of how to do usersapce things with OP-TEE and do a PR aganst the docs, including everything I have learned from you in your support. I cannot overstress how great your support and ideas have been to help me achieve this. If we meet in person, I'm buying beers (or another beverage of your liking), promise. I might be at Embedded World this year, not totally sure yet, however.

Closing this one. Thanks again!

[johndoe31415]

That is a fork of OP-TEE OS repository. This may not be the best place to get accurate answers.

Indeed it is, and there are quite a few commits on top of d0b742d1564834dac903f906168d7357063d5459. Those are by STM to get the STM32MP1 working. But you are right, this is not the clean master repository. After you've mentioned this, I have had a look at the diff. It is quite vast. Do you believe vanilla OP-TEE could even work on the STM32MP1?

EDIT: Also an addition, my working version does still use the STM fork, so that doesn't seem to have been the root cause of my issues.