OPTEE PKCS11 Object Size Limit

[embetrix]

I noticed that I cannot import object in PKCS11 TA that are over 4K:

4KB still okay:

# export PKCS11_MODULE_PATH=/usr/lib/libckteec.so.0
# export PIN="12345"
# export TOKEN_NAME="token0"
# dd if=/dev/urandom of=/tmp/data.bin bs=1024 count=4 
4+0 records in
4+0 records out
4096 bytes (4.1 kB) copied, 0.00013044 s, 31.4 MB/s
# pkcs11-tool  --pin $PIN --module $PKCS11_MODULE_PATH  --login  --write-object /tmp/data.bin --type data   --id 1137 --label DATA 
Using slot 0 with a present tokenI/TA: PKCS11 session 1: login
 (0x0)
Created Data Object:
Data object 1
  label:          'DATA'
  application:    ''
  app_id:         <empty>
  flags:           modifiable

5KB not okay:

# dd if=/dev/urandom of=/tmp/data.bin bs=1024 count=5 
5+0 records in
5+0 records out
5120 bytes (5.1 kB) copied, 0.000145072 s, 35.3 MB/s
# pkcs11-tool  --pin $PIN --module $PKCS11_MODULE_PATH  --login  --write-object /tmp/data.bin --type data   --id 1137 --label DATA 
Using slot 0 with a present tokenI/TA: PKCS11 session 1: login
 (0x0)
error: PKCS11 function C_CreateObject failed: rv = CKR_DEVICE_MEMORY (0x31)
Aborting.

where can I increase this limit ?

[etienne-lms]

The pkcs11 TA heap (where the TA allocates memory) default size is 32kBytes. It can be tuned with CFG_PKCS11_TA_HEAP_SIZE=<size-in-bytes>. Indeed 32kB is not very big. Maybe we should default to something bigger. In any case, you can set this size according to your needs. The max size depends on the overall TA_RAM size (CFG_TZDRAM_SIZE), used by core for all TAs (maybe there are other TAs than the pkcs11 TA in your system).

[embetrix]

@etienne-lms I increased the CFG_PKCS11_TA_HEAP_SIZE to 128KB but it doesn't solve the problem, there seems to be a hard limit at 4096Bytes:

# dd if=/dev/urandom of=/tmp/data.bin bs=1 count=4096
4096+0 records in
4096+0 records out
4096 bytes (4.1 kB) copied, 0.0235162 s, 174 kB/s
# pkcs11-tool  --pin $PIN --module $PKCS11_MODULE_PATH  --login  --write-object /tmp/data.bin --type data   --id 1137 --label DATA 
Using slot 0 with a present token (0x0)
I/TA: PKCS11 session 1: login
Created Data Object:
Data object 1
  label:          'DATA'
  application:    ''
  app_id:         <empty>
  flags:           modifiable
# dd if=/dev/urandom of=/tmp/data.bin bs=1 count=4097
4097+0 records in
4097+0 records out
4097 bytes (4.1 kB) copied, 0.0235554 s, 174 kB/s
# pkcs11-tool  --pin $PIN --module $PKCS11_MODULE_PATH  --login  --write-object /tmp/data.bin --type data   --id 1137 --label DATA 
Using slot 0 with a present token (0x0)
I/TA: PKCS11 session 1: login
error: PKCS11 function C_CreateObject failed: rv = CKR_DEVICE_MEMORY (0x31)

my CFG_TZDRAM_SIZE = 0x01e00000

[etienne-lms]

That's strange. I don't see where such limit is set. I've tried, unsing QEMU with OP-TEE's qemu_armv8a distribution built with: make all CFG_PKCS11_TA=y CFG_PKCS11_TA_HEAP_SIZE=128000

From the non-secure console:

(...)
Welcome to Buildroot, type root or test to login
buildroot login: root
# export PS1="shell> "
shell> 
shell> export PKCS11_MODULE_PATH=/usr/lib/libckteec.so.0
shell> export PIN="12345"
shell> export TOKEN_NAME="token0"
shell> 
shell> pkcs11-tool --module $PKCS11_MODULE_PATH --label test --slot 0 --init-token --so-pin $PIN
Token successfully initialized
shell> pkcs11-tool --module $PKCS11_MODULE_PATH --label test --so-pin $PIN --init-pin --pin $PIN 
Using slot 0 with a present token (0x0)
User PIN successfully initialized
shell> 
shell> dd if=/dev/urandom of=/tmp/data.bin bs=1 count=4096
4096+0 records in
4096+0 records out
shell> pkcs11-tool --module $PKCS11_MODULE_PATH --login  --pin $PIN --write-object /tmp/data.bin --type data --label sz4096
Using slot 0 with a present token (0x0)
Created Data Object:
Data object 1
  label:          'sz4096'
  application:    ''
  app_id:         <empty>
  flags:           modifiable
shell> 
shell> dd if=/dev/urandom of=/tmp/data.bin bs=1 count=15000
15000+0 records in
15000+0 records out
shell> pkcs11-tool --module $PKCS11_MODULE_PATH --login  --pin $PIN --write-object /tmp/data.bin --type data --id 15000 --label sz15000
Using slot 0 with a present token (0x0)
Created Data Object:
Data object 1
  label:          'sz15000'
  application:    ''
  app_id:         <empty>
  flags:           modifiable
shell> 
shell> pkcs11-tool --module $PKCS11_MODULE_PATH --login  --pin $PIN  --list-objects
Using slot 0 with a present token (0x0)
Data object 1
  label:          'sz15000'
  application:    ''
  app_id:         <empty>
  flags:           modifiable
Data object 2
  label:          'sz4096'
  application:    ''
  app_id:         <empty>
  flags:           modifiable
shell> 

[embetrix]

@etienne-lms I used optee-os 3.19 Version on imx8mp , I also increased CFG_CORE_HEAP_SIZE

CFG_CORE_HEAP_SIZE=262144 
CFG_PKCS11_TA_HEAP_SIZE=131072 

still hitting the same issue :-(

I can see in the code that

in https://github.com/OP-TEE/optee_os/blob/master/ta/pkcs11/src/user_ta_header_defines.h#L19

#define TA_DATA_SIZE CFG_PKCS11_TA_HEAP_SIZE

https://github.com/OP-TEE/optee_os/blob/master/ta/pkcs11/src/user_ta_header_defines.h#L17C1

#define TA_STACK_SIZE (4 * 1024)

Maybe this is the limitation ?

[etienne-lms]

I don't think TA_STACK_SIZE=4096 is the limitation. The pkcs11 TA does not use its stack to store object's data payloads or attributes. All are stored in the heap which size is given by TA_DATA_SIZE that is set by CFG_PKCS11_TA_HEAP_SIZE.

[etienne-lms]

Add some trace messages (e.g. EMSG(...)) in error cases of entry_create_object() and create_object(). I hope you'll find where the TA fails.

[embetrix]

@etienne-lms I tested it also on an stm32mp1 it give me the same errors !

I will try to build some debug message to pin-point where this come from.

[etienne-lms]

I've- just tested on my stm32mp157c-dk2 boards. The commands above (https://github.com/OP-TEE/optee_os/issues/6482#issuecomment-1822887214) work fine. I'm using the latest top master branches of optee_* Git repositories.

edited: I forgot to say I built my images with: make PLATFORM=stm32mp1-157C_DK2 CFG_TEE_CORE_LOG_LEVEL=2 CFG_PKCS11_TA=y all CFG_PKCS11_TA_HEAP_SIZE=128000

With that config, I manage to create 14 objects of 15000 bytes each. Maybe the objects do not have the size I expect...

[embetrix]

thanks for the information.

I used the versions:

3.19 for the imx8mp 3.16 for the stm32mp1

for both I set _CFG_PKCS11_TA_HEAPSIZE=131072

I don't have the possibility to update to latest for those device, so I need to find out which changes I can cherry-pick to solve this issue

[etienne-lms]

You should be able to pick the source tree of ta/pkcs11/ from optee_os master branch and build that TA sources with your several build env based on older versions of OP-TEE. I would expect that it works ok. Oh, likely that you'll need to revert changes form commit c7f1b4f7881f.

[embetrix]

@etienne-lms I had to increase also the CFG_TEE_RAM_VA_SIZE

with following:

CFG_CORE_HEAP_SIZE=2097152 
CFG_TEE_RAM_VA_SIZE=4194304 
CFG_PKCS11_TA_HEAP_SIZE=524288 

I can import without errors

root@imx8mp:~# dd if=/dev/urandom of=/tmp/data.bin bs=1024 count=4                                                                                                                                           
4+0 records in
4+0 records out
4096 bytes (4.1 kB) copied, 0.00011981 s, 34.2 MB/s
root@imx8mp:~# 
root@imx8mp:~# 
root@imx8mp:~# 
root@imx8mp:~# pkcs11-tool  --pin $PIN --module $PKCS11_MODULE_PATH  --login  --write-object /tmp/data.bin --type data --label DATA                                                                                                                                              
Using slot 0 with a present tokenI/TA: PKCS11 session 1: login
 (0x0)
Created Data Object:
Data object 1
  label:          'DATA'
  application:    ''
  app_id:         <empty>
  flags:           modifiable
root@imx8mp:~# pkcs11-tool  --pin $PIN --module $PKCS11_MODULE_PATH  --login  --read-object  --type data   --label DATA -o data.bin.out                                                                                                                                            
Using slot 0 with a present token (0x0)
I/TA: PKCS11 session 1: login
root@imx8mp:~# diff data.bin.out /tmp/data.bin

however data over 5000 Bytes seems to be truncated :

root@imx8mp:~# dd if=/dev/urandom of=/tmp/data.bin bs=1024 count=6
6+0 records in
6+0 records out
6144 bytes (6.1 kB) copied, 0.000165208 s, 37.2 MB/s
root@imx8mp:~# pkcs11-tool  --pin $PIN --module $PKCS11_MODULE_PATH  --login  --write-object /tmp/data.bin --type data --label DATA
Using slot 0 with a present token (0x0)
I/TA: PKCS11 session 1: login
Created Data Object:
Data object 1
  label:          'DATA'
  application:    ''
  app_id:         <empty>
  flags:           modifiable
root@imx8mp:~# pkcs11-tool  --pin $PIN --module $PKCS11_MODULE_PATH  --login  --read-object  --type data   --label DATA -o data.bin.out
Using slot 0 with a present tokenI/TA: PKCS11 session 1: login
 (0x0)
root@imx8mp:~# diff data.bin.out /tmp/data.bin
Binary files data.bin.out and /tmp/data.bin differ
root@imx8mp:~# ls -l data.bin.out
-rw-r--r-- 1 root root 5000 Nov 27 11:25 data.bin.out

[embetrix]

I tested the same setup with SoftHSM, I get also the truncation to 5000 Bytes, there seems to be a limit in the standard.

[embetrix]

https://raymii.org/s/articles/Storing_arbitraty_data_in_the_Nitrokey_HSM.html

[embetrix]

the 5000 Bytes limit is actually set in pkcs11-tool:

https://github.com/OpenSC/OpenSC/blob/master/src/tools/pkcs11-tool.c#L3925

[etienne-lms]

https://raymii.org/s/articles/Storing_arbitraty_data_in_the_Nitrokey_HSM.html

the 5000 Bytes limit is actually set in pkcs11-tool: https://github.com/OpenSC/OpenSC/blob/master/src/tools/pkcs11-tool.c#L3925

Thanks for the info and the pointers.