Benchmark: can't create mobj for timestamp buffer

[Arslan8]

Hi,

I am trying to benchmark op-tee on hikey 620. Firstly, I faced some build issues. Just using the command-line option CFG_TEE_BENCHMARK=y results in buildroot not being able to find the optee-benchmark folder. Upon further investigation, it looks like the source site (BR2_PACKAGE_OPTEE_BENCHMARK_EXT_SITE) is not defined properly in the makefile. Setting the source site on the command line resolves the build issue, however the benchmarking app results in a missing TA error. To overcome this I updated the makefile instead.

After fixing the missing ta and build issue. I see the following error while benchmarking my target app:

E/TC:? 0 alloc_benchmark_buffer:72 Benchmark: can't create mobj for timestamp buffer

I have tried increasing the memory by increasing both PGT_CACHE_SIZE and the configurable heap size option. Furthermore, instead of the intended app, I have tried using a helloworld TA with minimum stack and heap requirements (1K in total). However, I still see the same issue, which hinted that it might not be an overall memory constraint issue, so I skimmed through the code to see the reason.

Going through the code, the error is triggered when the PTA makes an RPC call to the op-tee driver which fails. The exact failing line is in handle_rpc_func_cmd_shm_alloc:

                if (!pages || !page_num) {
                        arg->ret = TEEC_ERROR_OUT_OF_MEMORY;
                        printk(KERN_EMERG "Bad alloc");
                        goto bad;
                }

Here both the pages and page_num are zero or null. The driver call stack in the driver looks like this: handle_rpc_func_cmd_shm_alloc-> tee_shm_alloc->pool_op_alloc->optee_shm_register. Going through the functions I don't see any code updating the pages or page_num field so it was bound to fail. It is possible that I might be missing something here. However, adding a print shows that this kind of allocation (OPTEE_MSG_RPC_SHM_TYPE_GLOBAL) only happens once when I use the benchmark app so quite possible that this code is missing. Based on my findings I have a couple of questions:

1. Based on the build issues, I am wondering Is the benchmarking suite supported on hikey?
2. Who is responsible for fixing up the pages and page_num field?
3. Possible fix?

Also, I wasn't sure if this is the right place for this issue so please let me know if this is not the case. I can move it to the relevant project...

[jenswi-linaro]

I sounds like some patches are missing from the kernel. Are you building using the instructions at https://optee.readthedocs.io/en/latest/building/gits/build.html#get-and-build-the-solution ?

[Arslan8]

@jbech-linaro thanks for the quick response..Yes, i am using the instructions from the link you provided... I did not specify a specific branch while checking out in repo init command though...

[jenswi-linaro]

What's the output when the kernel boots, something like:

optee: probing for conduit method.
optee: revision 3.13 (c4b79363)
optee: dynamic shared memory is enabled
optee: initialized driver

Note the line with "dynamic shared memory is enabled".

[Arslan8]

@jenswi-linaro a quick grep on kernel output gives me this:

[    2.803412] optee: probing for conduit method.
[    2.812567] optee: revision 3.13 (a07b8a39)ynamic shared memory is enabled
[    2.825072] optee: dynamic shared memory is enabled
D/TC:? 0 tee_ta_init_pseudo_ta_session:299 Lookup pseudo TA 7011a688-ddde-4053-a5a9-7b3c4ddf13b8
D/TC:? 0 tee_ta_init_pseudo_ta_session:312 Open device.pta
D/TC:? 0 tee_ta_init_pseudo_ta_session:329 device.pta : 7011a688-ddde-4053-a5a9-7b3c4ddf13b8
D/TC:? 0 tee_ta_close_session:512 csess 0x3f084a20 id 1
D/TC:? 0 tee_ta_close_session:531 Destroy session
[    2.884104] optee: initialized driver
Set permissions on /dev/tee*: OK
Create/set permissions on /data/tee: OK
Starting tee-supplicant: OK
D/TC:? 0 tee_ta_init_session_with_context:607 Re-open TA 7011a688-ddde-4053-a5a9-7b3c4ddf13b8
Starting network: D/TC:? 0 tee_ta_close_session:512 csess 0x3f084870 id 1
D/TC:? 0 tee_ta_close_session:531 Destroy session

Grep on dmesg:

[    2.803412] optee: probing for conduit method.
[    2.812567] optee: revision 3.13 (a07b8a39)
[    2.825072] optee: dynamic shared memory is enabled
[    2.884104] optee: initialized driver

It has the line you are looking for..

[jenswi-linaro]

Looks OK. It seems it could be a configuration error or even bitrot on the secure side with regards to CFG_TEE_BENCHMARK=y. I guess this because it seems that secure world is trying to allocate buffer of size 0.

[Sadiq1811]

We have an x86 based OPTEE and want to port this framework to x86 for our use-cases. So, for quick hands on, tried it on OPTEE 3.13 QEMU environment and ended up in the same issue.

Looks like even after porting to x86, we might end up in the same issue. @jenswi-linaro your thoughts on this?

[jenswi-linaro]

As far as I know nothing has changed here during the last year so CFG_TEE_BENCHMARK=y may still need some work to get running.

[Sadiq1811]

Does it make sense to tag this issue as an enhancement so that the issue doesn't get auto-closed ?

[jforissier]

@Sadiq1811 tag added. If the "benchmark" framework is useful to you, feel free to send fixes and/or improvements. I can review and test such patches.

[th0ma5b]

FYI, I confirm I get the same issue in this environment: repo init -u https://github.com/OP-TEE/manifest.git -m qemu_v8.xml -b 3.16.0

I fixed the build by adding this line in build/common.mk: BENCHMARK_APP_PATH ?= $(ROOT)/optee_benchmark

when running benchmark, I get the following error (similar error as previously reported):

[NW] # benchmark optee_example_hello_world [Benchmark] INFO: 1. Opening Benchmark Static TA... [Benchmark] INFO: 2. Allocating per-core buffers, cores detected = 2 [Benchmark] ERROR: TEEC_InvokeCommand: 0xffff000c

[SW] D/TC:? 0 tee_ta_init_pseudo_ta_session:296 Lookup pseudo TA 0b9a63b0-b4c6-4c85-a284-a228ef547b4e D/TC:? 0 tee_ta_init_pseudo_ta_session:309 Open benchmark.ta D/TC:? 0 tee_ta_init_pseudo_ta_session:326 benchmark.ta : 0b9a63b0-b4c6-4c85-a284-a228ef547b4e D/TC:? 0 tee_ta_init_session_with_context:607 Re-open TA 0b9a63b0-b4c6-4c85-a284-a228ef547b4e D/TC:? 0 get_benchmark_memref:120 Benchmark: Sending back timestamp buffer paddr = 0x0 D/TC:? 0 tee_ta_close_session:512 csess 0xe17eb90 id 3 D/TC:? 0 tee_ta_close_session:531 Destroy session E/TC:? 0 alloc_benchmark_buffer:72 Benchmark: can't create mobj for timestamp buffer D/TC:? 0 tee_ta_invoke_command:823 Error: ffff000c of 4 D/TC:? 0 tee_ta_init_session_with_context:607 Re-open TA 0b9a63b0-b4c6-4c85-a284-a228ef547b4e D/TC:? 0 get_benchmark_memref:120 Benchmark: Sending back timestamp buffer paddr = 0x0 D/TC:? 0 tee_ta_close_session:512 csess 0xe17ea70 id 3 D/TC:? 0 tee_ta_close_session:531 Destroy session

I believe the benchmarked TA is not yet involved at this stage because even non-existant TA report same log/error..

• Do you know a past version with a known working benchmark feature?

[jforissier]

The benchmark code in the Linux driver was dropped with OP-TEE release 3.16.0 because it was too difficult to rebase onto newer kernels. see https://github.com/linaro-swg/linux/issues/98. The latest tag with benchmark support in linaro-swg/linux is https://github.com/linaro-swg/linux/commits/optee-3.15.0 (see commit https://github.com/linaro-swg/linux/commit/d9b0331b46540fa67c0f16e391940f12fde1288b).

[th0ma5b]

Hi Jerome, thanks for clarifying.

[hearyouagain]

@th0ma5b do u have met the problem “[Benchmark: can't create mobj for timestamp buffer” when u use optee_benchmark on optee_os 3.15.0？