search

mofanv / darknetz

runs several layers of a deep learning model in TrustZone
MIT License
86 stars 29 forks source link

is it possible to check the memory usage in TA while the Training running? #26

Closed Song-175 closed 3 years ago

Song-175 commented 3 years ago

Hi, mofan

I'm using the darknetz with classifier.c which is quite a basic thing, but I would like the check the total memory usage of TA and memory usage by each layer while training.

I tried several ways to figure out that as below

  1. checked the VMSIZE / VMRSS / MAX (By each epoch)
  2. checked the total size of region 0 to region 8 when the error occurs in TA (sum of all mapping sizes)
  3. checked the TA_RAM, TEE_RAM_RX, TEE_RAM_RW

but still, I'm wondering which one is presenting the exact total memory size of TA. (each epoch) And I have no idea how can I check the TA memory usage of each layer while training.

Thanks!

mofanv commented 3 years ago

Hi, sorry for the late response. 1- You actually cannot access TA memory usage using VMSIZE / VMRSS / MAX. These values are measured in Rich OS, i.e. the outside REE, not TEE. 2- total size of region 0 to region 8 could work but it must end the application. 3- TA_RAM, TEE_RAM_RX, TEE_RAM_RW are the size when you define the mapping size allocated for different parts (runtime or application), which will not change during running.

One good realtime measure is to use mdbg_check(1) provided by OPTEE. You will need compile it with corresponding flags. See it here: https://github.com/OP-TEE/optee_os/blob/master/mk/config.mk#L72 . Calling mdbg_check(1) will print a list of all the currently allocated buffers, e.g. memory used during training of each layer, in the Secure World Terminal. This is also how we measure it in our next work https://github.com/mofanv/PPFL/blob/main/client_side_trustzone/ta/darknetp_ta.c#L828

Song-175 commented 3 years ago

Thanks for the quality response.

I will keep study based on what u told me. Thanks a lot again.

Cheers