Upon most calls to the functions in pymef3_file.*, there is a call to initialize_meflib(). That function initializes a MEF_globals variable and initializes/stores a number of lookup tables (e.g. AES_initialize_rsbox_tableor SHA256_initialize_k_table). However, this memory is never freed.
This seems little, but when a session is being read, then each file is checked for a password (see _check_password in mef_session.py). A MEF3 dataset can consist of a large number of seperate files, for example imagine 200 channels, with each 5 segments and 3 files per segment (.tdat, .tidx and .tmet), that is 3000 calls to check_mef_password leaking a bit of memory each.
Below shows 50mb leakage on 10.000 calls:
Line # Mem usage Increment Occurrences Line Contents
=============================================================
112 99.1 MiB 99.1 MiB 1 @profile
113 def test(self):
114 149.8 MiB 0.0 MiB 10001 for i in range(10000):
115 149.8 MiB 0.0 MiB 10000 a = '<path>\sub-MEF_ses-ieeg01_ieeg.mefd/EKG.timd\EKG-000000.segd\EKG-000000.tdat'
116 149.8 MiB 50.7 MiB 10000 check_mef_password(a, '')
117 149.8 MiB 0.0 MiB 1 print('')
after applying the changes in this PR (that free the MEF globals at the end):
Line # Mem usage Increment Occurrences Line Contents
=============================================================
112 99.1 MiB 99.1 MiB 1 @profile
113 def test(self):
114 99.1 MiB 0.0 MiB 10001 for i in range(10000):
115 99.1 MiB 0.0 MiB 10000 a = '<path>\sub-MEF_ses-ieeg01_ieeg.mefd/EKG.timd\EKG-000000.segd\EKG-000000.tdat'
116 99.1 MiB 0.0 MiB 10000 check_mef_password(a, '')
117 99.1 MiB 0.0 MiB 1 print('')
I've applied and tested this with the read functions. In this PR I also made sure the submodule points to the latest meflib (which now has the free_meflib function).
@cimbi I'm almost there, I have just one more PR after this one and that should be all!
Hi!
Upon most calls to the functions in
pymef3_file.*, there is a call toinitialize_meflib(). That function initializes aMEF_globalsvariable and initializes/stores a number of lookup tables (e.g.AES_initialize_rsbox_tableorSHA256_initialize_k_table). However, this memory is never freed.This seems little, but when a session is being read, then each file is checked for a password (see
_check_passwordinmef_session.py). A MEF3 dataset can consist of a large number of seperate files, for example imagine 200 channels, with each 5 segments and 3 files per segment (.tdat, .tidx and .tmet), that is 3000 calls to check_mef_password leaking a bit of memory each.Below shows 50mb leakage on 10.000 calls:
after applying the changes in this PR (that free the MEF globals at the end):
I've applied and tested this with the read functions. In this PR I also made sure the submodule points to the latest meflib (which now has the free_meflib function).
@cimbi I'm almost there, I have just one more PR after this one and that should be all!
Best, Max