nilearn / nilearn

Machine learning for NeuroImaging in Python
http://nilearn.github.io
Other
1.17k stars 595 forks source link

Memory profiling of the masker objects #3399

Open htwangtw opened 1 year ago

htwangtw commented 1 year ago

This is related to #3398.

We have started to profile the masker object to find out the where's most of the memory consumption happening, aiming to see if there's possibility to trim the code / process.

Through some rough and bare bone methods, we found resampling and apply_mask seems to be taking up most of the consumption. However, with the existing profiling tools we tried (memory_profiler (line by line / time tracking), guppy3 (memory by type of objects)) won't give us detailed traced back of the masker methods.

A few directions:

WIP repo:

https://github.com/htwangtw/masker-memory-profile

Profile of memory usage by time of NiftiMasker niftimasker_memory_profile

htwangtw commented 1 year ago

This profiler might be what we want, as it look into the modules from a glance: https://github.com/bloomberg/memray