search

stonne-simulator / stonne

STONNE: A Simulation Tool for Neural Networks Engines
MIT License
115 stars 26 forks source link

Memory leaks during big simulation runs - Destructors not being called #24

Closed isai-roman closed 1 year ago

isai-roman commented 1 year ago

Hello Adrian,

Hey I've been having serious memory issues with big simulation runs (i.e. for instance take MobileNetV2, TPU 256x256 and corresponding architecture attached). The process was killed by the OOM killer in our servers in the lab because of memory starvation (Please check the picture; the memory needs are in the order of TB). So I start digging in and I partially solved the biggest memory issues changing the abstract classes of the hardware components. Since they do not have any virtual destructor, the destructor of the derived classes were not called (reference about this topic, https://www.quantstart.com/articles/C-Virtual-Destructors-How-to-Avoid-Memory-Leaks/). Could you please help us fix this in your master branch? If you need to check my commit, please let me know.

Regards,

Isai.

photo_2023-06-10_13-09-39 tpu_256_256_pes.cfg.zip

Adrian-2105 commented 1 year ago

Hi Isai,

First of all, thanks a lot for taking your time going deep into the code and debugging it. We are sorry for this inconveniences and we will try to fix them as soon as possible. We know that STONNE suffers from some bugs like this one and we plan to fix and update them in upcoming releases (but they will take some time yet).

Your contribution will be really well appreciated. Please, do a pull request to the STONNE master branch and we can discuss more in detail about the changes you included.

Thanks for your work! I will be really helpful to the rest of STONNE users :-)

isai-roman commented 1 year ago

Hi,

No problem, I hope it helps. The pull request is on the way.

Regards,

Isai.