DiamondLightSource / httomo

High-throughput tomography pipeline
https://diamondlightsource.github.io/httomo/
BSD 3-Clause "New" or "Revised" License
4 stars 3 forks source link

Fix memory leak caused by reader/writer objects not being garbage collected during pipeline execution #265

Closed yousefmoazzam closed 3 months ago

yousefmoazzam commented 3 months ago

Fixes #264

yousefmoazzam commented 3 months ago

The reason why reader/writer objects weren't being garbage collected during pipeline execution

Following on from seeing this helpful stackoverflow post, further inspection of the documentation for weakref.finalize() brought the following warning to attention:

weakref-finalize-warning

This explains the issue! The functions being passed to weakref.finalize() are indeed "bound methods" on the objects being passed in:

Therefore, due to passing references to bound methods of the objects to be "finalized", the objects in question (reader and writer objects) were destined to never be garbage collected until the program terminated.

Confirmation of that particular memory leak going away with this fix

Here is the original memory leak from the original issue for easy comparison:

memory-leak-highlighted

After swapping out the regular references to the bound finalize methods on the reader/writer for weak references using weakref.WeakMethod, the RAM usage appears to be as expected for the 20GB data when using the pipeline described in #264:

weakmethod-fix