Closed yousefmoazzam closed 3 months ago
Following on from seeing this helpful stackoverflow post, further inspection of the documentation for weakref.finalize()
brought the following warning to attention:
This explains the issue! The functions being passed to weakref.finalize()
are indeed "bound methods" on the objects being passed in:
reader
is an instance of DataSetStoreReader
, then the bound method reader.finalize
is passed in to weakref.finalize()
, which contains a reference to the reader
object, incurring the consequences of the above warningwriter
is an instance of DataSetStoreWriter
, then the bound method writer.finalize
is passed in to weakref.finalize()
, which contains a reference to the writer
object, incurring the consequences of the above warningTherefore, due to passing references to bound methods of the objects to be "finalized", the objects in question (reader and writer objects) were destined to never be garbage collected until the program terminated.
Here is the original memory leak from the original issue for easy comparison:
After swapping out the regular references to the bound finalize
methods on the reader/writer for weak references using weakref.WeakMethod
, the RAM usage appears to be as expected for the 20GB data when using the pipeline described in #264:
Fixes #264