BUG: Reference cycle problem with multiple windows open

[larsoner]

This code:

import gc
import time
from pyvistaqt import BackgroundPlotter

plotters = list()
time.sleep(1)
for _ in range(20):
    plotter = BackgroundPlotter()
    plotter.show()
    plotters.append(plotter)
    #for plotter in plotters:
    plotter.close()
    plotter.deep_clean()
del plotter, plotters
gc.collect()
time.sleep(2)

Run with:

mprof run mem.py; mprof plot

Seems fine:

But when you uncomment the for plotter in plotters line and unindent it so that all plots are created, then all plots closed (rather than one created, one closed, repeating) we get:

In other words, there is ~150 MB in residual memory that cannot be cleared. I've spent some time trying to figure it out, no luck so far.

I know this seems like a minor problem, but when you have real plots and data it matters. Over in MNE we have an example that opens about 20 windows at a time, and it causes a permanent ~1GB memory jump in a sphinx-gallery build -- it's the 1 GB step function around 1600 seconds here (I suspect most of our step jumps in doc builds are due to this):

Here is a MNE-specfic example with real data and 10 plots, individual create-and-close:

Create-all, close-all:

Even mayavi seems to have the problem, though:

This makes me wonder if this is under the hood a problem with VTK's interactor class or something. Still, it would be nice to figure out what attributes could be deleted or tweaked to make it so that these can actually get garbage collected.

[GuillaumeFavelier]

Could it be related to pyvista _ALL_PLOTTERS variable? What happens if you try pyvista.close_all() at the end of the script?

[larsoner]

Same result (testing the PyVista minimal example, not the MNE ones, though I'm confident they will behave the same way):

[GuillaumeFavelier]

I tried resolving some reference cycles (decreasing the reference count: self.interactor, self.frame, self.app_window, ...), I tried force-freeing Qt objects (deleteLater() and sip.delete()). I also looked at QVTKRenderWindowInteractor, I tried the same approach with ref count (_Iren, _RenderWindow), explicitly deleting _hidden widget and calling (again) Finalize() to free vtk items.

What I did not try is to decrease the ref count of callables. From stackoverflow:

Note that when it comes to signal/slot connections, PyQt treats wrapped C++ slots and Python instance methods differently. The reference counts of these types of callable are not increased when they are connected to signals, whereas lambdas, defined functions, partial objects and static methods are. This means that if all other references to the latter types of callable are deleted, any remaining signal connections will keep them alive. Disconnecting the signals will allow such connected callables to be garbage-collected, if necessary.

I did not notice any significant improvement. Do you have any ideas @pyvista/developers ?

[GuillaumeFavelier]

I think https://github.com/pyvista/pyvistaqt/pull/66 could help with this especially if a test is designed for it. I'll do my best to go back on it when I can to make it move forward :+1:

[larsoner]

Pretty sure this is VTK+Qt's problem, not PyVista's. :(

Closing for https://gitlab.kitware.com/vtk/vtk/-/issues/18588 but I'll try to remember to report back here if it's ever fixed