Recording stops after several days

[masto2305]

I installed the software on a Raspberry PI 4 with Hifiberry soundcard. The software runs for several days without problems. After about 4-5 days there is no more recording, in tksidviewer there is no more update.

In console the following message is displayed.

Exception in thread Thread-94746: Traceback (most recent call last): File "/usr/lib/python3.9/threading.py", line 954, in _bootstrap_inner self.run() File "/usr/lib/python3.9/threading.py", line 1266, in run self.function(*self.args, *self.kwargs) File "/home/Markus/supersid/supersid/sidtimer.py", line 70, in _ontimer self.callback() File "/home/Markus/supersid/supersid/./supersid.py", line 166, in on_timer Pxx, freqs = self.psd(data, self.sampler.NFFT, File "/home/Markus/supersid/supersid/tksidviewer.py", line 208, in get_psd Pxx[channel], freqs = self.axes.psd( File "/usr/lib/python3/dist-packages/matplotlib/init.py", line 1449, in inner return func(ax, map(sanitize_sequence, args), **kwargs) File "/usr/lib/python3/dist-packages/matplotlib/axes/axes.py", line 7020, in psd pxx, freqs = mlab.psd(x=x, NFFT=NFFT, Fs=Fs, detrend=detrend, File "/usr/lib/python3/dist-packages/matplotlib/mlab.py", line 699, in psd Pxx, freqs = csd(x=x, y=None, NFFT=NFFT, Fs=Fs, detrend=detrend, File "/usr/lib/python3/dist-packages/matplotlib/mlab.py", line 756, in csd Pxy, freqs, = _spectral_helper(x=x, y=y, NFFT=NFFT, Fs=Fs, File "/usr/lib/python3/dist-packages/matplotlib/mlab.py", line 499, in _spectral_helper result = np.fft.fft(result, n=pad_to, axis=0)[:numFreqs, :] File "<__array_function__ internals>", line 180, in fft File "/home/Markus/.local/lib/python3.9/site-packages/numpy/fft/_pocketfft.py", line 215, in fft output = _raw_fft(a, n, axis, False, True, inv_norm) File "/home/Markus/.local/lib/python3.9/site-packages/numpy/fft/_pocketfft.py", line 73, in _raw_fft r = pfi.execute(a, is_real, is_forward, fct) numpy.core._exceptions.MemoryError: Unable to allocate 2.88 MiB for an array with shape (46, 4096) and data type complex128

[kunkleb]

I had the same problem when running the tksidviewer on a Raspberry Pi B+ board. At that time, I assumed that the board just didn't have enough horsepower to run tksidviewer efficiently. I've since been running the software on a Raspberry Pi Zero 2 W in text mode and it has been running continuously for months now without a reboot.

Bill Monitor 9408

[sberl]

I think there is a memory leak of some kind in the tkviewer. The system seems to gradually run out of memory. It takes varying amounts of time on different platforms. It also happens on my Dell desktop machine, so not Raspberry PI specific.

[fenrog]

numpy.core._exceptions.MemoryError: Unable to allocate 2.88 MiB for an array with shape (46, 4096) and data type complex128 Yes, there must be a memory leak. numpy reports it but is not necessarily the culprit.

[fenrog]

Today I started supersid with the tk viewer in windows and monitored the amount of memory occupied by Python while supersid was running (tasklist | grep -i Python in a loop). The memory usage is a bit of a zig-zag but overall it is rising. I instrumented the code with gc.garbage but according to the Python garbage collector, there are no uncollectable objects. I'll try different approaches and also compare text and tk viewer.

[fenrog]

The memory leak can be shown with two Python modules. I used the memory-profiler as a black box test and objgraph as white box test.

memory-profiler

• mprof run python supersid.py -c ..\Config\supersid.cfg.KlHo -v tk
• mprof plot -> Very obvious there is a memory leak with the tk viewer, but not so with the text viewer.

objgraph

• added objgraph.show_growth() to the end of on_timer() in supersid.py This counts objects of specific types and reports when the number is increasing. Types like these showed up:
• FontProperties
• _XYPair
• MarkerStyle
• Line2D
• MouseEvent
• MarkerStyle

Based on the types matplotlib + tkinter are the suspects. There are several hits when searching for the combination of memory leak + tkinter + matplotlib

Matplotlib has an example how to integrate with tk. https://matplotlib.org/stable/gallery/user_interfaces/embedding_in_tk_sgskip.html But eventthis appears not to be fully free of memory leaks.

Incrementally transfroming embedding_in_tk_sgskip.py into tksidviewer.py could help to understand which functional changes lead to increased memory leaks.

[fenrog]

So far three different memory leaks identified in dedikated test software based on https://matplotlib.org/stable/gallery/user_interfaces/embedding_in_tk_sgskip.html.

This SW has been modified incrementally to make it more lookalike tksidviewer.py. Mostly it has been used as standalone script, partially as a test type viewer in combination with supersid.py.

leak 1) tkSidViewer.py get_psd() cyclically calling self.axes.clear() + self.axes.psd(). This repeated generation of new plots causes a random leak of Python objects that are created but never deleted. This can be seen in the output of objgraph.show_growth().

leak 2) This goes along with leak 1) and has to be ahndeld separately in the fix for 1). Fix for 1) is to update just the line data instead of redrawing the entire plot. This would still redrwa the y-axis each time, depending on the min and max values of the PSD. Each redraw of the y-axis causes additional Python objects to be cretaed that are never removed again. The solution for this is to keep track of alltime low and all-time high for the y-axis and to set the limits of the y-axis according to these all-time min/max values.

leak 3) The timer that triggers the data provision is relevant. This may or may not be related to the difference between the test software and the real software. To be investigated. When the threading.Timer() of sidtimer.py is used, there is a constant memory leak. When the tk timer tk_root.after() is used, there is no memory leak. https://github.com/fenrog/supersid/commit/d0f09d096d93124694d14816c4aeffc89b68a23d allows to swicth between both timers while all the rest of the code is the same. When monitored with mprof, the leak becomes visible. In this case, no Python objects are involved. The leak mjust be in the native libaries in teh background. mprof run python embedding_in_tk_sgskip.py -c ..\Config\supersid.cfg.mem -t sid - memory leak mprof run python embedding_in_tk_sgskip.py -c ..\Config\supersid.cfg.mem -t tk - no memory leak

[fenrog]

https://github.com/fenrog/supersid/commit/128247c67f20a7cb9e2ba15a71f734a0a91864ba has no more memory leak when displaying random test data. The tests have been executed on Windows 11.

Next steps: Integration with the existing software and test in a Linux environment.

[fenrog]

ready for merge with https://github.com/sberl/supersid/pull/84 - heavily improved although not fully fixed, at least not on Windows