Shared modules used by the ππ’π£π΄π€π³πͺπ±π΅ π΄πΆπͺπ΅π¦. Includes a graphical exception handler, debug tools, configuration management, cross platform filepath conversions, unit conversions and custom GUI widgets.
Every time the dtype attribute of the "connection table" dataset is called, some kB of memory are used that are never freed after that. In the Worker process the memory usage was increasing for every shot (I called the function in get_final values 6 times for different child devices), which is a severe problem if 10,000s of shots are run without restarts.
The memory leak seems to be caused by the mixed usage of fixed length and variable length columns, and can be reproduced with the following code:
import h5py
import numpy as np
#create test file
f = h5py.File("test.h5","x")
# create dataset with fixed and vlen dtype
dtype = [("name","S256"),("string",h5py.special_dtype(vlen=str))]
data = np.array(10*[("test","test")], dtype=dtype)
f.create_dataset("test", data=data)
for _ in range(10000):
f["test"]["name"].dtype
# f["test"]["name"][0] also causes a memory leak
f.close()
With the changes in the PR I don't see any memory leak. Accessing the data first by column and then by row also causes a memory leak. Thus it could be possible that a similar problem occurs in some other functions, too.
Good find Johannes! This is rather annoying little feature. Since the change is small and entirely equivalent in function, I'm going to go ahead and merge.
I had a memory leak in a device Worker, caused by the
properties.get
function (connection_table_properties) in this line: https://github.com/labscript-suite/labscript-utils/blob/2a8bc1d99f7b0dbb6860a773901be5bdef9079bf/labscript_utils/properties.py#L127Every time the dtype attribute of the "connection table" dataset is called, some kB of memory are used that are never freed after that. In the Worker process the memory usage was increasing for every shot (I called the function in get_final values 6 times for different child devices), which is a severe problem if 10,000s of shots are run without restarts.
I resolved the problem for me by not using
properties.get
. There are already reports of some related issues.The memory leak seems to be caused by the mixed usage of fixed length and variable length columns, and can be reproduced with the following code:
With the changes in the PR I don't see any memory leak. Accessing the data first by column and then by row also causes a memory leak. Thus it could be possible that a similar problem occurs in some other functions, too.