labscript-suite / labscript-utils

Shared modules used by the 𝘭𝘒𝘣𝘴𝘀𝘳π˜ͺ𝘱𝘡 𝘴𝘢π˜ͺ𝘡𝘦. Includes a graphical exception handler, debug tools, configuration management, cross platform filepath conversions, unit conversions and custom GUI widgets.
http://labscriptsuite.org
Other
2 stars 47 forks source link

Fix memory leak of h5py when accessing dataset with composite dtypes. #92

Closed johannesschabbauer closed 1 year ago

johannesschabbauer commented 1 year ago

I had a memory leak in a device Worker, caused by the properties.get function (connection_table_properties) in this line: https://github.com/labscript-suite/labscript-utils/blob/2a8bc1d99f7b0dbb6860a773901be5bdef9079bf/labscript_utils/properties.py#L127

Every time the dtype attribute of the "connection table" dataset is called, some kB of memory are used that are never freed after that. In the Worker process the memory usage was increasing for every shot (I called the function in get_final values 6 times for different child devices), which is a severe problem if 10,000s of shots are run without restarts.

I resolved the problem for me by not using properties.get. There are already reports of some related issues.

The memory leak seems to be caused by the mixed usage of fixed length and variable length columns, and can be reproduced with the following code:

import h5py
import numpy as np

#create test file
f = h5py.File("test.h5","x") 
# create dataset with fixed and vlen dtype
dtype = [("name","S256"),("string",h5py.special_dtype(vlen=str))]
data = np.array(10*[("test","test")], dtype=dtype)
f.create_dataset("test", data=data)

for _ in range(10000):
    f["test"]["name"].dtype
    # f["test"]["name"][0] also causes a memory leak

f.close()

With the changes in the PR I don't see any memory leak. Accessing the data first by column and then by row also causes a memory leak. Thus it could be possible that a similar problem occurs in some other functions, too.

dihm commented 1 year ago

Good find Johannes! This is rather annoying little feature. Since the change is small and entirely equivalent in function, I'm going to go ahead and merge.