Cache File and Dataset IDs for DataStubs

NeurodataWithoutBorders / matnwb

A Matlab interface for reading and writing NWB files

BSD 2-Clause "Simplified" License

49 stars 32 forks source link

Cache File and Dataset IDs for DataStubs #412

Open lawrence-mbf opened 2 years ago

lawrence-mbf commented 2 years ago

[ ] cache file and dataset ids on load and destroy them if possible.
[ ] use a different solution for bound pipes which may cause file corruption if the dataset is not properly closed.

lawrence-mbf commented 1 year ago

Currently blocked by https://github.com/NeurodataWithoutBorders/matnwb/pull/448#issuecomment-1256521418

ehennestad commented 2 weeks ago

@lawrence-mbf

Is the idea here to keep h5 IDs in memory to avoid opening and closing files/datasets more times than needed?

lawrence-mbf commented 2 weeks ago

@ehennestad I believe this was because network drives were incredibly slow to process when opened/closed. Data processing is much faster if the file ids were kept open, though you would have to be diligent to maintain the file ids so you don't get leaks or use after fcloses.