NeurodataWithoutBorders / matnwb

A Matlab interface for reading and writing NWB files
BSD 2-Clause "Simplified" License
49 stars 32 forks source link

Cache File and Dataset IDs for DataStubs #412

Open lawrence-mbf opened 2 years ago

lawrence-mbf commented 2 years ago
lawrence-mbf commented 1 year ago

Currently blocked by https://github.com/NeurodataWithoutBorders/matnwb/pull/448#issuecomment-1256521418

ehennestad commented 2 weeks ago

@lawrence-mbf

Is the idea here to keep h5 IDs in memory to avoid opening and closing files/datasets more times than needed?

lawrence-mbf commented 2 weeks ago

@ehennestad I believe this was because network drives were incredibly slow to process when opened/closed. Data processing is much faster if the file ids were kept open, though you would have to be diligent to maintain the file ids so you don't get leaks or use after fcloses.