search

NeurodataWithoutBorders / matnwb

A Matlab interface for reading and writing NWB files
BSD 2-Clause "Simplified" License
50 stars 32 forks source link

lazy loading of datasets #27

Closed nclack closed 6 years ago

lawrence-mbf commented 6 years ago

types.untyped.Datastub has been implemented which takes the place of any non-string, non-compound, array dataset.

oruebel commented 6 years ago

@nclack just to clarify if I understand this correctly this means if a dataset is a string or a compound dataset then it will be loaded when the file is read and otherwise it will be loaded lazily when the user requests the data field, correct?

lawrence-mbf commented 6 years ago

Right now, if the dataset is a H5T_String or Compound it will load the dataset data (for some of the metadata stuff in the NWB file). The specific requirements for what data is lazily loaded is subject to change in future commits.

oruebel commented 6 years ago

Thanks for the clarification. IT's great to see the lazy load for MatNWB. In terms of in-memory size, I think loading strings and compound datasets should be reasonable. The compound types might get a bit larger but right now should be Ok. For large file, loading datasets (even if they are small) may still pose a problem with regard to initial load time but will see how this goes.