Loaders: read_hdf5: use hdf5prop (HDF memmapfile)

[farhi]

The hdf5prop at:

• https://fr.mathworks.com/matlabcentral/fileexchange/31703-hdf5-diskmap-class?s_tid=FX_rc1_behav

allows to create a handle into an HDF5 file, without actually loading it. This means one can handle very large data sets, larger than memory, and only get selected portions directly from file.

Installation requires first to get:

• https://fr.mathworks.com/matlabcentral/fileexchange/17172-hdf5tools

and then this hdf5prop (which overloads an updated h5datacreate)

Example:

• a=hdf5prop('/home/farhi/dev/iFit/Data/153265_153282.hdf','entry1/data1/DATA')

One needs to know the exact path to data sets.

This could be done in:

• read_hdf5: line 91: val = H5D.read to val = hdf5prop(filename, ...)
• read_hdf5: line 75: h5read(...)

[farhi]

See also:

• https://github.com/venkatachalamlab/lazy-matlab-arrays
• http://dylan-muir.com/articles/mapped_tensor/ and https://github.com/DylanMuir/MappedTensor
• https://github.com/DylanMuir/LazyTensorTransform
• set memmapfile for large data sets ? to be done in iLoad ?
• in-place operations with e.g. x=2x, not y=2x; x=y;

[farhi]

See also #201, which can constitute a more generic way to handle in-memory compressed data.

[farhi]

For HDF in iData/estruct, one could indeed replace full data set read by a call to h5 routines with given subsref indices in initial file. So probably one needs to create an object with its subsref/subsasgn method, and the file handle. Probably NetCDF could also be supported, but probably outdated these days (and this is an HDF now)..