SilverLabUCL / PySilverLabNWB

Python tools for working with Silver Lab data in the NWB2 format
MIT License
1 stars 0 forks source link

compress NWB2 data with gzip #25

Open alessandrofelder opened 4 years ago

alessandrofelder commented 4 years ago

Current size differences between generated NWB1 and NWB2 files that we write using pynwb are (at least to a large part) explained by the fact that PySilverLabNWB does not do any compression, while the NWB1 data is compressed with GZIP level 4.

https://pynwb.readthedocs.io/en/latest/tutorials/general/advanced_hdf5_io.html#compression-and-other-i-o-filters explains how pynwb can be used to specify compression of a dataset.

ageorgou commented 4 years ago

Unfortunately, it seems that we can't specify the compression when we write the file, but rather we need to do it when creating each Dataset.

However, since most of our data is acquisitions, we can change NWBFile::add_time_series_data to use a wrapped Dataset, as in the link above. I think that should take care of the bulk of changes.

ageorgou commented 4 years ago

Partially addressed in #30, #31. May still be able to do something more.