BouchardLab / nsds_lab_to_nwb

Python package to convert NSDS Lab data to NWB files.
https://nsds-lab-to-nwb.readthedocs.io/en/latest/
0 stars 4 forks source link

[WIP] improvements to tdt reading and nwb saving #124

Closed JesseLivezey closed 2 years ago

JesseLivezey commented 2 years ago

Description and related issues

Calling tdt.read_block() by default loads all data. This can be very slow if you just need some metadata and not all of the neural data. Relatedly, creating multiple TDTReader objects across classes will mean that all of the data gets loaded every time it is used, even if you only need the mark track.

This PR makes the TDTReader somewhat lazy, changes the order of things in the NWBBuilder so wait until the last step to load any neural/mark data (makes debugging faster if metadata steps fail), and compresses the timeseries data which makes files 50-80% of their original size (should also make saving/loading faster, but need to test that).

Checklist:

JesseLivezey commented 2 years ago

Outdated, closing and the compression can be pulled in at another time.