Mesoscope Lossless Compression

[k1o0]

• [x] Look at TensorStore compression
• [ ] Look at NWB
• [x] Look at Zarr with different back ends
• [x] C.f. Krumin's results (bzip2, ask him which sessions)
• [x] Compression task with validation (read after write)
• [x] Test for compression
• [x] Look at which methods allow random access

NWB cannibalizes and compresses the tif files. This may be most convenient for us as we could stream the data (although you can randomly access bzip2 files, we'd have to write an interface for random access of tif data). Suite2p also supports NWB as an input. It remains to be seen whether NWB fully supports mesoscope data specifically.

Determine whether switching to NWB is viable. This would allow streaming of raw data and aligns with community standards. This item is a fair bit of work.

Lossless: switch to NWB

• [ ] check that the NWB file contains the metadata from ScanImage
• [ ] check that Suite2P works with the ScanImage
• [ ] document results in PPT

[k1o0]

NWB seems more and more like the format to use. This would potentially replace the bzip2 compression. Some things to try:

• [ ] Convert tiffs to NWB for 1 session
• [ ] Check data can be streamed, compressed, and that raw tiff data in tact
• [ ] Check suite2p can be run with NWB input (it does for standard 2p data)
• [ ] Consider dataset type name
• [ ] Create task for compressing to NWB