Question: running motion correction on volumetric data

[ssapir]

For better support, please use the template below to submit your issue. When your issue gets resolved please remember to close it.

• Tell us a bit about your setup:

  1. Operating system (Linux/macOS/Windows): linux

  2. Python version (3.x): 3.9.6

  3. Working environment (Python IDE/Jupyter Notebook/other): other (python script on a cluster)

  4. Which of the demo scripts you're using for your analysis (if applicable): https://github.com/flatironinstitute/CaImAn/blob/master/demos/notebooks/demo_caiman_cnmf_3D.ipynb https://github.com/liorgolgher/CaImAn/blob/master/demos/notebooks/demo_motion_correction.py

  5. CaImAn version*: 1.9.7

  6. CaImAn installation process (pip install ./pip install -e ./conda): pip install -e

• If you're not reporting an error, type your message below We have a working pipeline for 2D. Based on your instructions, our preprocessing stage, in the 2D recordings, create multiple 1GB tiff files, passed to the motion correction stage, and joined afterwards using save_memmap_join function.

I am trying to add a caiman pipeline on 3D + Time 2p calcium imaging, for ~100GB recordings. I use the following demo files as an example: https://github.com/flatironinstitute/CaImAn/blob/master/demos/notebooks/demo_caiman_cnmf_3D.ipynb and https://github.com/liorgolgher/CaImAn/blob/master/demos/notebooks/demo_motion_correction.py

The demo creates a single 4D tiff file (or a single HDF5 file). Assuming your instruction for multiple small tiff files holds, I didn't find closed issues or in the documentation defining the format and name of the tiff files in the 4D case, to make sure the analysis correctly identifies the dimentions when analysing 3D motion correction.

Can you please help me adjusting the pipeline? Should we use a single HDF5 in this case? If we should still use multiple tiffs, what is the order of the dimentions? Since the tiffs are limited by number of frames (due to size), how should we create the data files to make sure the correct order is preserved?

Thank you for your help

[j-friedrich]

While you can provide a list of tiff files, using a single HDF5 file apparently works best in this case.

I considered the 3D demo but pretended to have 100 tiff files:

fname1=[fname]*100
cm.load(fname1, is3D=True).save('Y.hdf5')
fname2 = ['Y.hdf5']

Motion correcting the list of tiff files fname1 was about 3 times slower than processing hdf5 fname2. The latter approach made better use of the available computing cores.

The difference is less drastic if the individual tiff files include more frames, but as you point out the tiffs are limited by number of frames, especially if the number of voxels is large.