search

flatironinstitute / CaImAn

Computational toolbox for large scale Calcium Imaging Analysis, including movie handling, motion correction, source extraction, spike deconvolution and result visualization.
https://caiman.readthedocs.io
GNU General Public License v2.0
630 stars 365 forks source link

Enhancing dot product for sparse matrices during ROI spatial updating #1060

Open oterocoronel opened 1 year ago

oterocoronel commented 1 year ago

When trying to refine the spatial components using the ellipse mode, these lines (lines 891-892 in spatial.py) made it crash because it couldn't allocate enough space in memory:

 cm[:, i] = old_div(
                    np.dot(Coor[c], A[:, :nr].todense()), A[:, :nr].sum(axis=0))

In particular, A[:, :nr].todense() doesn't seem to scale well with large recordings, since it requires allocating in memory a volume of shape = n_rois x dim1 x dim2 (x dim3). In my case, I am working with a 4D dataset and using CNMF-3D, so the size was ~ 10000 x 600 x 600 x 15 and that's why I ran OOM. But I think this may even be an issue for large/dense 2D recordings and/or users with limited RAM. I managed to overcome this issue by calling the "dot" function of the sparse matrix, so my new version looks like this:

 cm[:, i] = old_div(
                    A[:, :nr].T.dot(Coor[c].T).T, A[:, :nr].sum(axis=0))

Seems to be working just fine. Letting you guys know in case it's useful for others.

EricThomson commented 1 year ago

Again, thanks for this suggestion. Will definitely look into it.