During source extraction stage, patch processing fails at "Generating Background" with dimensionality error

[zbarry]

Environment:

Python 3.6, RHEL 6, PyCharm pydev debugger Fresh clone of CaImAn master branch (commit 6bbade6)

Issue:

caiman.source_extraction.cnmf.map_reduce - run_CNMF_patches() fails when arriving at "Generating background" section of code block.

Error log:

~~ snip ~~ Generating residuals entering the deconvolution 16 out of total 41 temporal components updated 27 out of total 41 temporal components updated 35 out of total 41 temporal components updated 38 out of total 41 temporal components updated 40 out of total 41 temporal components updated 41 out of total 41 temporal components updated 16 out of total 41 temporal components updated 27 out of total 41 temporal components updated 35 out of total 41 temporal components updated 38 out of total 41 temporal components updated 40 out of total 41 temporal components updated 41 out of total 41 temporal components updated stopping: overall temporal component not changing significantly Estimate low rank Background 48 True 448.51756167411804 Transforming patches into full matrix Skipped %d Empty Patch 24 Generating background /data/mibs-inscopix/external/CaImAn-unmod/caiman/source_extraction/cnmf/map_reduce.py:424: RuntimeWarning: divide by zero encountered in true_divide (1. / mask, (np.arange(d), np.arange(d))), dtype=np.float32) Compressing background components with a low rank NMF Traceback (most recent call last): File "/scratch/pycharm/helpers/pydev/pydevd.py", line 1664, in main() File "/scratch/pycharm/helpers/pydev/pydevd.py", line 1658, in main globals = debugger.run(setup['file'], None, None, is_module) File "/scratch/pycharm/helpers/pydev/pydevd.py", line 1085, in run runpy._run_module_as_main(module_name, alter_argv=False) File "/scratch/anaconda3/envs/inscopix/lib/python3.6/runpy.py", line 193, in _run_module_as_main "main", mod_spec) File "/scratch/anaconda3/envs/inscopix/lib/python3.6/runpy.py", line 85, in _run_code exec(code, run_globals) File "/data/mibs-inscopix/src/inscopix/run_caiman.py", line 401, in main() File "/data/mibs-inscopix/src/inscopix/run_caiman.py", line 395, in main args.func(args) File "/data/mibs-inscopix/src/inscopix/run_caiman.py", line 89, in stage_cnmf num_processes=args.numprocs File "/data/mibs-inscopix/src/inscopix/caiman/cnmf.py", line 116, in run_cnmf cnm.fit(y) File "/data/mibs-inscopix/external/CaImAn-unmod/caiman/source_extraction/cnmf/cnmf.py", line 571, in fit del_duplicates=self.del_duplicates) File "/data/mibs-inscopix/external/CaImAn-unmod/caiman/source_extraction/cnmf/map_reduce.py", line 446, in run_CNMF_patches b = np.fmax(Bm.dot(F_tot.dot(f.T)).dot( File "/scratch/anaconda3/envs/inscopix/lib/python3.6/site-packages/scipy/sparse/base.py", line 322, in dot return self * other File "/scratch/anaconda3/envs/inscopix/lib/python3.6/site-packages/scipy/sparse/base.py", line 476, in mul raise ValueError('dimension mismatch') ValueError: dimension mismatch

Debugging information from pydev - dimensionalities:

dataset.hdf5

shape: (400, 540, 720) frames, height, width

my_function_calling_caiman.py

Post-motion correction, where input_filename is a .mmap:

yr, dims, t = cm.load_memmap(input_filename)
y = yr.T.reshape((t,) + dims, order='F')

y.shape: (400, 540, 720)

caiman.source_extraction.cnmf.map_reduce - run_CNMF_patches()

Code on erroring line 449:

except np.linalg.LinAlgError:  # singular matrix
    b = np.fmax(Bm.dot(scipy.linalg.lstsq(f.T, F_tot.T)[0].T), 0)

f.T.shape: (400, 48) F_tot.T.shape: (400, 2304) scipy.linalg.lstsq(f.T, F_tot.T)[0].T.shape: (2304, 48) Bm.shape: (388800, 1152)

scipy.sparse.base - dot()

self.shape: (388800, 1152) other.shape: (2304, 48)

[epnev]

@zbarry Are you using any downsampling? There is a factor of 2 mismatch that makes me wonder

[zbarry]

@epnev - I'm self-downsampling spatially by 2x in the hdf5 file I pass to the motion correction procedure, and I'm using tsub, ssub=2 for CNMF fitting. I agree that's suspicious, although:

As an update, I've realized the issue only cropped up when I freshly cloned the repository. When I went back to my old (slightly modified) CaImAn copy, everything worked fine. I'm going to diff them and look into this more. I'll keep you posted.

[epnev]

Thanks @zbarry . In general, the ssub parameter will only do downsampling internally to boost SNR and increase speed but after processing is done, it will upsample and produce results with the original FOV.

[zbarry]

Ok, I've recloned the repository and checked out the branch I was originally working with:

git log commit f76d8b6e6d19efb9f10c57817a6f167d3d19db11 (HEAD -> master, origin/master, origin/HEAD) Merge: a2c3c84 2859366 Author: eftychios pnevmatikakis epnevmatikakis@gmail.com Date: Thu May 17 18:45:36 2018 -0400 Merge pull request #251 from flatironinstitute/dev fixing issue with tiff files

Procedure to do this for completeness (edit: 2859366010be1f212f0d52581d958060a455564d is a dev branch commit which I accidentally checked out. Both it and the revision I've actually been using (above, f76d8b6e6d19efb9f10c57817a6f167d3d19db11) do not suffer from this issue):

(inscopix) [barryza1@... caiman-diffing]$ git clone https://github.com/flatironinstitute/CaImAn/ Cloning into 'CaImAn'... remote: Counting objects: 17251, done. remote: Compressing objects: 100% (148/148), done. remote: Total 17251 (delta 130), reused 146 (delta 73), pack-reused 17030 Receiving objects: 100% (17251/17251), 439.20 MiB | 9.97 MiB/s, done. Resolving deltas: 100% (11488/11488), done. Checking out files: 100% (257/257), done. (inscopix) [barryza1@... caiman-diffing]$ cd CaImAn (inscopix) [barryza1@... CaImAn]$ git checkout 2859366010be1f212f0d52581d958060a455564d Note: checking out '2859366010be1f212f0d52581d958060a455564d'.

You are in 'detached HEAD' state. You can look around, make experimental changes and commit them, and you can discard any commits you make in this state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may do so (now or later) by using -b with the checkout command again. Example:

git checkout -b

HEAD is now at 28593660 Merge branch 'master' into dev (inscopix) [barryza1@... CaImAn]$ pip uninstall caiman Uninstalling caiman-1.0: Would remove: /scratch/anaconda3/envs/inscopix/lib/python3.6/site-packages/caiman.egg-link Proceed (y/n)? y Successfully uninstalled caiman-1.0 (inscopix) [barryza1@... CaImAn]$ pip install -e . Obtaining file:///data/mibs-inscopix/external/caiman-diffing/CaImAn Installing collected packages: caiman Running setup.py develop for caiman Successfully installed caiman

^ with the above, there is no dimensionality mismatch

Following this with:

(inscopix) [barryza1@... CaImAn]$ pip uninstall caiman Uninstalling caiman-1.0: Would remove: /scratch/anaconda3/envs/inscopix/lib/python3.6/site-packages/caiman.egg-link Proceed (y/n)? y Successfully uninstalled caiman-1.0 (inscopix) [barryza1@... CaImAn]$ git checkout master Previous HEAD position was 28593660 Merge branch 'master' into dev Switched to branch 'master' Your branch is up to date with 'origin/master'. (inscopix) [barryza1@... CaImAn]$ pip install -e . Obtaining file:///data/mibs-inscopix/external/caiman-diffing/CaImAn Installing collected packages: caiman Running setup.py develop for caiman Successfully installed caiman

and running my pipeline will produce the error again.

I've diff'd the caiman.source_extraction.cnmf package and there are a lot of changes since my revision I'm using. Think this is above my pay grade.

[epnev]

@j-friedrich This might be related to #302. Can you take a look when possible?

[j-friedrich]

Apparently using 48 patches with 48 background components each. According to the beginning of the trace Skipped %d Empty Patch 24 for 24 of the patches the function cnmf_patches returns None, so the condition if (np.sum(np.abs(np.diff(images.reshape(timesteps, -1).T)))) > 0.1: in line 127 of map_reduce.py is False. Seems we have this condition to skip empty parts of the FOV where there are no temporal fluorescence changes, but don't deal properly with it downstream. Is that the case for your data?

[zbarry]

@j-friedrich - should mention that the original movie is 3000 frames long, and here, I've truncated it to 400 just for pipeline testing. I'm pretty sure that's why so many patches are empty. Also, not sure exactly what "the case" is specifically referring to here, sorry!

[j-friedrich]

Nevertheless, seems bit odd that half of the FOV is empty. Regarding the dimensionality mismatch, F_tot is initialized assuming there are no empty patches as is almost always the case. Does inserting
F_tot = F_tot[:count_bgr] before or after the similar lines 403-404 C_tot = C_tot[:count, :] YrA_tot = YrA_tot[:count, :] fix the issue?

[zbarry]

Should mention that I'm working with 1p data, and I don't know about what these datasets generally look like (I'm new to neuroimaging), but ours definitely have large empty patches. Here's an example of a characteristic correlation map from our data:

https://gist.github.com/zbarry/8392882436e9ab658f20e4982ed02d0f/raw/833b3b262a5d1e7c65dadfd9ae7b1a913d66e5b1/caiman_correlation_image.png

This on top of barely sampling the full dataset by truncating the time series would produce the tons of empty patches that we're seeing, I think.

Will try those mods soon when I get a chance!

[zbarry]

Update: the CNMF fitting process executes completely without error after adding your F_tot = F_tot[:count_bgr] mod. I haven't run the analysis on a full length movie yet, so I can't compare to my previous results, but the contours that I'm getting out look sensical. I'll run the movies in their entirety to check for consistency with the old results.

[epnev]

I added this line with https://github.com/flatironinstitute/CaImAn/commit/0fb81a8fd4b58c0a5b568a5e87e9ce36be980ee9 and am closing. Reopen if the problem remain.