search

flatironinstitute / CaImAn

Computational toolbox for large scale Calcium Imaging Analysis, including movie handling, motion correction, source extraction, spike deconvolution and result visualization.
https://caiman.readthedocs.io
GNU General Public License v2.0
640 stars 370 forks source link

infs or nans error in cnmf.fit() #513

Closed davegrays closed 5 years ago

davegrays commented 5 years ago

Ubuntu 18, python 3.6.7, tested on Jupyter and command line

ValueError running cmnf.fit() on calcium videos with no nans, infs, negative values, or pixels with temporal stdev = 0

2019-04-18 19:09:38,337 [114] INFO root: Memory mapping 2019-04-18 19:09:38,339 [114] INFO root: Updating Spatial Components using lasso lars 2019-04-18 19:09:38,979 [114] INFO root: thresholding components 2019-04-18 19:09:38,986 [114] INFO root: Computing residuals 2019-04-18 19:09:38,987 [114] INFO root: Updating done in 0s 2019-04-18 19:09:38,988 [114] INFO root: Removing created tempfiles 2019-04-18 19:09:38,990 [114] INFO root: update temporal ... 2019-04-18 19:09:38,991 [114] INFO root: deconvolution ... 2019-04-18 19:09:38,993 [114] INFO root: Generating residuals 2019-04-18 19:09:38,999 [114] INFO root: entering the deconvolution 2019-04-18 19:09:39,001 [114] INFO root: 1 out of total 4 temporal components updated 2019-04-18 19:09:39,003 [114] INFO root: 3 out of total 4 temporal components updated 2019-04-18 19:09:39,006 [114] INFO root: 4 out of total 4 temporal components updated 2019-04-18 19:09:39,008 [114] INFO root: 1 out of total 4 temporal components updated 2019-04-18 19:09:39,011 [114] INFO root: 3 out of total 4 temporal components updated 2019-04-18 19:09:39,012 [114] INFO root: 4 out of total 4 temporal components updated 2019-04-18 19:09:39,014 [114] INFO root: stopping: overall temporal component not changing significantly 2019-04-18 19:09:39,016 [114] INFO root: refinement... 2019-04-18 19:09:39,017 [114] INFO root: merging components ... 2019-04-18 19:09:39,019 [114] WARNING py.warnings: /caiman/caiman/source_extraction/cnmf/merging.py:142: SparseEfficiencyWarning: Changing the sparsity structure of a csc_matrix is expensive. lil_matrix is more efficient. C_corr[ii, overlap_indeces] = corr_values

2019-04-18 19:09:39,046 [114] INFO root: Merging components [0 1 2] 2019-04-18 19:09:39,059 [114] INFO root: Updating spatial ... 2019-04-18 19:09:39,060 [114] INFO root: Computing support of spatial components 2019-04-18 19:09:39,065 [114] WARNING py.warnings: /caiman/caiman/source_extraction/cnmf/spatial.py:903: SparseEfficiencyWarning: Changing the sparsity structure of a csc_matrix is expensive. lil_matrix is more efficient. dist_indicator[:, i] = scipy.sparse.coo_matrix(np.squeeze(np.reshape(A_temp, (d, 1)))[:,None] > 0)

2019-04-18 19:09:39,099 [114] INFO root: Memory mapping 2019-04-18 19:09:39,101 [114] INFO root: Updating Spatial Components using lasso lars 2019-04-18 19:09:39,711 [114] INFO root: thresholding components 2019-04-18 19:09:39,716 [114] INFO root: Computing residuals 2019-04-18 19:09:39,717 [114] INFO root: Updating done in 0s 2019-04-18 19:09:39,718 [114] INFO root: Removing created tempfiles 2019-04-18 19:09:39,721 [114] INFO root: update temporal ... 2019-04-18 19:09:39,724 [114] INFO root: Generating residuals 2019-04-18 19:09:39,730 [114] INFO root: entering the deconvolution 2019-04-18 19:09:39,733 [114] INFO root: 1 out of total 2 temporal components updated 2019-04-18 19:09:39,736 [114] INFO root: 2 out of total 2 temporal components updated 2019-04-18 19:09:39,740 [114] INFO root: 1 out of total 2 temporal components updated 2019-04-18 19:09:39,743 [114] INFO root: 2 out of total 2 temporal components updated 2019-04-18 19:09:39,745 [114] INFO root: stopping: overall temporal component not changing significantly 2019-04-18 19:09:39,753 [114] WARNING py.warnings: /usr/local/lib/python3.6/dist-packages/numpy/core/memmap.py:331: FutureWarning: Using a non-tuple sequence for multidimensional indexing is deprecated; use arr[tuple(seq)] instead of arr[seq]. In the future this will be interpreted as an array index, arr[np.array(seq)], which will result either in an error or a different result. res = super(memmap, self).getitem(index)

2019-04-18 19:09:39,778 [114] WARNING root: Parallel processing in a single patch is not available for loaded in memory or sliced data. 2019-04-18 19:09:39,802 [114] INFO root: (300, 31, 31) 2019-04-18 19:09:39,803 [114] INFO root: Using 1 processes 2019-04-18 19:09:39,804 [114] INFO root: using 225 pixels per process 2019-04-18 19:09:39,805 [114] INFO root: using 5000 block_size_spat 2019-04-18 19:09:39,806 [114] INFO root: using 5000 block_size_temp 2019-04-18 19:09:39,807 [114] INFO root: preprocessing ... 2019-04-18 19:09:39,809 [114] INFO root: Checking for missing data entries (NaN) 2019-04-18 19:09:39,855 [114] INFO root: initializing ... 2019-04-18 19:09:39,856 [114] INFO root: Variance Normalization 2019-04-18 19:09:39,858 [114] INFO root: Roi Initialization... 2019-04-18 19:09:39,859 [114] INFO root: Greedy initialization of spatial and temporal components using spatial Gaussian filtering 2019-04-18 19:09:39,877 [114] INFO root: Using rolling sum for initialization (RollingGreedyROI) 2019-04-18 19:09:39,993 [114] INFO root: Refining Components using HALS NMF iterations 2019-04-18 19:09:39,998 [114] WARNING py.warnings: /caiman/caiman/source_extraction/cnmf/initialization.py:864: RuntimeWarning: invalid value encountered in true_divide V[m, m]), 0, np.inf)

2019-04-18 19:09:40,054 [114] INFO root: update spatial ... 2019-04-18 19:09:40,056 [114] INFO root: Computing support of spatial components 2019-04-18 19:09:40,060 [114] WARNING py.warnings: /caiman/caiman/source_extraction/cnmf/spatial.py:903: RuntimeWarning: invalid value encountered in greater dist_indicator[:, i] = scipy.sparse.coo_matrix(np.squeeze(np.reshape(A_temp, (d, 1)))[:,None] > 0)

2019-04-18 19:09:40,100 [114] INFO root: Memory mapping 2019-04-18 19:09:40,102 [114] INFO root: Updating Spatial Components using lasso lars 2019-04-18 19:09:40,112 [114] INFO root: thresholding components 2019-04-18 19:09:40,119 [114] INFO root: eliminating 5 empty spatial components 2019-04-18 19:09:40,121 [114] INFO root: Computing residuals 2019-04-18 19:09:40,123 [114] INFO root: Updating done in 0s 2019-04-18 19:09:40,124 [114] INFO root: Removing created tempfiles 2019-04-18 19:09:40,126 [114] INFO root: update temporal ... 2019-04-18 19:09:40,127 [114] INFO root: deconvolution ... 2019-04-18 19:09:40,129 [114] INFO root: Generating residuals 2019-04-18 19:09:40,134 [114] INFO root: entering the deconvolution

ValueError Traceback (most recent call last)

in 6 # Do really from scratch, with no parallel processing (ie just replaced dview with None: 7 get_ipython().run_line_magic('pdb', '') ----> 8 cnm = CalciumMethods.do_cnmf(config, images, None) /pipeline/pipeline/calcium/CalciumMethods.py in do_cnmf(config, images, dview) 128 config.caiman_params.change_params({'p': 0}) # deconvolution is turned off (p=0) 129 cnm = cnmf.CNMF(n_processes=n_processes, params=config.caiman_params, dview=dview) --> 130 cnm = cnm.fit(images) 131 return cnm 132 /caiman/caiman/source_extraction/cnmf/cnmf.py in fit(self, images, indeces) 540 low_rank_background=self.params.get('patch', 'low_rank_background'), 541 del_duplicates=self.params.get('patch', 'del_duplicates'), --> 542 indeces=indeces) 543 544 self.estimates.bl, self.estimates.c1, self.estimates.g, self.estimates.neurons_sn = None, None, None, None /caiman/caiman/source_extraction/cnmf/map_reduce.py in run_CNMF_patches(file_name, shape, params, gnb, dview, memory_fact, border_pix, low_rank_background, del_duplicates, indeces) 254 255 else: --> 256 file_res = list(map(cnmf_patches, args_in)) 257 258 logging.info('Elapsed time for processing patches: \ /caiman/caiman/source_extraction/cnmf/map_reduce.py in cnmf_patches(args_in) 132 cnm = cnmf.CNMF(n_processes=1, params=opts) 133 --> 134 cnm = cnm.fit(images) 135 return [idx_, shapes, scipy.sparse.coo_matrix(cnm.estimates.A), 136 cnm.estimates.b, cnm.estimates.C, cnm.estimates.f, /caiman/caiman/source_extraction/cnmf/cnmf.py in fit(self, images, indeces) 487 logging.info('deconvolution ...') 488 --> 489 self.update_temporal(Yr) 490 491 if not self.skip_refinement: /caiman/caiman/source_extraction/cnmf/cnmf.py in update_temporal(self, Y, use_init, **kwargs) 832 self.estimates.g, self.estimates.YrA, self.estimates.lam = update_temporal_components( 833 Y, self.estimates.A, self.estimates.b, self.estimates.C, self.estimates.f, dview=self.dview, --> 834 **self.params.get_group('temporal')) 835 return self 836 /caiman/caiman/source_extraction/cnmf/temporal.py in update_temporal_components(Y, A, b, Cin, fin, bl, c1, g, sn, nb, ITER, block_size_temp, num_blocks_per_run_temp, debug, dview, **kwargs) 213 logging.info("entering the deconvolution ") 214 C, S, bl, YrA, c1, sn, g, lam = update_iteration(parrllcomp, len_parrllcomp, nb, C, S, bl, nr, --> 215 ITER, YrA, c1, sn, g, Cin, T, nA, dview, debug, AA, kwargs) 216 ff = np.where(np.sum(C, axis=1) == 0) # remove empty components 217 if np.size(ff) > 0: # Eliminating empty temporal components /caiman/caiman/source_extraction/cnmf/temporal.py in update_iteration(parrllcomp, len_parrllcomp, nb, C, S, bl, nr, ITER, YrA, c1, sn, g, Cin, T, nA, dview, debug, AA, kwargs) 390 dview.results.clear() 391 --> 392 if scipy.linalg.norm(Cin - C, 'fro') <= 1e-3*scipy.linalg.norm(C, 'fro'): 393 logging.info("stopping: overall temporal component not changing" + 394 " significantly") /usr/local/lib/python3.6/dist-packages/scipy/linalg/misc.py in norm(a, ord, axis, keepdims) 135 """ 136 # Differs from numpy only in non-finite handling and the use of blas. --> 137 a = np.asarray_chkfinite(a) 138 139 # Only use optimized norms if axis and keepdims are not specified. /usr/local/lib/python3.6/dist-packages/numpy/lib/function_base.py in asarray_chkfinite(a, dtype, order) 496 if a.dtype.char in typecodes['AllFloat'] and not np.isfinite(a).all(): 497 raise ValueError( --> 498 "array must not contain infs or NaNs") 499 return a 500 ValueError: array must not contain infs or NaNs > /usr/local/lib/python3.6/dist-packages/numpy/lib/function_base.py(498)asarray_chkfinite() 496 if a.dtype.char in typecodes['AllFloat'] and not np.isfinite(a).all(): 497 raise ValueError( --> 498 "array must not contain infs or NaNs") 499 return a 500
epnev commented 5 years ago

@davegrays My guess is that the patch that throws the error might have not components. What is the value of rf that you're using, and do you have large areas in your FOV with no activity?

davegrays commented 5 years ago

@epnev Yes, there are large areas with no activity because the tissue is enclosed by the bounding box. Increasing the rf to 60 still results in errors like these. The attached image shows the first (120x120) patch that produces this error. How can we tell caiman to ignore empty patches like these? Is it really necessary to fail when NaNs are produced, or can it just move on to the next patch?

calcium_screenshot_crop
epnev commented 5 years ago

@davegrays Thanks. I thought we had dealt with this is the past, but apparently not. I'll take a look tomorrow. In the meantime, you can try using a larger rf value to make sure more signal is captured within each patch.

epnev commented 5 years ago

@davegrays I actually could not reproduce your error in a file where I intentionally filled a patch which random noise (or just zeros). The pipeline detects no neurons and leaves the results empty. So it's not exactly clear what's happening.

I created a new branch issue_513 with the changes shown here: https://github.com/flatironinstitute/CaImAn/commit/5f53fc7b53ccbdca93d84ab60b4e4f9a56403e64

Can you tell me if you can process your file with no errors by using this branch? Thanks

rbedi commented 5 years ago

@epnev Your fix got us past that error, but we hit another NaN error during NMF on the background component. Proposed a fix here: #517 - is this a reasonable way to get around this problem? Thanks!

epnev commented 5 years ago

I merged everything into dev. I'm closing this issue but feel free to reopen if you still see the same problem.