"ValueError: operands could not be broadcast...." in components_evaluation.py: 324 when `method_init` set to `corr_pnr`

[ajkswamy]

For better support, please use the template below to submit your issue. When your issue gets resolved please remember to close it.

Sometimes errors while running CNMF occur during parallel processing which prevents the log to provide a meaningful error message. Please reproduce your error with setting dview=None.

If you need to upgrade CaImAn follow the instructions given in the documentation.

• Tell us a bit about your setup:

  1. Operating system (Linux/macOS/Windows): Linux

  2. Python version (3.x): 3.7

  3. Working environment (Python IDE/Jupyter Notebook/other): Terminal

  4. Which of the demo scripts you're using for your analysis (if applicable):

  5. CaImAn version*: 1.8.5

  6. CaImAn installation process (pip install ./pip install -e ./conda): conda *You can get the CaImAn version by creating a params object and then typing params.data['caiman_version']. If the field doesn't exist, type N/A and consider upgrading)

• Describe the issue that you are experiencing Data: MMAP file

Code:

import caiman
import numpy as np

fname_new = "tmpy4htcouu_d1_106_d2_153_d3_1_order_C_frames_576_.mmap"
Yr, dims, T = caiman.load_memmap(fname_new)
images = np.reshape(Yr.T, [T] + list(dims), order='F')

c, dview, n_processes = caiman.cluster.setup_cluster(
    backend='local', n_processes=None, single_thread=False)

opts = caiman.source_extraction.cnmf.params.CNMFParams(params_dict={"method_init": "corr_pnr"})

cnm = caiman.source_extraction.cnmf.CNMF(n_processes, params=opts, dview=dview)

cnm = cnm.fit(images)

cnm2 = cnm.refit(images, dview=dview)

cnm2.estimates.evaluate_components(images, cnm2.params, dview=dview)

• Copy error log below

  /home/aj/Installations/anaconda3/envs/test/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:516: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
  /home/aj/Installations/anaconda3/envs/test/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:517: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
  /home/aj/Installations/anaconda3/envs/test/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:518: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
  /home/aj/Installations/anaconda3/envs/test/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:519: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
  /home/aj/Installations/anaconda3/envs/test/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:520: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
  /home/aj/Installations/anaconda3/envs/test/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
  /home/aj/Installations/anaconda3/envs/test/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
  /home/aj/Installations/anaconda3/envs/test/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
  /home/aj/Installations/anaconda3/envs/test/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
  /home/aj/Installations/anaconda3/envs/test/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
  /home/aj/Installations/anaconda3/envs/test/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
  /home/aj/Installations/anaconda3/envs/test/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
  WARNING:root:using CNMF-E's ringmodel for background hence setting key normalize_init in group init automatically to False.
  /home/aj/Installations/anaconda3/envs/test/lib/python3.7/site-packages/numpy/core/_asarray.py:83: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray
  return array(a, dtype, copy=False, order=order)
  /home/aj/Installations/anaconda3/envs/test/lib/python3.7/site-packages/scipy/sparse/dia.py:311: RuntimeWarning: divide by zero encountered in remainder
  c = np.arange(num_rows, dtype=np.intc) - (offsets % max_dim)[:, None]
  1
  /home/aj/Installations/anaconda3/envs/test/lib/python3.7/site-packages/sklearn/decomposition/_nmf.py:1077: ConvergenceWarning: Maximum number of iterations 200 reached. Increase it to improve convergence.
  " improve convergence." % max_iter, ConvergenceWarning)
  /home/aj/Installations/anaconda3/envs/test/lib/python3.7/site-packages/caiman/source_extraction/cnmf/initialization.py:1376: FutureWarning: `rcond` parameter will change to the default of machine precision times ``max(M, N)`` where M and N are the input matrix dimensions.
  To use the future default and silence this warning we advise to pass `rcond=None`, to keep using the old, explicitly pass `rcond=-1`.
  f_in = np.linalg.lstsq(b_in, B)[0]
  /home/aj/Installations/anaconda3/envs/test/lib/python3.7/site-packages/caiman/components_evaluation.py:819: UserWarning: There were no components to evaluate. Check your parameter settings.
  warnings.warn("There were no components to evaluate. Check your parameter settings.")
  USING MODEL:/home/aj/caiman_data/model/cnn_model.json
  WARNING:tensorflow:From /home/aj/Installations/anaconda3/envs/test/lib/python3.7/site-packages/tensorflow/python/ops/init_ops.py:97: calling VarianceScaling.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
  Instructions for updating:
  Call initializer instance with the dtype argument instead of passing it to the constructor
  WARNING:tensorflow:From /home/aj/Installations/anaconda3/envs/test/lib/python3.7/site-packages/tensorflow/python/ops/init_ops.py:97: calling VarianceScaling.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
  Instructions for updating:
  Call initializer instance with the dtype argument instead of passing it to the constructor
  WARNING:tensorflow:From /home/aj/Installations/anaconda3/envs/test/lib/python3.7/site-packages/tensorflow/python/ops/init_ops.py:97: calling Zeros.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
  Instructions for updating:
  Call initializer instance with the dtype argument instead of passing it to the constructor
  WARNING:tensorflow:From /home/aj/Installations/anaconda3/envs/test/lib/python3.7/site-packages/tensorflow/python/ops/init_ops.py:97: calling Zeros.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
  Instructions for updating:
  Call initializer instance with the dtype argument instead of passing it to the constructor
  2020-09-01 17:24:34.111771: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
  2020-09-01 17:24:34.130010: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 3601000000 Hz
  2020-09-01 17:24:34.130464: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x562d42f4d080 executing computations on platform Host. Devices:
  2020-09-01 17:24:34.130474: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): <undefined>, <undefined>
  2020-09-01 17:24:34.141188: W tensorflow/compiler/jit/mark_for_compilation_pass.cc:1412] (One-time warning): Not using XLA:CPU for cluster because envvar TF_XLA_FLAGS=--tf_xla_cpu_global_jit was not set.  If you want XLA:CPU, either set that envvar, or use experimental_jit_scope to enable XLA:CPU.  To confirm that XLA is active, pass --vmodule=xla_compilation_cache=1 (as a proper command-line flag, not via TF_XLA_FLAGS) or set the envvar XLA_FLAGS=--xla_hlo_profile.
  Traceback (most recent call last):
  File "1.py", line 19, in <module>
  cnm2.estimates.evaluate_components(images, cnm2.params, dview=dview)
  File "/home/aj/Installations/anaconda3/envs/test/lib/python3.7/site-packages/caiman/source_extraction/cnmf/estimates.py", line 1020, in evaluate_components
  min_SNR_reject=opts['SNR_lowest'])
  File "/home/aj/Installations/anaconda3/envs/test/lib/python3.7/site-packages/caiman/components_evaluation.py", line 629, in estimate_components_quality_auto
  use_cnn, gSig_range)
  File "/home/aj/Installations/anaconda3/envs/test/lib/python3.7/site-packages/caiman/components_evaluation.py", line 668, in select_components_from_metrics
  predictions, _ = evaluate_components_CNN(A, dims, gSig)
  File "/home/aj/Installations/anaconda3/envs/test/lib/python3.7/site-packages/caiman/components_evaluation.py", line 324, in evaluate_components_CNN
  coms = np.maximum(coms, half_crop)
  ValueError: operands could not be broadcast together with shapes (0,) (2,)

• If you're not reporting an error, type your message below

[j-friedrich]

With your parameter settings no components have been detected, therefore evaluate_components fails. Please use parameters that are appropriate for your dataset. When using corr_pnr as initialization method (usually used for microendoscopic 1p data) the crucial parameters controlling the number of detected components are min_pnr and min_corr, cf. https://github.com/flatironinstitute/CaImAn/blob/master/demos/notebooks/demo_pipeline_cnmfE.ipynb

[ajkswamy]

Thank you for diagnosing the problem and for the advice. I will check if setting these parameters help. Otherwise, I will reopen the issue.