search

flatironinstitute / CaImAn

Computational toolbox for large scale Calcium Imaging Analysis, including movie handling, motion correction, source extraction, spike deconvolution and result visualization.
https://caiman.readthedocs.io
GNU General Public License v2.0
623 stars 364 forks source link

Broadcasting error when embedding patch results into full FoV #521

Closed zbarry closed 5 years ago

zbarry commented 5 years ago

CNMF-E stage fails with an array broadcasting error after individual patch processing is finished and patches are trying to be integrated.

Environment

CentOS 7 / Jupyter notebook / Python 3.6

# Name                    Version                   Build  Channel
alabaster                 0.7.12                   py36_0  
altair                    2.4.1                    py36_0    conda-forge
asn1crypto                0.24.0                   py36_0  
astroid                   2.2.5                    py36_0  
atomicwrites              1.3.0                    py36_1  
attrs                     19.1.0                   py36_1  
babel                     2.6.0                    py36_0  
backcall                  0.1.0                    py36_0  
bcrypt                    3.1.6                    pypi_0    pypi
biopython                 1.73                     pypi_0    pypi
blas                      1.0                         mkl  
bleach                    3.1.0                    py36_0  
bokeh                     1.1.0                    py36_0  
bzip2                     1.0.6                h14c3975_5  
ca-certificates           2019.1.23                     0  
caiman                    1.0                       dev_0    <develop>
cairo                     1.14.12              h7636065_2  
certifi                   2019.3.9                 py36_0  
cffi                      1.12.2           py36h2e261b9_1  
chardet                   3.0.4                    py36_1  
click                     7.0                      py36_0  
cloudpickle               0.8.1                      py_0  
colorama                  0.4.1                    pypi_0    pypi
colorcet                  2.0.1                      py_0    pyviz/label/dev
cryptography              2.3.1            py36hc365091_0  
cycler                    0.10.0                   py36_0  
cython                    0.29.7           py36he6710b0_0  
cytoolz                   0.9.0.1          py36h14c3975_1  
dask                      1.2.0                      py_0  
dask-core                 1.2.0                      py_0  
dataclasses               0.6                      pypi_0    pypi
datashader                0.7.0                      py_0    pyviz/label/dev
datashape                 0.5.4                    py36_1  
datatoolkit               0.0.1                     dev_0    <develop>
dbus                      1.13.6               h746ee38_0  
decorator                 4.4.0                    py36_1  
defusedxml                0.5.0                    py36_1  
distributed               1.27.0                   py36_0  
docutils                  0.14                     py36_0  
entrypoints               0.3                      py36_0  
expat                     2.2.6                he6710b0_0  
ffmpeg                    4.0                  h04d0a96_0  
filelock                  3.0.10                   pypi_0    pypi
flatbuffers               1.10                     pypi_0    pypi
fontconfig                2.12.6               h49f89f6_0  
freetype                  2.8                  hab7d2ae_1  
funcsigs                  1.0.2                    pypi_0    pypi
future                    0.17.1                   pypi_0    pypi
glib                      2.56.2               hd408876_0  
gmp                       6.1.2                h6c8ec71_1  
graphite2                 1.3.13               h23475e2_0  
gst-plugins-base          1.14.0               hbbd80ab_1  
gstreamer                 1.14.0               hb453b48_1  
h5py                      2.8.0            py36h989c5e5_3  
harfbuzz                  1.7.6                h5f0a787_1  
hdf5                      1.10.2               hba1933b_1  
heapdict                  1.0.0                    py36_2  
holoviews                 1.12.1                     py_0    pyviz/label/dev
icu                       58.2                 h9c2bf20_1  
idna                      2.8                      py36_0  
imageio                   2.5.0                    py36_0  
imagesize                 1.1.0                    py36_0  
inscopix-analysis         0.0.1                     dev_0    <develop>
intel-openmp              2019.3                      199  
ipykernel                 5.1.0            py36h39e3cac_0  
ipyparallel               6.2.3                    py36_0  
ipython                   7.4.0            py36h39e3cac_0  
ipython_genutils          0.2.0                    py36_0  
ipywidgets                7.4.2                    py36_0  
isort                     4.3.17                   py36_0  
jasper                    1.900.1              hd497a04_4  
jedi                      0.13.3                   py36_0  
jeepney                   0.4                      py36_0  
jinja2                    2.10.1                   py36_0  
jpeg                      9b                   h024ee3a_2  
jpkl                      0.0.1                     dev_0    <develop>
jsonschema                3.0.1                    py36_0  
jupyter_client            5.2.4                    py36_0  
jupyter_core              4.4.0                    py36_0  
keyring                   18.0.0                   py36_0  
kiwisolver                1.0.1            py36hf484d3e_0  
lazy-object-proxy         1.3.1            py36h14c3975_2  
libedit                   3.1.20181209         hc058e9b_0  
libffi                    3.2.1                hd88cf55_4  
libgcc-ng                 8.2.0                hdf63c60_1  
libgfortran-ng            7.3.0                hdf63c60_0  
libopencv                 3.4.1                h1a3b859_1  
libopus                   1.3                  h7b6447c_0  
libpng                    1.6.36               hbc83047_0  
libprotobuf               3.5.2                h6f1eeef_0  
libsodium                 1.0.16               h1bed415_0  
libstdcxx-ng              8.2.0                hdf63c60_1  
libtiff                   4.0.10               h2733197_2  
libvpx                    1.7.0                h439df22_0  
libxcb                    1.13                 h1bed415_1  
libxml2                   2.9.9                he19cac6_0  
llvmlite                  0.28.0           py36hd408876_0  
locket                    0.2.0                    py36_1  
markdown                  3.0.1                    py36_0  
markupsafe                1.1.1            py36h7b6447c_0  
matplotlib                2.2.2            py36h0e671d2_1  
mccabe                    0.6.1                    py36_1  
mistune                   0.8.4            py36h7b6447c_0  
mkl                       2019.3                      199  
mkl_fft                   1.0.10           py36ha843d7b_0  
mkl_random                1.0.2            py36hd81dba3_0  
more-itertools            6.0.0                    py36_0  
msgpack-python            0.6.1            py36hfd86e86_1  
multipledispatch          0.6.0                    py36_0  
natsort                   6.0.0                      py_0  
nbconvert                 5.4.1                    py36_3  
nbformat                  4.4.0                    py36_0  
ncurses                   6.1                  he6710b0_1  
networkx                  2.3                        py_0  
notebook                  5.7.8                    py36_0  
numba                     0.43.1           py36h962f231_0  
numpy                     1.16.2           py36h7e9f1db_0  
numpy-base                1.16.2           py36hde5b4d6_0  
numpydoc                  0.8.0                    py36_0  
olefile                   0.46                     py36_0  
opencv                    3.4.1            py36h6fd60c2_2  
openssl                   1.0.2r               h7b6447c_0  
packaging                 19.0                     py36_0  
pandas                    0.24.2           py36he6710b0_0  
pandas-flavor             0.1.2                    pypi_0    pypi
pandoc                    2.2.3.2                       0  
pandocfilters             1.4.2                    py36_1  
panel                     0.5.1                      py_0    pyviz/label/dev
param                     1.9.0                      py_0    pyviz/label/dev
paramiko                  2.4.2                    pypi_0    pypi
parso                     0.4.0                      py_0  
partd                     0.3.10                   py36_1  
patsy                     0.5.1                    py36_0  
pcre                      8.43                 he6710b0_0  
peakutils                 1.3.2                      py_0    conda-forge
pexpect                   4.7.0                    py36_0  
phantomjs                 2.1.1                         0    pyviz/label/dev
pickleshare               0.7.5                    py36_0  
pillow                    5.1.0            py36h3deb7b8_0  
pims                      0.4.1                      py_1    conda-forge
pip                       19.0.3                   py36_0  
pipeline                  0.0.1                     dev_0    <develop>
pixman                    0.38.0               h7b6447c_0  
pluggy                    0.9.0                    py36_0  
prometheus_client         0.6.0                    py36_0  
prompt_toolkit            2.0.9                    py36_0  
psutil                    5.6.1            py36h7b6447c_0  
ptyprocess                0.6.0                    py36_0  
py                        1.8.0                    py36_0  
py-opencv                 3.4.1            py36h0676e08_1  
pyasn1                    0.4.5                    pypi_0    pypi
pycodestyle               2.5.0                    py36_0  
pycparser                 2.19                     py36_0  
pyct                      0.4.6                      py_0    pyviz/label/dev
pyct-core                 0.4.6                      py_0    pyviz/label/dev
pyflakes                  2.1.1                    py36_0  
pygments                  2.3.1                    py36_0  
pyjanitor                 0.16.6                   pypi_0    pypi
pylint                    2.3.1                    py36_0  
pynacl                    1.3.0                    pypi_0    pypi
pyopenssl                 19.0.0                   py36_0  
pyparsing                 2.4.0                      py_0  
pyqt                      5.9.2            py36h751905a_0  
pyrsistent                0.14.11          py36h7b6447c_0  
pysocks                   1.6.8                    py36_0  
pytest                    4.4.0                    py36_1  
python                    3.6.6                h6e4f718_2  
python-dateutil           2.8.0                    py36_0  
pytz                      2019.1                     py_0  
pyviz_comms               0.7.2                      py_0    pyviz/label/dev
pywavelets                1.0.3            py36hdd07704_1  
pyyaml                    5.1              py36h7b6447c_0  
pyzmq                     18.0.0           py36he6710b0_0  
qt                        5.9.5                h7e424d6_0  
qtawesome                 0.5.7                    py36_1  
qtconsole                 4.4.3                    py36_0  
qtpy                      1.7.0                    py36_1  
ray                       0.6.5                    pypi_0    pypi
readline                  7.0                  h7b6447c_5  
redis                     3.2.1                    pypi_0    pypi
requests                  2.21.0                   py36_0  
rope                      0.12.0                   py36_0  
ruamel-yaml               0.15.92                  pypi_0    pypi
scikit-image              0.14.2           py36he6710b0_0  
scikit-learn              0.20.3           py36hd81dba3_0  
scikit-video              1.1.11                   pypi_0    pypi
scipy                     1.2.1            py36h7c811a0_0  
seaborn                   0.9.0                    py36_0  
secretstorage             3.1.1                    py36_0  
selenium                  3.141.0          py36h7b6447c_0  
send2trash                1.5.0                    py36_0  
setuptools                41.0.0                   py36_0  
shapely                   1.6.4.post2              pypi_0    pypi
sip                       4.19.8           py36hf484d3e_0  
six                       1.12.0                   py36_0  
slicerator                1.0.0                      py_0    conda-forge
snowballstemmer           1.2.1                    py36_0  
sortedcontainers          2.1.0                    py36_0  
sphinx                    2.0.1                      py_0  
sphinxcontrib-applehelp   1.0.1                      py_0  
sphinxcontrib-devhelp     1.0.1                      py_0  
sphinxcontrib-fulltoc     1.2.0                    pypi_0    pypi
sphinxcontrib-htmlhelp    1.0.2                      py_0  
sphinxcontrib-jsmath      1.0.1                      py_0  
sphinxcontrib-qthelp      1.0.2                      py_0  
sphinxcontrib-serializinghtml 1.1.1                      py_0  
spur                      0.3.20                   pypi_0    pypi
spyder                    3.3.4                    py36_0  
spyder-kernels            0.4.4                    py36_0  
sqlite                    3.27.2               h7b6447c_0  
statsmodels               0.9.0            py36h035aef0_0  
tblib                     1.3.2                    py36_0  
terminado                 0.8.2                    py36_0  
testpath                  0.3.1                    py36_0  
tifffile                  0.15.1          py36h3010b51_1001    conda-forge
tk                        8.6.8                hbc83047_0  
toolz                     0.9.0                    py36_0  
tornado                   6.0.2            py36h7b6447c_0  
tqdm                      4.31.1                   py36_1  
traitlets                 4.3.2                    py36_0  
typed-ast                 1.3.1            py36h7b6447c_0  
typing                    3.6.6                    pypi_0    pypi
urllib3                   1.24.1                   py36_0  
wcwidth                   0.1.7                    py36_0  
webencodings              0.5.1                    py36_1  
wheel                     0.33.1                   py36_0  
widgetsnbextension        3.4.2                    py36_0  
wrapt                     1.11.1           py36h7b6447c_0  
wurlitzer                 1.0.2                    py36_0  
xarray                    0.11.3                   py36_0  
xlrd                      1.2.0                    pypi_0    pypi
xxhash                    1.3.0                    pypi_0    pypi
xz                        5.2.4                h14c3975_4  
yaml                      0.1.7                had09818_2  
zeromq                    4.3.1                he6710b0_3  
zict                      0.1.4                    py36_0  
zlib                      1.2.11               h7b6447c_3  
zstd                      1.3.7                h0b5b093_0  
commit 0330aebeaceba873fb8ec2c8a16e9331ad885491 (HEAD -> master, origin/master, origin/HEAD)
Author: eftychios pnevmatikakis <eap2111@columbia.edu>
Date:   Thu Apr 18 09:51:17 2019 -0400

    Update ISSUE_TEMPLATE.md

CNMF-E params

print(f"""n_processes={num_processes},
method_init={method_init},  # use this for 1 photon
k={num_neurons_patch},  # neurons per patch
gSig={(g_sigma, g_sigma)},  # half size of neuron
gSiz={(g_size_cnmf, g_size_cnmf)},  # in general 3*gSig+1
merge_thresh={merge_thresh},  # threshold for merging
p={ar_p},  # order of autoregressive process to fit
dview={dview},  # if None it will run on a single thread
tsub={tsub},  # downsampling factor in time for initialization, increase if you have memory problems
ssub={ssub},  # downsampling factor in space for initialization, increase if you have memory problems
Ain={None},  # if you want to initialize with some preselcted components you can pass them here as boolean vectors
rf={(patch_half_size, patch_half_size)},  # half size of the patch
stride={(stride, stride)},  # overlap among patches (keep it at least large as 4 times the neuron size)
only_init_patch={True},  # just leave it as is
gnb={num_bknd_comp},  # number of background components
nb_patch={num_bknd_comp_patch},  # number of background components per patch
method_deconvolution={method_deconvolution},  # could use 'cvxpy' alternatively
low_rank_background={low_rank_background},  # leave as is
update_background_components={update_background_components},  # sometimes setting to False improve the results
min_corr={min_corr},  # min peak value from correlation image
min_pnr={min_pnr},  # min peak to noise ration from PNR image
normalize_init={False},  # just leave as is
center_psf={True},  # leave as is for 1 photon
ssub_B={ssub_b},
ring_size_factor={ring_size_factor},
del_duplicates={True}  # whether to remove duplicates from initialization
""")

yields:

n_processes=20,
method_init=corr_pnr,  # use this for 1 photon
k=None,  # neurons per patch
gSig=(3, 3),  # half size of neuron
gSiz=(7, 7),  # in general 3*gSig+1
merge_thresh=0.8,  # threshold for merging
p=1,  # order of autoregressive process to fit
dview=(<multiprocessing.pool.Pool object at 0x7fc6438fa198>,),  # if None it will run on a single thread
tsub=1,  # downsampling factor in time for initialization, increase if you have memory problems
ssub=1,  # downsampling factor in space for initialization, increase if you have memory problems
Ain=None,  # if you want to initialize with some preselcted components you can pass them here as boolean vectors
rf=(70, 70),  # half size of the patch
stride=(36, 36),  # overlap among patches (keep it at least large as 4 times the neuron size)
only_init_patch=True,  # just leave it as is
gnb=0,  # number of background components
nb_patch=0,  # number of background components per patch
method_deconvolution=oasis,  # could use 'cvxpy' alternatively
low_rank_background=None,  # leave as is
update_background_components=True,  # sometimes setting to False improve the results
min_corr=0.8,  # min peak value from correlation image
min_pnr=15,  # min peak to noise ration from PNR image
normalize_init=False,  # just leave as is
center_psf=True,  # leave as is for 1 photon
ssub_B=2,
ring_size_factor=1.4,
del_duplicates=True  # whether to remove duplicates from initialization

Error log

See: https://gist.github.com/zbarry/12e19f78775b915e9e05606d8b7bff28 for full log.

WARNING:root:gnb=0, hence setting keys nb_patch and low_rank_background in group patch automatically.
WARNING:root:using CNMF-E's ringmodel for background hence setting key normalize_init in group init automatically to False.
/scratch/miniconda3/envs/jupyter/lib/python3.7/site-packages/numpy/core/memmap.py:331: FutureWarning: Using a non-tuple sequence for multidimensional indexing is deprecated; use `arr[tuple(seq)]` instead of `arr[seq]`. In the future this will be interpreted as an array index, `arr[np.array(seq)]`, which will result either in an error or a different result.
  res = super(memmap, self).__getitem__(index)
WARNING:root:Changing key init_batch in group online from 200 to 500
INFO:root:(500, 540, 720)
WARNING:root:Changing key medw in group spatial from None to (3, 3)
WARNING:root:Changing key se in group spatial from None to [[1 1 1]
 [1 1 1]
 [1 1 1]]
WARNING:root:Changing key ss in group spatial from None to [[1 1 1]
 [1 1 1]
 [1 1 1]]
INFO:root:Using 20 processes
INFO:root:using 4000 pixels per process
INFO:root:using 5000 block_size_spat
INFO:root:using 5000 block_size_temp
WARNING:root:Changing key n_pixels_per_process in group preprocess from 4000 to 4900
WARNING:root:Changing key n_pixels_per_process in group spatial from 4000 to 4900
INFO:root:Patch size: (140, 140)
/scratch/miniconda3/envs/jupyter/lib/python3.7/site-packages/numpy/core/memmap.py:331: FutureWarning: Using a non-tuple sequence for multidimensional indexing is deprecated; use `arr[tuple(seq)]` instead of `arr[seq]`. In the future this will be interpreted as an array index, `arr[np.array(seq)]`, which will result either in an error or a different result.
  res = super(memmap, self).__getitem__(index)
/scratch/miniconda3/envs/jupyter/lib/python3.7/site-packages/numpy/core/memmap.py:331: FutureWarning: Using a non-tuple sequence for multidimensional indexing is deprecated; use `arr[tuple(seq)]` instead of `arr[seq]`. In the future this will be interpreted as an array index, `arr[np.array(seq)]`, which will result either in an error or a different result.
  res = super(memmap, self).__getitem__(index)
/scratch/miniconda3/envs/jupyter/lib/python3.7/site-packages/numpy/core/memmap.py:331: FutureWarning: Using a non-tuple sequence for multidimensional indexing is deprecated; use `arr[tuple(seq)]` instead of `arr[seq]`. In the future this will be interpreted as an array index, `arr[np.array(seq)]`, which will result either in an error or a different result.
  res = super(memmap, self).__getitem__(index)
/scratch/miniconda3/envs/jupyter/lib/python3.7/site-packages/numpy/core/memmap.py:331: FutureWarning: Using a non-tuple sequence for multidimensional indexing is deprecated; use `arr[tuple(seq)]` instead of `arr[seq]`. In the future this will be interpreted as an array index, `arr[np.array(seq)]`, which will result either in an error or a different result.
  res = super(memmap, self).__getitem__(index)
/scratch/miniconda3/envs/jupyter/lib/python3.7/site-packages/numpy/core/memmap.py:331: FutureWarning: Using a non-tuple sequence for multidimensional indexing is deprecated; use `arr[tuple(seq)]` instead of `arr[seq]`. In the future this will be interpreted as an array index, `arr[np.array(seq)]`, which will result either in an error or a different result.
  res = super(memmap, self).__getitem__(index)
/scratch/miniconda3/envs/jupyter/lib/python3.7/site-packages/numpy/core/memmap.py:331: FutureWarning: Using a non-tuple sequence for multidimensional indexing is deprecated; use `arr[tuple(seq)]` instead of `arr[seq]`. In the future this will be interpreted as an array index, `arr[np.array(seq)]`, which will result either in an error or a different result.
  res = super(memmap, self).__getitem__(index)
/scratch/miniconda3/envs/jupyter/lib/python3.7/site-packages/numpy/core/memmap.py:331: FutureWarning: Using a non-tuple sequence for multidimensional indexing is deprecated; use `arr[tuple(seq)]` instead of `arr[seq]`. In the future this will be interpreted as an array index, `arr[np.array(seq)]`, which will result either in an error or a different result.
  res = super(memmap, self).__getitem__(index)
/scratch/miniconda3/envs/jupyter/lib/python3.7/site-packages/numpy/core/memmap.py:331: FutureWarning: Using a non-tuple sequence for multidimensional indexing is deprecated; use `arr[tuple(seq)]` instead of `arr[seq]`. In the future this will be interpreted as an array index, `arr[np.array(seq)]`, which will result either in an error or a different result.
  res = super(memmap, self).__getitem__(index)
/scratch/miniconda3/envs/jupyter/lib/python3.7/site-packages/numpy/core/memmap.py:331: FutureWarning: Using a non-tuple sequence for multidimensional indexing is deprecated; use `arr[tuple(seq)]` instead of `arr[seq]`. In the future this will be interpreted as an array index, `arr[np.array(seq)]`, which will result either in an error or a different result.
  res = super(memmap, self).__getitem__(index)
/scratch/miniconda3/envs/jupyter/lib/python3.7/site-packages/numpy/core/memmap.py:331: FutureWarning: Using a non-tuple sequence for multidimensional indexing is deprecated; use `arr[tuple(seq)]` instead of `arr[seq]`. In the future this will be interpreted as an array index, `arr[np.array(seq)]`, which will result either in an error or a different result.
  res = super(memmap, self).__getitem__(index)
/scratch/miniconda3/envs/jupyter/lib/python3.7/site-packages/numpy/core/memmap.py:331: FutureWarning: Using a non-tuple sequence for multidimensional indexing is deprecated; use `arr[tuple(seq)]` instead of `arr[seq]`. In the future this will be interpreted as an array index, `arr[np.array(seq)]`, which will result either in an error or a different result.
  res = super(memmap, self).__getitem__(index)
/scratch/miniconda3/envs/jupyter/lib/python3.7/site-packages/numpy/core/memmap.py:331: FutureWarning: Using a non-tuple sequence for multidimensional indexing is deprecated; use `arr[tuple(seq)]` instead of `arr[seq]`. In the future this will be interpreted as an array index, `arr[np.array(seq)]`, which will result either in an error or a different result.
  res = super(memmap, self).__getitem__(index)
/scratch/miniconda3/envs/jupyter/lib/python3.7/site-packages/numpy/core/memmap.py:331: FutureWarning: Using a non-tuple sequence for multidimensional indexing is deprecated; use `arr[tuple(seq)]` instead of `arr[seq]`. In the future this will be interpreted as an array index, `arr[np.array(seq)]`, which will result either in an error or a different result.
  res = super(memmap, self).__getitem__(index)
/scratch/miniconda3/envs/jupyter/lib/python3.7/site-packages/numpy/core/memmap.py:331: FutureWarning: Using a non-tuple sequence for multidimensional indexing is deprecated; use `arr[tuple(seq)]` instead of `arr[seq]`. In the future this will be interpreted as an array index, `arr[np.array(seq)]`, which will result either in an error or a different result.
  res = super(memmap, self).__getitem__(index)
/scratch/miniconda3/envs/jupyter/lib/python3.7/site-packages/numpy/core/memmap.py:331: FutureWarning: Using a non-tuple sequence for multidimensional indexing is deprecated; use `arr[tuple(seq)]` instead of `arr[seq]`. In the future this will be interpreted as an array index, `arr[np.array(seq)]`, which will result either in an error or a different result.
  res = super(memmap, self).__getitem__(index)
/scratch/miniconda3/envs/jupyter/lib/python3.7/site-packages/numpy/core/memmap.py:331: FutureWarning: Using a non-tuple sequence for multidimensional indexing is deprecated; use `arr[tuple(seq)]` instead of `arr[seq]`. In the future this will be interpreted as an array index, `arr[np.array(seq)]`, which will result either in an error or a different result.
  res = super(memmap, self).__getitem__(index)
/scratch/miniconda3/envs/jupyter/lib/python3.7/site-packages/numpy/core/memmap.py:331: FutureWarning: Using a non-tuple sequence for multidimensional indexing is deprecated; use `arr[tuple(seq)]` instead of `arr[seq]`. In the future this will be interpreted as an array index, `arr[np.array(seq)]`, which will result either in an error or a different result.
  res = super(memmap, self).__getitem__(index)
/scratch/miniconda3/envs/jupyter/lib/python3.7/site-packages/numpy/core/memmap.py:331: FutureWarning: Using a non-tuple sequence for multidimensional indexing is deprecated; use `arr[tuple(seq)]` instead of `arr[seq]`. In the future this will be interpreted as an array index, `arr[np.array(seq)]`, which will result either in an error or a different result.
  res = super(memmap, self).__getitem__(index)
/scratch/miniconda3/envs/jupyter/lib/python3.7/site-packages/numpy/core/memmap.py:331: FutureWarning: Using a non-tuple sequence for multidimensional indexing is deprecated; use `arr[tuple(seq)]` instead of `arr[seq]`. In the future this will be interpreted as an array index, `arr[np.array(seq)]`, which will result either in an error or a different result.
  res = super(memmap, self).__getitem__(index)
/scratch/miniconda3/envs/jupyter/lib/python3.7/site-packages/numpy/core/memmap.py:331: FutureWarning: Using a non-tuple sequence for multidimensional indexing is deprecated; use `arr[tuple(seq)]` instead of `arr[seq]`. In the future this will be interpreted as an array index, `arr[np.array(seq)]`, which will result either in an error or a different result.
  res = super(memmap, self).__getitem__(index)
WARNING:root:Changing key n_processes in group patch from 20 to 1
WARNING:root:Changing key rf in group patch from (70, 70) to None
WARNING:root:Changing key stride in group patch from (36, 36) to None
WARNING:root:Changing key n_processes in group patch from 20 to 1
WARNING:root:Changing key rf in group patch from (70, 70) to None
/data/external_repos/CaImAn/caiman/source_extraction/cnmf/cnmf.py:389: FutureWarning: Using a non-tuple sequence for multidimensional indexing is deprecated; use `arr[tuple(seq)]` instead of `arr[seq]`. In the future this will be interpreted as an array index, `arr[np.array(seq)]`, which will result either in an error or a different result.
  dims_sliced = images[indeces].shape[1:]
/data/external_repos/CaImAn/caiman/source_extraction/cnmf/cnmf.py:392: FutureWarning: Using a non-tuple sequence for multidimensional indexing is deprecated; use `arr[tuple(seq)]` instead of `arr[seq]`. In the future this will be interpreted as an array index, `arr[np.array(seq)]`, which will result either in an error or a different result.
  images = images[indeces]
WARNING:root:Parallel processing in a single patch is not available for loaded in memory or sliced data.
WARNING:root:Changing key init_batch in group online from 500 to 5000
INFO:root:(5000, 141, 141)
WARNING:root:Changing key stride in group patch from (36, 36) to None
INFO:root:Using 1 processes
INFO:root:using 4900 pixels per process
/data/external_repos/CaImAn/caiman/source_extraction/cnmf/cnmf.py:389: FutureWarning: Using a non-tuple sequence for multidimensional indexing is deprecated; use `arr[tuple(seq)]` instead of `arr[seq]`. In the future this will be interpreted as an array index, `arr[np.array(seq)]`, which will result either in an error or a different result.
  dims_sliced = images[indeces].shape[1:]
INFO:root:using 5000 block_size_spat
INFO:root:using 5000 block_size_temp
/data/external_repos/CaImAn/caiman/source_extraction/cnmf/cnmf.py:392: FutureWarning: Using a non-tuple sequence for multidimensional indexing is deprecated; use `arr[tuple(seq)]` instead of `arr[seq]`. In the future this will be interpreted as an array index, `arr[np.array(seq)]`, which will result either in an error or a different result.
  images = images[indeces]
INFO:root:preprocessing ...
WARNING:root:Parallel processing in a single patch is not available for loaded in memory or sliced data.
INFO:root:Checking for missing data entries (NaN)
WARNING:root:Changing key init_batch in group online from 500 to 5000
INFO:root:(5000, 141, 141)
INFO:root:Using 1 processes
INFO:root:using 4900 pixels per process
INFO:root:using 5000 block_size_spat
INFO:root:using 5000 block_size_temp
INFO:root:preprocessing ...
INFO:root:Checking for missing data entries (NaN)
WARNING:root:Changing key n_processes in group patch from 20 to 1
WARNING:root:Changing key rf in group patch from (70, 70) to None
WARNING:root:Changing key stride in group patch from (36, 36) to None
/data/external_repos/CaImAn/caiman/source_extraction/cnmf/cnmf.py:389: FutureWarning: Using a non-tuple sequence for multidimensional indexing is deprecated; use `arr[tuple(seq)]` instead of `arr[seq]`. In the future this will be interpreted as an array index, `arr[np.array(seq)]`, which will result either in an error or a different result.
  dims_sliced = images[indeces].shape[1:]
/data/external_repos/CaImAn/caiman/source_extraction/cnmf/cnmf.py:392: FutureWarning: Using a non-tuple sequence for multidimensional indexing is deprecated; use `arr[tuple(seq)]` instead of `arr[seq]`. In the future this will be interpreted as an array index, `arr[np.array(seq)]`, which will result either in an error or a different result.
  images = images[indeces]
WARNING:root:Parallel processing in a single patch is not available for loaded in memory or sliced data.
WARNING:root:Changing key init_batch in group online from 500 to 5000
INFO:root:(5000, 141, 141)
INFO:root:Using 1 processes
INFO:root:using 4900 pixels per process
INFO:root:using 5000 block_size_spat
INFO:root:using 5000 block_size_temp
INFO:root:preprocessing ...
INFO:root:Checking for missing data entries (NaN)
WARNING:root:Changing key n_processes in group patch from 20 to 1
WARNING:root:Changing key rf in group patch from (70, 70) to None
WARNING:root:Changing key stride in group patch from (36, 36) to None
/data/external_repos/CaImAn/caiman/source_extraction/cnmf/cnmf.py:389: FutureWarning: Using a non-tuple sequence for multidimensional indexing is deprecated; use `arr[tuple(seq)]` instead of `arr[seq]`. In the future this will be interpreted as an array index, `arr[np.array(seq)]`, which will result either in an error or a different result.
  dims_sliced = images[indeces].shape[1:]
/data/external_repos/CaImAn/caiman/source_extraction/cnmf/cnmf.py:392: FutureWarning: Using a non-tuple sequence for multidimensional indexing is deprecated; use `arr[tuple(seq)]` instead of `arr[seq]`. In the future this will be interpreted as an array index, `arr[np.array(seq)]`, which will result either in an error or a different result.
  images = images[indeces]
WARNING:root:Parallel processing in a single patch is not available for loaded in memory or sliced data.
WARNING:root:Changing key init_batch in group online from 500 to 5000
INFO:root:(5000, 141, 141)
INFO:root:Using 1 processes
INFO:root:using 4900 pixels per process
INFO:root:using 5000 block_size_spat
INFO:root:using 5000 block_size_temp
INFO:root:preprocessing ...
INFO:root:Checking for missing data entries (NaN)
WARNING:root:Changing key n_processes in group patch from 20 to 1
WARNING:root:Changing key rf in group patch from (70, 70) to None
WARNING:root:Changing key stride in group patch from (36, 36) to None
/data/external_repos/CaImAn/caiman/source_extraction/cnmf/cnmf.py:389: FutureWarning: Using a non-tuple sequence for multidimensional indexing is deprecated; use `arr[tuple(seq)]` instead of `arr[seq]`. In the future this will be interpreted as an array index, `arr[np.array(seq)]`, which will result either in an error or a different result.
  dims_sliced = images[indeces].shape[1:]
/data/external_repos/CaImAn/caiman/source_extraction/cnmf/cnmf.py:392: FutureWarning: Using a non-tuple sequence for multidimensional indexing is deprecated; use `arr[tuple(seq)]` instead of `arr[seq]`. In the future this will be interpreted as an array index, `arr[np.array(seq)]`, which will result either in an error or a different result.
  images = images[indeces]
WARNING:root:Parallel processing in a single patch is not available for loaded in memory or sliced data.
WARNING:root:Changing key init_batch in group online from 500 to 5000
INFO:root:(5000, 141, 141)
INFO:root:Using 1 processes
INFO:root:using 4900 pixels per process
INFO:root:using 5000 block_size_spat
INFO:root:using 5000 block_size_temp
INFO:root:preprocessing ...
INFO:root:Checking for missing data entries (NaN)
WARNING:root:Changing key n_processes in group patch from 20 to 1
WARNING:root:Changing key rf in group patch from (70, 70) to None
WARNING:root:Changing key stride in group patch from (36, 36) to None
/data/external_repos/CaImAn/caiman/source_extraction/cnmf/cnmf.py:389: FutureWarning: Using a non-tuple sequence for multidimensional indexing is deprecated; use `arr[tuple(seq)]` instead of `arr[seq]`. In the future this will be interpreted as an array index, `arr[np.array(seq)]`, which will result either in an error or a different result.
  dims_sliced = images[indeces].shape[1:]
/data/external_repos/CaImAn/caiman/source_extraction/cnmf/cnmf.py:392: FutureWarning: Using a non-tuple sequence for multidimensional indexing is deprecated; use `arr[tuple(seq)]` instead of `arr[seq]`. In the future this will be interpreted as an array index, `arr[np.array(seq)]`, which will result either in an error or a different result.
  images = images[indeces]
WARNING:root:Parallel processing in a single patch is not available for loaded in memory or sliced data.
WARNING:root:Changing key init_batch in group online from 500 to 5000
INFO:root:(5000, 141, 140)
INFO:root:Using 1 processes
INFO:root:using 4900 pixels per process
INFO:root:using 5000 block_size_spat
INFO:root:using 5000 block_size_temp
INFO:root:preprocessing ...
INFO:root:Checking for missing data entries (NaN)
WARNING:root:Changing key n_processes in group patch from 20 to 1
WARNING:root:Changing key rf in group patch from (70, 70) to None
WARNING:root:Changing key stride in group patch from (36, 36) to None
/data/external_repos/CaImAn/caiman/source_extraction/cnmf/cnmf.py:389: FutureWarning: Using a non-tuple sequence for multidimensional indexing is deprecated; use `arr[tuple(seq)]` instead of `arr[seq]`. In the future this will be interpreted as an array index, `arr[np.array(seq)]`, which will result either in an error or a different result.
  dims_sliced = images[indeces].shape[1:]
/data/external_repos/CaImAn/caiman/source_extraction/cnmf/cnmf.py:392: FutureWarning: Using a non-tuple sequence for multidimensional indexing is deprecated; use `arr[tuple(seq)]` instead of `arr[seq]`. In the future this will be interpreted as an array index, `arr[np.array(seq)]`, which will result either in an error or a different result.
  images = images[indeces]
WARNING:root:Parallel processing in a single patch is not available for loaded in memory or sliced data.
WARNING:root:Changing key init_batch in group online from 500 to 5000
INFO:root:(5000, 141, 141)
INFO:root:Using 1 processes
INFO:root:using 4900 pixels per process
INFO:root:using 5000 block_size_spat
INFO:root:using 5000 block_size_temp
INFO:root:preprocessing ...
INFO:root:Checking for missing data entries (NaN)
WARNING:root:Changing key n_processes in group patch from 20 to 1
WARNING:root:Changing key rf in group patch from (70, 70) to None
WARNING:root:Changing key stride in group patch from (36, 36) to None
/data/external_repos/CaImAn/caiman/source_extraction/cnmf/cnmf.py:389: FutureWarning: Using a non-tuple sequence for multidimensional indexing is deprecated; use `arr[tuple(seq)]` instead of `arr[seq]`. In the future this will be interpreted as an array index, `arr[np.array(seq)]`, which will result either in an error or a different result.
  dims_sliced = images[indeces].shape[1:]
/data/external_repos/CaImAn/caiman/source_extraction/cnmf/cnmf.py:392: FutureWarning: Using a non-tuple sequence for multidimensional indexing is deprecated; use `arr[tuple(seq)]` instead of `arr[seq]`. In the future this will be interpreted as an array index, `arr[np.array(seq)]`, which will result either in an error or a different result.
  images = images[indeces]
WARNING:root:Parallel processing in a single patch is not available for loaded in memory or sliced data.
WARNING:root:Changing key init_batch in group online from 500 to 5000
INFO:root:(5000, 141, 141)
INFO:root:Using 1 processes
INFO:root:using 4900 pixels per process
INFO:root:using 5000 block_size_spat
INFO:root:using 5000 block_size_temp
INFO:root:preprocessing ...
INFO:root:Checking for missing data entries (NaN)
WARNING:root:Changing key n_processes in group patch from 20 to 1
WARNING:root:Changing key rf in group patch from (70, 70) to None
WARNING:root:Changing key stride in group patch from (36, 36) to None
/data/external_repos/CaImAn/caiman/source_extraction/cnmf/cnmf.py:389: FutureWarning: Using a non-tuple sequence for multidimensional indexing is deprecated; use `arr[tuple(seq)]` instead of `arr[seq]`. In the future this will be interpreted as an array index, `arr[np.array(seq)]`, which will result either in an error or a different result.
  dims_sliced = images[indeces].shape[1:]
/data/external_repos/CaImAn/caiman/source_extraction/cnmf/cnmf.py:392: FutureWarning: Using a non-tuple sequence for multidimensional indexing is deprecated; use `arr[tuple(seq)]` instead of `arr[seq]`. In the future this will be interpreted as an array index, `arr[np.array(seq)]`, which will result either in an error or a different result.
  images = images[indeces]
WARNING:root:Parallel processing in a single patch is not available for loaded in memory or sliced data.
WARNING:root:Changing key init_batch in group online from 500 to 5000
INFO:root:(5000, 141, 140)
INFO:root:Using 1 processes
INFO:root:using 4900 pixels per process
INFO:root:using 5000 block_size_spat
INFO:root:using 5000 block_size_temp
INFO:root:preprocessing ...
INFO:root:Checking for missing data entries (NaN)

~~ snip ~~

INFO:root:In total, 0 neurons were initialized.
INFO:root:Computing ring model background
INFO:root:Computing ring model background
INFO:root:0 neurons have been initialized
INFO:root:0 neurons have been initialized
INFO:root:In total, 0 neurons were initialized.
INFO:root:0 neurons have been initialized
INFO:root:In total, 1 neurons were initialized.
INFO:root:0 neurons have been initialized
INFO:root:In total, 0 neurons were initialized.
INFO:root:Computing ring model background
INFO:root:Computing ring model background
INFO:root:0 neurons have been initialized
INFO:root:In total, 1 neurons were initialized.
INFO:root:In total, 0 neurons were initialized.
INFO:root:Computing ring model background
INFO:root:In total, 0 neurons were initialized.
INFO:root:0 neurons have been initialized
INFO:root:Computing ring model background
INFO:root:In total, 13 neurons were initialized.
INFO:root:Computing ring model background
INFO:root:Computing ring model background
INFO:root:In total, 4 neurons were initialized.
INFO:root:Computing ring model background
INFO:root:In total, 0 neurons were initialized.
INFO:root:0 neurons have been initialized
INFO:root:Computing ring model background
INFO:root:Computing ring model background
INFO:root:In total, 8 neurons were initialized.
INFO:root:Computing ring model background
INFO:root:Updating spatial components
INFO:root:Computing support of spatial components
INFO:root:Memory mapping
INFO:root:Updating Spatial Components using lasso lars
INFO:root:Updating spatial components
INFO:root:Computing support of spatial components
/data/external_repos/CaImAn/caiman/source_extraction/cnmf/spatial.py:903: SparseEfficiencyWarning: Changing the sparsity structure of a csc_matrix is expensive. lil_matrix is more efficient.
  dist_indicator[:, i] = scipy.sparse.coo_matrix(np.squeeze(np.reshape(A_temp, (d, 1)))[:,None] > 0)
INFO:root:Memory mapping
INFO:root:Updating Spatial Components using lasso lars
INFO:root:100 neurons have been initialized
INFO:root:In total, 102 neurons were initialized.
INFO:root:In total, 64 neurons were initialized.
INFO:root:Computing ring model background
INFO:root:Computing ring model background
INFO:root:In total, 81 neurons were initialized.
INFO:root:Updating spatial components
INFO:root:Computing support of spatial components
INFO:root:Memory mapping
INFO:root:Updating Spatial Components using lasso lars
INFO:root:thresholding components
INFO:root:Computing residuals
INFO:root:Updating done in 5s
INFO:root:Removing created tempfiles
INFO:root:Updating temporal components
INFO:root:Generating residuals
INFO:root:Computing ring model background
INFO:root:In total, 69 neurons were initialized.
INFO:root:Computing ring model background
INFO:root:Updating spatial components
INFO:root:Computing support of spatial components
INFO:root:Memory mapping
INFO:root:Updating Spatial Components using lasso lars
INFO:root:thresholding components
INFO:root:Computing residuals
INFO:root:Updating done in 7s
INFO:root:Removing created tempfiles
INFO:root:Updating temporal components
INFO:root:Generating residuals
INFO:root:thresholding components
INFO:root:Computing residuals
INFO:root:Updating done in 4s
INFO:root:Removing created tempfiles
INFO:root:Updating temporal components
INFO:root:Generating residuals
/scratch/miniconda3/envs/jupyter/lib/python3.7/site-packages/scipy/sparse/dia.py:300: RuntimeWarning: divide by zero encountered in remainder
  c = np.arange(num_rows, dtype=np.intc) - (offsets % max_dim)[:, None]
INFO:root:entering the deconvolution 
INFO:root:stopping: overall temporal component not changing significantly
INFO:root:Searching for more neurons in the residual
INFO:root:Updating spatial components
INFO:root:Computing support of spatial components
/data/external_repos/CaImAn/caiman/source_extraction/cnmf/spatial.py:903: SparseEfficiencyWarning: Changing the sparsity structure of a csc_matrix is expensive. lil_matrix is more efficient.
  dist_indicator[:, i] = scipy.sparse.coo_matrix(np.squeeze(np.reshape(A_temp, (d, 1)))[:,None] > 0)
INFO:root:Memory mapping
INFO:root:Updating Spatial Components using lasso lars
INFO:root:Updating spatial components
INFO:root:Computing support of spatial components
INFO:root:Memory mapping
INFO:root:Updating Spatial Components using lasso lars
INFO:root:100 neurons have been initialized
INFO:root:Updating spatial components
INFO:root:Computing support of spatial components
/data/external_repos/CaImAn/caiman/source_extraction/cnmf/spatial.py:903: SparseEfficiencyWarning: Changing the sparsity structure of a csc_matrix is expensive. lil_matrix is more efficient.
  dist_indicator[:, i] = scipy.sparse.coo_matrix(np.squeeze(np.reshape(A_temp, (d, 1)))[:,None] > 0)
INFO:root:Memory mapping
INFO:root:Updating Spatial Components using lasso lars
INFO:root:Updating spatial components
INFO:root:Computing support of spatial components
INFO:root:Updating spatial components
INFO:root:Computing support of spatial components
INFO:root:Memory mapping
INFO:root:Updating Spatial Components using lasso lars
INFO:root:Memory mapping
INFO:root:Updating Spatial Components using lasso lars
INFO:root:entering the deconvolution 
/scratch/miniconda3/envs/jupyter/lib/python3.7/site-packages/scipy/sparse/dia.py:300: RuntimeWarning: divide by zero encountered in remainder
  c = np.arange(num_rows, dtype=np.intc) - (offsets % max_dim)[:, None]
INFO:root:entering the deconvolution 
INFO:root:stopping: overall temporal component not changing significantly
INFO:root:Searching for more neurons in the residual
INFO:root:9 out of total 20 temporal components updated
INFO:root:thresholding components
INFO:root:Computing residuals
INFO:root:Updating done in 5s
INFO:root:Removing created tempfiles
INFO:root:Updating temporal components
INFO:root:Generating residuals
INFO:root:Updating spatial components
INFO:root:Computing support of spatial components
INFO:root:16 out of total 20 temporal components updated
/data/external_repos/CaImAn/caiman/source_extraction/cnmf/spatial.py:903: SparseEfficiencyWarning: Changing the sparsity structure of a csc_matrix is expensive. lil_matrix is more efficient.
  dist_indicator[:, i] = scipy.sparse.coo_matrix(np.squeeze(np.reshape(A_temp, (d, 1)))[:,None] > 0)
INFO:root:18 out of total 20 temporal components updated
INFO:root:20 out of total 20 temporal components updated
INFO:root:Memory mapping

~~ snip ~~

INFO:root:54 out of total 56 temporal components updated
INFO:root:56 out of total 56 temporal components updated
INFO:root:stopping: overall temporal component not changing significantly
INFO:root:Returning background as b0 and W
/scratch/miniconda3/envs/jupyter/lib/python3.7/site-packages/scipy/sparse/dia.py:300: RuntimeWarning: divide by zero encountered in remainder
  c = np.arange(num_rows, dtype=np.intc) - (offsets % max_dim)[:, None]
INFO:root:thresholding components
INFO:root:Computing residuals
INFO:root:Updating done in 34s
INFO:root:Removing created tempfiles
INFO:root:Updating temporal components
INFO:root:Generating residuals
INFO:root:entering the deconvolution 
INFO:root:25 out of total 180 temporal components updated
INFO:root:48 out of total 180 temporal components updated
INFO:root:71 out of total 180 temporal components updated
INFO:root:91 out of total 180 temporal components updated
INFO:root:114 out of total 180 temporal components updated
INFO:root:129 out of total 180 temporal components updated
INFO:root:144 out of total 180 temporal components updated
INFO:root:155 out of total 180 temporal components updated
INFO:root:165 out of total 180 temporal components updated
INFO:root:174 out of total 180 temporal components updated
INFO:root:178 out of total 180 temporal components updated
INFO:root:180 out of total 180 temporal components updated
INFO:root:25 out of total 180 temporal components updated
INFO:root:48 out of total 180 temporal components updated
INFO:root:71 out of total 180 temporal components updated
INFO:root:91 out of total 180 temporal components updated
INFO:root:114 out of total 180 temporal components updated
INFO:root:129 out of total 180 temporal components updated
INFO:root:144 out of total 180 temporal components updated
INFO:root:155 out of total 180 temporal components updated
INFO:root:165 out of total 180 temporal components updated
INFO:root:174 out of total 180 temporal components updated
INFO:root:178 out of total 180 temporal components updated
INFO:root:180 out of total 180 temporal components updated
INFO:root:stopping: overall temporal component not changing significantly
INFO:root:Returning background as b0 and W
/scratch/miniconda3/envs/jupyter/lib/python3.7/site-packages/scipy/sparse/dia.py:300: RuntimeWarning: divide by zero encountered in remainder
  c = np.arange(num_rows, dtype=np.intc) - (offsets % max_dim)[:, None]
INFO:root:Elapsed time for processing patches:                  313s
INFO:root:Embedding patches results into whole FOV
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-11-674e24576074> in <module>
     52     )
     53 
---> 54     cnm.fit(y)
     55 
     56     cnm_objs[g_sigma, g_size_ratio] = cnm

/data/external_repos/CaImAn/caiman/source_extraction/cnmf/cnmf.py in fit(self, images, indeces)
    541                     low_rank_background=self.params.get('patch', 'low_rank_background'),
    542                     del_duplicates=self.params.get('patch', 'del_duplicates'),
--> 543                     indeces=indeces)
    544 
    545             self.estimates.bl, self.estimates.c1, self.estimates.g, self.estimates.neurons_sn = None, None, None, None

/data/external_repos/CaImAn/caiman/source_extraction/cnmf/map_reduce.py in run_CNMF_patches(file_name, shape, params, gnb, dview, memory_fact, border_pix, low_rank_background, del_duplicates, indeces)
    348             count_bgr += b.shape[-1]
    349             if nb_patch >= 0:
--> 350                 F_tot[patch_id * nb_patch:(patch_id + 1) * nb_patch] = f
    351             else:  # full background per patch
    352                 F_tot = np.concatenate([F_tot, f])

ValueError: could not broadcast input array from shape (0,5000) into shape (0,500)
zbarry commented 5 years ago

FWIW, this is also happening on the dev branch. y.shape -> (500, 540, 720).

epnev commented 5 years ago

@zbarry Possibly unrelated but your environment says you're using python 3.6 yet the log mentions python 3.7

zbarry commented 5 years ago

@epnev that's actually because I realized I was working in 3.7 in the notebook when I started making the issue, then tried a 3.6 kernel and got the same error.

I see these lines pop up multiple times:

WARNING:root:Changing key init_batch in group online from 500 to 5000
INFO:root:(5000, 141, 141)

Not sure what that refers to yet, but those are also the two sizes in the broadcast mismatch.

epnev commented 5 years ago

That's strange indeed. Can you send the script you're trying to run? I'll try to take a look later tonight or tomorrow.

zbarry commented 5 years ago

Errrrr.... you know what? This is where the 5000 came from:

filename = 'blarg_d1_540_d2_720_d3_1_order_C_frames_5000_.mmap'

yr, dims, t = cm.load_memmap(filename)
y = yr.T.reshape((t,) + dims, order='F')

# y = y[1000:3000]  # perturbation at approximately frame 2000
y = y[2000:2500]
#y = y[2000:2200]

y_vis = y.transpose((1, 2, 0))

I was passing the time-cropped y without even thinking about CaImAn pulling from y.filename. Whoops. Don't usually run CaImAn from notebooks like this. Thanks, haha.