Can't set up caimanmanager

[slceto]

For better support, please use the template below to submit your issue. When your issue gets resolved please remember to close it.

Sometimes errors while running CNMF occur during parallel processing which prevents the log to provide a meaningful error message. Please reproduce your error with setting dview=None.

If you need to upgrade CaImAn follow the instructions given in the documentation.

• Tell us a bit about your setup:

  1. Operating system (Linux/macOS/Windows): Windows 10
  2. Python version (3.x): 3.7
  3. Working environment (Python IDE/Jupyter Notebook/other): Anaconda Prompt (Anaconda3)
  4. Which of the demo scripts you're using for your analysis (if applicable): N/A
  5. CaImAn version*: Package-based, install date 11/11/2019
  6. CaImAn installation process (pip install ./pip install -e ./conda): pip install *You can get the CaImAn version by creating a params object and then typing params.data['caiman_version']. If the field doesn't exist, type N/A and consider upgrading)

• Describe the issue that you are experiencing The caimanmanager.py command does not complete

• Copy error log below (caiman) C:\Users\slcet>python C:\Users\slcet\Anaconda3\envs\caiman\bin\caimanmanager.py install Traceback (most recent call last): File "C:\Users\slcet\Anaconda3\envs\caiman\bin\caimanmanager.py", line 16, in from caiman.paths import caiman_datadir File "C:\Users\slcet\Anaconda3\envs\caiman\lib\site-packages\caiman__init__.py", line 3, in from .base.movies import movie, load, load_movie_chain File "C:\Users\slcet\Anaconda3\envs\caiman\lib\site-packages\caiman\base\movies.py", line 19, in import cv2 ImportError: DLL load failed: The specified module could not be found.

• If you're not reporting an error, type your message below

[pgunn]

Hi, From inside your caiman environment, please issue the following command and let me know what you get back:

conda list | findstr opencv

[slceto]

libopencv 4.1.2 h7e61296_0 conda-forge opencv 4.1.2 0 conda-forge py-opencv 4.1.2 py37h5ca1d4c_0 conda-forge

[slceto]

Also, if I just do

(caiman) C:\Users\slcet>caimanmanager.py install

I get

Traceback (most recent call last): File "C:\Users\slcet\Anaconda3\envs\caiman\bin\caimanmanager.py", line 16, in from caiman.paths import caiman_datadir ModuleNotFoundError: No module named 'caiman'

[pgunn]

From within your environment, please do this:

import sys
print(sys.path)

And give me the results. If you can also include what you see when you launch python that'd be helpful.

[slceto]

(caiman) C:\Users\slcet>import sys 'import' is not recognized as an internal or external command, operable program or batch file.

When launching python: Python 3.7.4 (default, Aug 9 2019, 18:34:13) [MSC v.1915 64 bit (AMD64)] :: Anaconda, Inc. on win32

Warning: This Python interpreter is in a conda environment, but the environment has not been activated. Libraries may fail to load. To activate this environment please see https://conda.io/activation

Type "help", "copyright", "credits" or "license" for more information.

[pgunn]

Sorry, I meant for you to do those two commands inside of python.

[slceto]

Ah, okay. Here is print(sys.path):

['', 'C:\Users\slcet\Anaconda3\python37.zip', 'C:\Users\slcet\Anaconda3\DLLs', 'C:\Users\slcet\Anaconda3\lib', 'C:\Users\slcet\Anaconda3', 'C:\Users\slcet\Anaconda3\lib\site-packages', 'C:\Users\slcet\Anaconda3\lib\site-packages\win32', 'C:\Users\slcet\Anaconda3\lib\site-packages\win32\lib', 'C:\Users\slcet\Anaconda3\lib\site-packages\Pythonwin']

[pgunn]

Ah, I think I know what's probably going on. Please send me the output of conda env list

[slceto]

conda environments:

# base * C:\Users\slcet\Anaconda3 caiman C:\Users\slcet\Anaconda3\envs\caiman

[pgunn]

Ok. So here's what happened, as far as I can tell: 1) You made the caiman environment, but you never switched into it to install caiman 2) You installed caiman into your base environment, but that didn't fully work because it was missing a lot of prerequisites that the environment was supposed to set up.

To fix this, first (if any of these fail, please stop and let me know): A) Do a conda remove caiman to remove caiman from your base environment B) Do a conda activate caiman to enter your caiman environment (note: In the future please do this EVERY time you launch a new shell before doing anything with caiman) C) Enter the caiman directory and redo the pip install D) Rerun the caimanmanager install. I believe it should work this time

Let me know how that goes.

[slceto]

What exactly do you mean by "enter the caiman directory"?

[pgunn]

Your checkout of the caiman sources, where you originally installed caiman with pip.

[slceto]

I've just followed the basic instructions for the Package-based process, so 1 )conda create -n caiman # caiman here refers to the name of the environment (you can pick any name you want) 2) conda activate caiman 3) conda install caiman -c conda-forge 4) caimanmanager.py install

Am I missing a step between (2) and (3)?

[pgunn]

Ah, when you filled in the template above you said you did a pip-based install. No worries.

In either case, if you do a conda activate caiman, and then from that shell, do a conda env list, you should see the star is next to caiman rather than base, and caimanmanager.py install should work. I think.

[slceto]

Yeah, sorry I had tried to do the developer install before with pip (gave up due to not being able to find Visual Studio 2017) and got mixed up.

But I'm still getting this: (caiman) C:\Users\slcet>conda env list

conda environments:

# base C:\Users\slcet\Anaconda3 caiman * C:\Users\slcet\Anaconda3\envs\caiman

(caiman) C:\Users\slcet>caimanmanager.py install Traceback (most recent call last): File "C:\Users\slcet\Anaconda3\envs\caiman\bin\caimanmanager.py", line 16, in from caiman.paths import caiman_datadir ModuleNotFoundError: No module named 'caiman'

[pgunn]

If you do a conda list | findstr caiman from inside the caiman environment, what do you get?

[slceto]

packages in environment at C:\Users\slcet\Anaconda3\envs\caiman:

caiman 1.6.3 py37he350917_0 conda-forge

[pgunn]

Ok, and also while still in the caiman environment, could you run python and redo that

import sys
print(sys.path)

thing?

[slceto]

['', 'C:\Users\slcet\Anaconda3\python37.zip', 'C:\Users\slcet\Anaconda3\DLLs', 'C:\Users\slcet\Anaconda3\lib', 'C:\Users\slcet\Anaconda3', 'C:\Users\slcet\Anaconda3\lib\site-packages', 'C:\Users\slcet\Anaconda3\lib\site-packages\win32', 'C:\Users\slcet\Anaconda3\lib\site-packages\win32\lib', 'C:\Users\slcet\Anaconda3\lib\site-packages\Pythonwin']

[pgunn]

That's puzzling. Can you do this (also from the caiman environemnt):

echo %PATH%

[slceto]

C:\Users\slcet\Anaconda3\envs\caiman;C:\Users\slcet\Anaconda3\envs\caiman\Library\mingw-w64\bin;C:\Users\slcet\Anaconda3\envs\caiman\Library\usr\bin;C:\Users\slcet\Anaconda3\envs\caiman\Library\bin;C:\Users\slcet\Anaconda3\envs\caiman\Scripts;C:\Users\slcet\Anaconda3\envs\caiman\bin;C:\Users\slcet\Anaconda3\condabin;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0;C:\Windows\System32\OpenSSH;C:\Program Files\MATLAB\R2017b\bin;C:\Program Files\MATLAB\R2017a\runtime\win64;C:\Program Files\MATLAB\R2017a\bin;C:\Program Files (x86)\NVIDIA Corporation\PhysX\Common;C:\Program Files\NVIDIA Corporation\NVIDIA NvDLISR;C:\Users\slcet\Anaconda3;C:\Users\slcet\Anaconda3\Library\mingw-w64\bin;C:\Users\slcet\Anaconda3\Library\usr\bin;C:\Users\slcet\Anaconda3\Library\bin;C:\Users\slcet\Anaconda3\Scripts;C:\Users\slcet\AppData\Local\Microsoft\WindowsApps

[pgunn]

Let's try this. Launch python this way instead:

C:\Users\slcet\Anaconda3\envs\caiman\bin\python.exe

and then retry that import sys/printing sys path thing.

[slceto]

All I have in my caiman\bin folder is a folder called pycache and the caimanmanager.py file

[pgunn]

Ok, so from your caiman environment, can you do this? conda list | findstr python

[slceto]

ipython 7.9.0 py37h5ca1d4c_0 conda-forge ipython_genutils 0.2.0 py_1 conda-forge python 3.7.3 h510b542_1 conda-forge python-dateutil 2.8.1 py_0 conda-forge

[pgunn]

Do you have a python.exe here? C:\Users\slcet\Anaconda3\envs\caiman

[slceto]

Yes, print(sys.path) from this python gives: ['', 'C:\Users\slcet\Anaconda3\envs\caiman\python37.zip', 'C:\Users\slcet\Anaconda3\envs\caiman\DLLs', 'C:\Users\slcet\Anaconda3\envs\caiman\lib', 'C:\Users\slcet\Anaconda3\envs\caiman', 'C:\Users\slcet\Anaconda3\envs\caiman\lib\site-packages', 'C:\Users\slcet\Anaconda3\envs\caiman\lib\site-packages\win32', 'C:\Users\slcet\Anaconda3\envs\caiman\lib\site-packages\win32\lib', 'C:\Users\slcet\Anaconda3\envs\caiman\lib\site-packages\Pythonwin']

[pgunn]

Great, that's looking more promising. See if you can run C:\Users\slcet\Anaconda3\envs\caiman\python.exe caimanmanager.py install

[slceto]

Doing this in the caiman environment in Anaconda Prompt gave me C:\Users\slcet\Anaconda3\envs\caiman\python.exe: can't open file 'caimanmanager.py': [Errno 2] No such file or directory

[pgunn]

Ah. Ok, so let's do it this way.

SET PATH=C:\Users\slcet\Anaconda3\envs\caiman;%PATH%

After that, if you just type python and repeat the import sys/print(sys.path) thing, you should hopefully see C:\Users\slcet\Anaconda3\envs\caiman\lib\site-packages and stuff like that in the results. Assuming that you do, try running caimanmanager.py install again.

[slceto]

should I set the path in the caiman python or the anaconda prompt?

[pgunn]

That SET PATH line is for your shell, not for python.

[slceto]

okay, after setting the path, print(sys.path) gives: ['', 'C:\Users\slcet\Anaconda3\envs\caiman\python37.zip', 'C:\Users\slcet\Anaconda3\envs\caiman\DLLs', 'C:\Users\slcet\Anaconda3\envs\caiman\lib', 'C:\Users\slcet\Anaconda3\envs\caiman', 'C:\Users\slcet\Anaconda3\envs\caiman\lib\site-packages', 'C:\Users\slcet\Anaconda3\envs\caiman\lib\site-packages\win32', 'C:\Users\slcet\Anaconda3\envs\caiman\lib\site-packages\win32\lib', 'C:\Users\slcet\Anaconda3\envs\caiman\lib\site-packages\Pythonwin']

but trying to install caimanmanager I still get Traceback (most recent call last): File "C:\Users\slcet\Anaconda3\envs\caiman\bin\caimanmanager.py", line 16, in from caiman.paths import caiman_datadir ModuleNotFoundError: No module named 'caiman'

[pgunn]

Do you have a 'C:\Users\slcet\Anaconda3\envs\caiman\lib\site-packages\caiman ?

[slceto]

yes, except \lib\ has a capital "L" making it \Lib\ if that matters

[pgunn]

I think that with most filesystems Windows supports, it's case-remembering but not case-distinguishing, so that shouldn't be an issue. So we've gotten it into the search path, and it appears to be there, but something's not quite coming together right.

Let's try, from a python prompt in your caiman environment (with that PATH change done in the same shell), directly doing:

import caiman

[pgunn]

Also, can you send me the output of this?

dir 'C:\Users\slcet\Anaconda3\envs\caiman\lib\site-packages\Caiman

[slceto]

this is what I get with import caimanTraceback (most recent call last): File "", line 1, in File "C:\Users\slcet\Anaconda3\envs\caiman\lib\site-packages\caiman__init__.py", line 3, in from .base.movies import movie, load, load_movie_chain File "C:\Users\slcet\Anaconda3\envs\caiman\lib\site-packages\caiman\base\movies.py", line 19, in import cv2 ImportError: DLL load failed: The specified module could not be found.

[pgunn]

So maybe this is an OpenCV issue. There are not some extra lines below indicating which DLL it can't find?

[slceto]

No more lines about the DLL

I can't get the dir 'C:\Users\slcet\Anaconda3\envs\caiman\lib\site-packages\Caiman line to work in either the shell or python

[slceto]

(syntax errors in both)^

[pgunn]

The dir line is meant for your shell. Try removing the single quote before the c:\

[slceto]

okay, that worked:

Volume in drive C has no label. Volume Serial Number is D8FE-7882

Directory of C:\Users\slcet\Anaconda3\envs\caiman\lib\site-packages\Caiman

11/11/2019 07:06 PM

. 11/11/2019 07:06 PM .. 11/11/2019 07:06 PM base 11/11/2019 07:06 PM behavior 10/09/2019 02:59 PM 16,631 cluster.py 10/09/2019 02:59 PM 28,868 components_evaluation.py 11/11/2019 07:06 PM external 11/11/2019 07:06 PM gui 10/09/2019 02:59 PM 24,916 mmapping.py 10/09/2019 02:59 PM 121,492 motion_correction.py 10/09/2019 02:59 PM 1,539 paths.py 11/11/2019 07:06 PM source_extraction 10/09/2019 02:59 PM 27,690 summary_images.py 11/11/2019 07:06 PM tests 11/11/2019 07:06 PM utils 10/09/2019 02:59 PM 333 init.py 11/11/2019 07:06 PM pycache 7 File(s) 221,469 bytes 10 Dir(s) 16,807,137,280 bytes free

[pgunn]

Can you send me the output of conda info ?

[slceto]

active environment : caiman active env location : C:\Users\slcet\Anaconda3\envs\caiman shell level : 2 user config file : C:\Users\slcet.condarc populated config files : conda version : 4.7.12 conda-build version : 3.18.9 python version : 3.7.4.final.0 virtual packages : __cuda=10.1 base environment : C:\Users\slcet\Anaconda3 (writable) channel URLs : https://repo.anaconda.com/pkgs/main/win-64 https://repo.anaconda.com/pkgs/main/noarch https://repo.anaconda.com/pkgs/r/win-64 https://repo.anaconda.com/pkgs/r/noarch https://repo.anaconda.com/pkgs/msys2/win-64 https://repo.anaconda.com/pkgs/msys2/noarch package cache : C:\Users\slcet\Anaconda3\pkgs C:\Users\slcet.conda\pkgs C:\Users\slcet\AppData\Local\conda\conda\pkgs envs directories : C:\Users\slcet\Anaconda3\envs C:\Users\slcet.conda\envs C:\Users\slcet\AppData\Local\conda\conda\envs platform : win-64 user-agent : conda/4.7.12 requests/2.22.0 CPython/3.7.4 Windows/10 Windows/10.0.18362 administrator : False netrc file : None offline mode : False

[pgunn]

This is probably as far as I can go tonight (it's 11:40PM here), as it's late and I also don't have a windows system at home to experiment with. I'll see if I can get some more things for us to try tomorrow when I get into work and have access to a Windows VM.

[slceto]

No problem, sorry for keeping you up so late!

[slceto]

Really appreciate the help

[pgunn]

I just made a new environment on a windows test box, and you have exactly the same version of OpenCV that I do, but my install works. I'm currently working on the theory that somehow your PATH stuff isn't being managed correctly by conda right now.

So we know for later, do you use conda for anything else on this host? If not, we may want to try a complete uninstall and reinstall later to see if we can get the plumbing working correctly.

In the meantime, let's give this a try. If you start a new shell, and in that shell, do the following:

conda activate caiman
SET PATH=C:\Users\slcet\Anaconda3\envs\caiman;C:\Users\slcet\Anaconda3\envs\caiman\Library\bin;%PATH%

Then, still in that shell, launch python, and then inside there, do an import cv2 and let me know if that works. If it does, then the OpenCV issue was just a path thing, and you can try the caimanmanager.py install again. If not, we'll keep trying to figure this out. It's also possible that your system has an association for .py for a particular python interpreter that we may need to undo.

[slceto]

I do not use conda for anything else (This is my first time trying to use Python at all).

Just to make sure I did what you asked properly, this is my whole sequence in Anaconda Prompt (import cv2 didn't work):

(base) C:\Users\slcet>conda activate caiman

C:\Users\slcet>set MKL_NUM_THREADS=1

C:\Users\slcet>set OPENBLAS_NUM_THREADS=1

(caiman) C:\Users\slcet>SET PATH=C:\Users\slcet\Anaconda3\envs\caiman;C:\Users\slcet\Anaconda3\envs\caiman\Library\bin;%PATH%

(caiman) C:\Users\slcet>python Python 3.7.3 | packaged by conda-forge | (default, Jul 1 2019, 22:01:29) [MSC v.1900 64 bit (AMD64)] :: Anaconda, Inc. on win32 Type "help", "copyright", "credits" or "license" for more information.

import cv2 Traceback (most recent call last): File "", line 1, in ImportError: DLL load failed: The specified module could not be found.