search

flatironinstitute / CaImAn

Computational toolbox for large scale Calcium Imaging Analysis, including movie handling, motion correction, source extraction, spike deconvolution and result visualization.
https://caiman.readthedocs.io
GNU General Public License v2.0
637 stars 370 forks source link

Installing via conda on HPC takes forever. Is there a way to install Caiman using pip? #994

Closed neuroady closed 2 years ago

neuroady commented 2 years ago

Is there a way to manually install everything? Sorry for the bad question format. But, this is really not working for me on HPC.

pgunn commented 2 years ago

Hello, I thought I had changed the instructions to prefer mamba, which shouldn't have this issue. Please give that a go

We recommend people not go with pip (it's possible, but less supported and some packages like z5py don't work with pip).

neuroady commented 2 years ago

Hi Pat, thanks for the quick reply. The problem is that installation of mamba also experiences exactly the same hangup. That setup hasn't finished either. Also, the installation instructions on conda-forge still recommend installing caiman directly (conda install -c conda-forge caiman or conda install -c conda-forge/label/cf202003 caiman). That's why I gave it a try.

pgunn commented 2 years ago

Ah, it sounds like either your cluster system isn't giving you much memory, or your network filesystem is not fast enough to use conda at a reasonable rate. I'm not sure how I can help you with that; if you need to you can try doing a checkout of the sources and use pip with the included requirements.txt, but note that some functionality (mostly zarr/n5 support, although dependencies are less solid there) may be limited and it'll be harder for me to support you.

I can't fix the install instructions on conda-forge; I know they're working on integrating the mamba solver into the next release of conda, but that's not out yet. I'm hoping when it's released this problem will be diminished, but people on HPC clusters will generally have some local oddities that need to be dealt with (strange race conditions, non-posix behaviour, etc).

neuroady commented 2 years ago

Thanks for that detailed answer, Pat. What you say about HPC makes a lot of sense. I'll try to get my uni's HPC support involved. Parallelly, I'll give requirements.txt a try.
I'll now close the issue. Because it appears the problems is mainly to do with HPC oddities. Thanks again for the help!

bcfvlad commented 2 years ago

Hello,

I am a colleague of @neuroady, we're trying CaImAn together.

I tried installing caiman using : pip install -r requirements.txt All packages and the requirements files are successfully installed, and then following the closed issue, i ran python setup.py build_ext -i and it gave me the following dump

`(caiman_pyvenv) [fr_vi1005@login1 CaImAn]$ python setup.py build_ext -i Compiling caiman/source_extraction/cnmf/oasis.pyx because it changed. [1/1] Cythonizing caiman/source_extraction/cnmf/oasis.pyx /home/fr/fr_fr/fr_vi1005/caiman_pyvenv/lib/python3.9/site-packages/Cython/Compiler/Main.py:369: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /home/fr/fr_fr/fr_vi1005/CaImAn/caiman/source_extraction/cnmf/oasis.pyx tree = Parsing.p_module(s, pxd, full_module_name) running build_ext building 'caiman.source_extraction.cnmf.oasis' extension creating build creating build/temp.linux-x86_64-3.9 creating build/temp.linux-x86_64-3.9/caiman creating build/temp.linux-x86_64-3.9/caiman/source_extraction creating build/temp.linux-x86_64-3.9/caiman/source_extraction/cnmf gcc -pthread -B /home/fr/fr_fr/fr_vi1005/anaconda3/compiler_compat -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -Wall -fPIC -O2 -isystem /home/fr/fr_fr/fr_vi1005/anaconda3/include -I/home/fr/fr_fr/fr_vi1005/anaconda3/include -fPIC -O2 -isystem /home/fr/fr_fr/fr_vi1005/anaconda3/include -fPIC -I/home/fr/fr_fr/fr_vi1005/caiman_pyvenv/lib/python3.9/site-packages/numpy/core/include -I/home/fr/fr_fr/fr_vi1005/caiman_pyvenv/include -I/home/fr/fr_fr/fr_vi1005/anaconda3/include/python3.9 -c caiman/source_extraction/cnmf/oasis.cpp -o build/temp.linux-x86_64-3.9/caiman/source_extraction/cnmf/oasis.o In file included from /home/fr/fr_fr/fr_vi1005/caiman_pyvenv/lib/python3.9/site-packages/numpy/core/include/numpy/ndarraytypes.h:1969:0, from /home/fr/fr_fr/fr_vi1005/caiman_pyvenv/lib/python3.9/site-packages/numpy/core/include/numpy/ndarrayobject.h:12, from /home/fr/fr_fr/fr_vi1005/caiman_pyvenv/lib/python3.9/site-packages/numpy/core/include/numpy/arrayobject.h:4, from caiman/source_extraction/cnmf/oasis.cpp:735: /home/fr/fr_fr/fr_vi1005/caiman_pyvenv/lib/python3.9/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]

warning "Using deprecated NumPy API, disable it with " \

^ caiman/source_extraction/cnmf/oasis.cpp: In function 'PyObject pyx_pf_6caiman_17source_extraction_4cnmf_5oasis_5OASIS_4fit_next_tmp(pyx_obj_6caiman_17source_extraction_4cnmf_5oasis_OASIS, PyObject, PyObject)': caiman/source_extraction/cnmf/oasis.cpp:5106:71: warning: comparison between signed and unsigned integer expressions [-Wsign-compare] if ((((pyx_v_newpool.t + pyx_v_k) < pyx_v_self->_y.size()) != 0)) { ^ caiman/source_extraction/cnmf/oasis.cpp: In function 'PyObject* pyx_pf_6caiman_17source_extraction_4cnmf_5oasis_20constrained_oasisAR1_2oasis(PyObject, std::vector<__pyx_t_6caiman_17source_extraction_4cnmf_5oasis_Pool>, __pyx_t_6caiman_17source_extraction_4cnmf_5oasis_SINGLE, PyArrayObject)': caiman/source_extraction/cnmf/oasis.cpp:10922:52: warning: comparison between signed and unsigned integer expressions [-Wsign-compare] pyx_t_1 = ((pyx_v_i < (pyx_v_P.size() - 1)) != 0); ^ caiman/source_extraction/cnmf/oasis.cpp: In function 'PyObject __pyx_pf_6caiman_17source_extraction_4cnmf_5oasis_20constrained_oasisAR1_8bar(PyObject, PyObject, PyObject, PyObject, PyObject)': caiman/source_extraction/cnmf/oasis.cpp:12968:62: warning: comparison between signed and unsigned integer expressions [-Wsign-compare] if (((pyx_cur_scope->pyx_v_P.size() < pyx_cur_scope->pyx_v_optimize_g) != 0)) { ^ caiman/source_extraction/cnmf/oasis.cpp: In function 'PyObject __pyx_pf_6caiman_17source_extraction_4cnmf_5oasis_2constrained_oasisAR1(PyObject, PyArrayObject*, __pyx_t_6caiman_17source_extraction_4cnmf_5oasis_SINGLE, pyx_t_6caiman_17source_extraction_4cnmf_5oasis_SINGLE, PyLongObject, PyLongObject, int, int, int, int, pyx_t_6caiman_17source_extraction_4cnmf_5oasis_SINGLE)': caiman/source_extraction/cnmf/oasis.cpp:14191:39: warning: comparison between signed and unsigned integer expressions [-Wsign-compare] for (pyx_t_1 = 0; pyx_t_1 < pyx_t_20; pyx_t_1+=1) { ^ caiman/source_extraction/cnmf/oasis.cpp:14201:75: warning: comparison between signed and unsigned integer expressions [-Wsign-compare] pyx_t_15 = ((pyx_v_i == (pyx_cur_scope->pyx_v_P.size() - 1)) != 0); ^ caiman/source_extraction/cnmf/oasis.cpp:14410:39: warning: comparison between signed and unsigned integer expressions [-Wsign-compare] for (pyx_t_1 = 0; pyx_t_1 < pyx_t_20; pyx_t_1+=1) { ^ caiman/source_extraction/cnmf/oasis.cpp:14528:58: warning: comparison between signed and unsigned integer expressions [-Wsign-compare] pyx_t_2 = ((pyx_v_count < (pyx_v_max_iter - 1)) != 0); ^ caiman/source_extraction/cnmf/oasis.cpp:14551:41: warning: comparison between signed and unsigned integer expressions [-Wsign-compare] for (pyx_t_1 = 0; pyx_t_1 < pyx_t_20; __pyx_t_1+=1) { ^ caiman/source_extraction/cnmf/oasis.cpp:14581:41: warning: comparison between signed and unsigned integer expressions [-Wsign-compare] for (pyx_t_1 = 0; pyx_t_1 < pyx_t_20; pyx_t_1+=1) { ^ caiman/source_extraction/cnmf/oasis.cpp:14914:41: warning: comparison between signed and unsigned integer expressions [-Wsign-compare] for (pyx_t_1 = 0; pyx_t_1 < pyx_t_20; pyx_t_1+=1) { ^ caiman/source_extraction/cnmf/oasis.cpp:15641:38: warning: comparison between signed and unsigned integer expressions [-Wsign-compare] pyx_t_25 = ((pyx_v_count < pyx_v_max_iter) != 0); ^ caiman/source_extraction/cnmf/oasis.cpp:15664:39: warning: comparison between signed and unsigned integer expressions [-Wsign-compare] for (pyx_t_1 = 0; pyx_t_1 < pyx_t_20; pyx_t_1+=1) { ^ caiman/source_extraction/cnmf/oasis.cpp:15674:75: warning: comparison between signed and unsigned integer expressions [-Wsign-compare] pyx_t_15 = ((pyx_v_i == (pyx_cur_scope->pyx_v_P.size() - 1)) != 0); ^ caiman/source_extraction/cnmf/oasis.cpp:15817:39: warning: comparison between signed and unsigned integer expressions [-Wsign-compare] for (pyx_t_1 = 0; pyx_t_1 < pyx_t_20; pyx_t_1+=1) { ^ caiman/source_extraction/cnmf/oasis.cpp:16030:39: warning: comparison between signed and unsigned integer expressions [-Wsign-compare] for (pyx_t_1 = 0; pyx_t_1 < pyx_t_20; pyx_t_1+=1) { ^ caiman/source_extraction/cnmf/oasis.cpp:16267:59: warning: comparison between signed and unsigned integer expressions [-Wsign-compare] pyx_t_25 = ((pyx_v_count < (pyx_v_max_iter - 1)) != 0); ^ caiman/source_extraction/cnmf/oasis.cpp:16290:41: warning: comparison between signed and unsigned integer expressions [-Wsign-compare] for (pyx_t_1 = 0; pyx_t_1 < pyx_t_20; pyx_t_1+=1) { ^ caiman/source_extraction/cnmf/oasis.cpp:16320:41: warning: comparison between signed and unsigned integer expressions [-Wsign-compare] for (pyx_t_1 = 0; pyx_t_1 < pyx_t_20; pyx_t_1+=1) { ^ caiman/source_extraction/cnmf/oasis.cpp:16718:41: warning: comparison between signed and unsigned integer expressions [-Wsign-compare] for (pyx_t_1 = 0; pyx_t_1 < pyx_t_20; pyx_t_1+=1) { ^ caiman/source_extraction/cnmf/oasis.cpp:17235:37: warning: comparison between signed and unsigned integer expressions [-Wsign-compare] for (pyx_t_1 = 0; pyx_t_1 < pyx_t_20; pyx_t_1+=1) { ^ caiman/source_extraction/cnmf/oasis.cpp:18052:39: warning: comparison between signed and unsigned integer expressions [-Wsign-compare] for (pyx_t_1 = 0; pyx_t_1 < pyx_t_20; __pyx_t_1+=1) { ^ g++ -pthread -B /home/fr/fr_fr/fr_vi1005/anaconda3/compiler_compat -shared -Wl,-rpath,/home/fr/fr_fr/fr_vi1005/anaconda3/lib -Wl,-rpath-link,/home/fr/fr_fr/fr_vi1005/anaconda3/lib -L/home/fr/fr_fr/fr_vi1005/anaconda3/lib -L/home/fr/fr_fr/fr_vi1005/anaconda3/lib -Wl,-rpath,/home/fr/fr_fr/fr_vi1005/anaconda3/lib -Wl,-rpath-link,/home/fr/fr_fr/fr_vi1005/anaconda3/lib -L/home/fr/fr_fr/fr_vi1005/anaconda3/lib build/temp.linux-x86_64-3.9/caiman/source_extraction/cnmf/oasis.o -o /home/fr/fr_fr/fr_vi1005/CaImAn/caiman/source_extraction/cnmf/oasis.cpython-39-x86_64-linux-gnu.so`

Ignoring the warning, I proceeded to run python caimanmanager.py install which gave me the following error: (caiman_pyvenv) [fr_vi1005@login1 CaImAn]$ python caimanmanager.py install Traceback (most recent call last): File "/home/fr/fr_fr/fr_vi1005/CaImAn/caimanmanager.py", line 16, in <module> from caiman.paths import caiman_datadir File "/home/fr/fr_fr/fr_vi1005/CaImAn/caiman/__init__.py", line 11, in <module> __version__ = pkg_resources.get_distribution('caiman').version File "/home/fr/fr_fr/fr_vi1005/caiman_pyvenv/lib/python3.9/site-packages/pkg_resources/__init__.py", line 466, in get_distribution dist = get_provider(dist) File "/home/fr/fr_fr/fr_vi1005/caiman_pyvenv/lib/python3.9/site-packages/pkg_resources/__init__.py", line 342, in get_provider return working_set.find(moduleOrReq) or require(str(moduleOrReq))[0] File "/home/fr/fr_fr/fr_vi1005/caiman_pyvenv/lib/python3.9/site-packages/pkg_resources/__init__.py", line 886, in require needed = self.resolve(parse_requirements(requirements)) File "/home/fr/fr_fr/fr_vi1005/caiman_pyvenv/lib/python3.9/site-packages/pkg_resources/__init__.py", line 772, in resolve raise DistributionNotFound(req, requirers) pkg_resources.DistributionNotFound: The 'caiman' distribution was not found and is required by the application

and due to this error, I cannot import caiman...well because it wasn't found.

I hope that there is a way to come around this issue.

pgunn commented 2 years ago

Hello, Two things: 1) What compiler(s) do you have available on your cluster? What distro is it running? 2) Please don't do the build_ext step yourself; just do a pip install . if you're going to do a pip-based install instead of conda. It will do the rest.

You might (or might not) run into the same issue with the second adjustment, but please try it. If it still fails we'll dig into why.

pgunn commented 2 years ago

One other thing ; if you have a way to instead run a docker container on your HPC environment, you might find the included dockerfile to be an easier way to do this; it should be almost as easy as conda. Compiling it yourself is the most hazard-prone route.

neuroady commented 2 years ago

Hello, Two things:

1. What compiler(s) do you have available on your cluster? What distro is it running?

2. Please don't do the build_ext step yourself; just do a `pip install .` if you're going to do a pip-based install instead of conda. It will do the rest.

You might (or might not) run into the same issue with the second adjustment, but please try it. If it still fails we'll dig into why.

Hi @pgunn, Thanks for this comment. We gave this a try. Here's how we did it.

  1. create a new venv (python 3.9.7)
  2. upadate pip
  3. cd into CaImAn (git clone), and run pip install -r requirements.txt
  4. in the same dir, run: pip install . no errors: 
    Processing /home/fr/fr_fr/fr_as1587/vlad/CaImAn
Preparing metadata (setup.py) ... done
Building wheels for collected packages: caiman
Building wheel for caiman (setup.py) ... done
Created wheel for caiman: filename=caiman-1.9.12-cp39-cp39-linux_x86_64.whl size=123121259 sha256=53e9b39f34cbb12d90833fc24c8dec66cd2f39211df17c2e84f3291362766e42
Stored in directory: /tmp/30402264.mg1.nemo.privat/pip-ephem-wheel-cache-0e47sqqw/wheels/8f/52/b0/a6a831f4c6e877512d69a1e6edaa518bec20ea2245e0b48349
Successfully built caiman
Installing collected packages: caiman
Successfully installed caiman-1.9.12

    Then we opened an ipython session and imported caiman. We get the following error dump:

    
(vlad_venv) [fr_as1587@n4655 CaImAn]$ ipython
Python 3.9.7 (default, Oct 27 2021, 01:23:21) 
Type 'copyright', 'credits' or 'license' for more information
IPython 8.4.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import caiman

ImportError Traceback (most recent call last) Input In [1], in <cell line: 1>() ----> 1 import caiman

File ~/vlad/CaImAn/caiman/init.py:4, in 1 #!/usr/bin/env python 3 import pkg_resources ----> 4 from .base.movies import movie, load, load_movie_chain, _load_behavior 5 from .base.timeseries import concatenate 6 from .cluster import start_server, stop_server

File ~/vlad/CaImAn/caiman/base/movies.py:59, in 56 from .traces import trace 58 from ..mmapping import load_memmap ---> 59 from ..utils import visualization 60 from .. import summary_images as si 61 from ..motion_correction import apply_shift_online, motion_correct_online

File ~/vlad/CaImAn/caiman/utils/visualization.py:37, in 34 import functools as fct 36 from ..base.rois import com ---> 37 from ..summary_images import local_correlations 39 try: 40 cv2.setNumThreads(0)

File ~/vlad/CaImAn/caiman/summary_images.py:23, in 20 from typing import Any, List, Optional, Tuple 22 import caiman as cm ---> 23 from caiman.source_extraction.cnmf.pre_processing import get_noise_fft 24 from caiman.source_extraction.cnmf.utilities import get_file_size 26 def max_correlation_image(Y, bin_size: int = 1000, eight_neighbours: bool = True, swap_dim: bool = True) -> np.ndarray:

File ~/vlad/CaImAn/caiman/source_extraction/cnmf/init.py:12, in 10 from . import spatial 11 from . import temporal ---> 12 from . import oasis 13 from . import params 14 from . import online_cnmf

ImportError: cannot import name 'oasis' from partially initialized module 'caiman.source_extraction.cnmf' (most likely due to a circular import) (/home/fr/fr_fr/fr_as1587/vlad/CaImAn/caiman/source_extraction/cnmf/init.py)


-----
This is something we encountered before as well.

For this out HPC environment is populated by: `python 3.9.7` and `gnu 10.2` ( we do have older gnu versions). The HPC runs over `CentOS 7.9 (Rev. 21), 3.10.0-1160.71.1`
neuroady commented 2 years ago

; if you have a way to instead run a docker container on your HPC environment, you might find the included dockerfile to be an easier way to do this; it should b

We do have singularity support, but no docker. Maybe we can try to install docker locally.

bcfvlad commented 2 years ago

I have retied the installation of caiman with the base conda on HPC, carefully following the instructions from scratch. The installation of mamba (conda-forge), caiman in the development mode (pip install -e .), as well as the caimanmanager steps (python caimanmanager.py install --inplace) all went smoothly and without any backlash.

As far as I understand, I have messed up my conda settings/configurations in the first place, which was creating problems all the way through.

Thanks for the help! @pgunn @neuroady

pgunn commented 2 years ago

Happy you got it working; the pip process you did probably also worked, but when installed that way caiman only works from outside the caiman source directory.

Best wishes

neuroady commented 2 years ago

can confirm that the pip process also works when importing from outside the source dir. Thanks again for all the support!