search

nel-lab / mesmerize-core

High level pandas-based API for batch analysis of Calcium Imaging data using CaImAn
Other
60 stars 15 forks source link

fresh ubuntu install - caiman #117

Closed katesal327 closed 1 year ago

katesal327 commented 2 years ago

Hi having errors installing caiman with a fresh ubuntu 22 os. Error is below:

Processing /home/katie/repos/CaImAn Preparing metadata (setup.py) ... done Building wheels for collected packages: caiman Building wheel for caiman (setup.py) ... error error: subprocess-exited-with-error

× python setup.py bdist_wheel did not run successfully. │ exit code: 1 ╰─> [11 lines of output] running bdist_wheel running build running build_py running build_ext building 'caiman.source_extraction.cnmf.oasis' extension x86_64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -fPIC -I/home/katie/python-venvs/mesmerize-core/lib/python3.10/site-packages/numpy/core/include -I/home/katie/python-venvs/mesmerize-core/include -I/usr/include/python3.10 -c caiman/source_extraction/cnmf/oasis.cpp -o build/temp.linux-x86_64-cpython-310/caiman/source_extraction/cnmf/oasis.o caiman/source_extraction/cnmf/oasis.cpp:29:10: fatal error: Python.h: No such file or directory 29 | #include "Python.h" | ^~ compilation terminated. error: command '/usr/bin/x86_64-linux-gnu-gcc' failed with exit code 1 [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip. ERROR: Failed building wheel for caiman Running setup.py clean for caiman Failed to build caiman Installing collected packages: caiman Running setup.py install for caiman ... error error: subprocess-exited-with-error

× Running setup.py install for caiman did not run successfully. │ exit code: 1 ╰─> [110 lines of output] running install /home/katie/python-venvs/mesmerize-core/lib/python3.10/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools. warnings.warn( running build running build_py creating build creating build/lib.linux-x86_64-cpython-310 creating build/lib.linux-x86_64-cpython-310/caiman copying caiman/paths.py -> build/lib.linux-x86_64-cpython-310/caiman copying caiman/mmapping.py -> build/lib.linux-x86_64-cpython-310/caiman copying caiman/components_evaluation.py -> build/lib.linux-x86_64-cpython-310/caiman copying caiman/cluster.py -> build/lib.linux-x86_64-cpython-310/caiman copying caiman/init.py -> build/lib.linux-x86_64-cpython-310/caiman copying caiman/summary_images.py -> build/lib.linux-x86_64-cpython-310/caiman copying caiman/motion_correction.py -> build/lib.linux-x86_64-cpython-310/caiman creating build/lib.linux-x86_64-cpython-310/caiman/tests copying caiman/tests/test_temporal.py -> build/lib.linux-x86_64-cpython-310/caiman/tests copying caiman/tests/test_memmapping.py -> build/lib.linux-x86_64-cpython-310/caiman/tests copying caiman/tests/test_movies.py -> build/lib.linux-x86_64-cpython-310/caiman/tests copying caiman/tests/comparison_humans.py -> build/lib.linux-x86_64-cpython-310/caiman/tests copying caiman/tests/init.py -> build/lib.linux-x86_64-cpython-310/caiman/tests copying caiman/tests/test_deconvolution.py -> build/lib.linux-x86_64-cpython-310/caiman/tests copying caiman/tests/test_pre_processing.py -> build/lib.linux-x86_64-cpython-310/caiman/tests copying caiman/tests/comparison_general.py -> build/lib.linux-x86_64-cpython-310/caiman/tests copying caiman/tests/test_spatial.py -> build/lib.linux-x86_64-cpython-310/caiman/tests copying caiman/tests/test_toydata.py -> build/lib.linux-x86_64-cpython-310/caiman/tests copying caiman/tests/test_tensorflow.py -> build/lib.linux-x86_64-cpython-310/caiman/tests copying caiman/tests/test_import.py -> build/lib.linux-x86_64-cpython-310/caiman/tests copying caiman/tests/test_onacid.py -> build/lib.linux-x86_64-cpython-310/caiman/tests copying caiman/tests/comparison_humans_online.py -> build/lib.linux-x86_64-cpython-310/caiman/tests copying

kushalkolar commented 2 years ago

This shows that your python header files are missing:

Python.h: No such file or directory

The python headers are in python3-dev, you should probably just install all the python3 packages anyways which you can do like this:

sudo apt install python3*

You're probably missing general build tools too since it's a fresh install so:

sudo apt install build-essential

video and image stuff

sudo apt install ffmpeg libavcodec-dev libx264-dev libjpeg-dev libpng-dev libtiff-dev zlib1g-dev

required for high performance linear algebra and parallelization 

sudo apt install libopenblas-dev liblapack-dev libtbb2 libtbb-dev

You can just install all of this using:

sudo apt install python3* build-essential ffmpeg libavcodec-dev libx264-dev libjpeg-dev libpng-dev libtiff-dev libopenblas-dev liblapack-dev libtbb2 libtbb-dev zlib1g-dev

That should be everything you need but it's possible I may have missed something.

katesal327 commented 2 years ago

okay I was able to install caiman successfully...next problem is running "caimanmanger.py install" this is my output Traceback (most recent call last): File "/home/katie/env/repos/CaImAn/caimanmanager.py", line 16, in from caiman.paths import caiman_datadir File "/home/katie/env/repos/CaImAn/caiman/init.py", line 4, in from .base.movies import movie, load, load_movie_chain, _load_behavior File "/home/katie/env/repos/CaImAn/caiman/base/movies.py", line 18, in import cv2 File "/home/katie/env/mesmerize-core/lib/python3.10/site-packages/cv2/init.py", line 181, in bootstrap() File "/home/katie/env/mesmerize-core/lib/python3.10/site-packages/cv2/init.py", line 153, in bootstrap native_module = importlib.import_module("cv2") File "/usr/lib/python3.10/importlib/init.py", line 126, in import_module return _bootstrap._gcd_import(name[level:], package, level) ImportError: libGL.so.1: cannot open shared object file: No such file or directory

katesal327 commented 2 years ago

opencv-python 4.6.0.66

is output from "pip list"

kushalkolar commented 2 years ago

this should give you libgl 

sudo apt install libgl*
katesal327 commented 2 years ago

assuming I can ignore the other errors for now? I am trying to install it successfully here first so I know what to do when I try to install it on the HPC.ked now thanks...assuming I can ignore the other errors for now? I am trying to install it successfully here first so I know what to do when I try to install it on the HPC.

2022-10-09 20:48:21.070823: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 AVX512F AVX512_VNNI FMA To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. 2022-10-09 20:48:21.309876: I tensorflow/core/util/util.cc:169] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable TF_ENABLE_ONEDNN_OPTS=0. 2022-10-09 20:48:21.371241: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /home/katie/env/mesmerize-core/lib/python3.10/site-packages/cv2/../../lib64: 2022-10-09 20:48:21.371273: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine. 2022-10-09 20:48:21.407775: E tensorflow/stream_executor/cuda/cuda_blas.cc:2981] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered 2022-10-09 20:48:22.117812: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /home/katie/env/mesmerize-core/lib/python3.10/site-packages/cv2/../../lib64: 2022-10-09 20:48:22.117920: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /home/katie/env/mesmerize-core/lib/python3.10/site-packages/cv2/../../lib64: 2022-10-09 20:48:22.117945: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.

katesal327 commented 2 years ago

_tested mesmerizecore import and this is the error:

In [1]: import mesmerize_core 2022-10-09 20:54:59.376347: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 AVX512F AVX512_VNNI FMA To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. 2022-10-09 20:54:59.454518: I tensorflow/core/util/util.cc:169] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable TF_ENABLE_ONEDNN_OPTS=0. 2022-10-09 20:54:59.457972: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /home/katie/env/mesmerize-core/lib/python3.10/site-packages/cv2/../../lib64: 2022-10-09 20:54:59.458002: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine. 2022-10-09 20:54:59.475300: E tensorflow/stream_executor/cuda/cuda_blas.cc:2981] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered 2022-10-09 20:54:59.849877: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /home/katie/env/mesmerize-core/lib/python3.10/site-packages/cv2/../../lib64: 2022-10-09 20:54:59.849958: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /home/katie/env/mesmerize-core/lib/python3.10/site-packages/cv2/../../lib64: 2022-10-09 20:54:59.849965: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.

ImportError Traceback (most recent call last) Cell In [1], line 1 ----> 1 import mesmerize_core

File ~/env/mesmerize-core/lib/python3.10/site-packages/mesmerize_core/init.py:1 ----> 1 from .batch_utils import ( 2 set_parent_raw_data_path, 3 get_parent_raw_data_path, 4 load_batch, 5 create_batch, 6 ) 7 from .caiman_extensions import * 8 from pathlib import Path

File ~/env/mesmerize-core/lib/python3.10/site-packages/mesmerize_core/batch_utils.py:7 3 from typing import Union 5 import pandas as pd ----> 7 from .algorithms import cnmf, mcorr 8 from .algorithms import cnmfe 9 from .utils import validate_path

File ~/env/mesmerize-core/lib/python3.10/site-packages/mesmerize_core/algorithms/cnmf.py:3 1 """Performs CNMF in a separate process""" 2 import click ----> 3 import caiman as cm 4 from caiman.source_extraction.cnmf import cnmf as cnmf 5 from caiman.source_extraction.cnmf.params import CNMFParams

File ~/env/repos/CaImAn/caiman/init.py:4 1 #!/usr/bin/env python 3 import pkg_resources ----> 4 from .base.movies import movie, load, load_movie_chain, _load_behavior 5 from .base.timeseries import concatenate 6 from .cluster import start_server, stop_server

File ~/env/repos/CaImAn/caiman/base/movies.py:59 56 from .traces import trace 58 from ..mmapping import load_memmap ---> 59 from ..utils import visualization 60 from .. import summary_images as si 61 from ..motion_correction import apply_shift_online, motion_correct_online

File ~/env/repos/CaImAn/caiman/utils/visualization.py:37 34 import functools as fct 36 from ..base.rois import com ---> 37 from ..summary_images import local_correlations 39 try: 40 cv2.setNumThreads(0)

File ~/env/repos/CaImAn/caiman/summary_images.py:23 20 from typing import Any, List, Optional, Tuple 22 import caiman as cm ---> 23 from caiman.source_extraction.cnmf.pre_processing import get_noise_fft 24 from caiman.source_extraction.cnmf.utilities import get_file_size 26 def max_correlation_image(Y, bin_size: int = 1000, eight_neighbours: bool = True, swap_dim: bool = True) -> np.ndarray:

File ~/env/repos/CaImAn/caiman/source_extraction/cnmf/init.py:12 10 from . import spatial 11 from . import temporal ---> 12 from . import oasis 13 from . import params 14 from . import online_cnmf

ImportError: cannot import name 'oasis' from partially initialized module 'caiman.source_extraction.cnmf' (most likely due to a circular import) (/home/katie/env/repos/CaImAn/caiman/source_extraction/cnmf/init.py)

In [2]:

kushalkolar commented 2 years ago

you can ignore the tensorflow warnings, the cpu is sufficient for inference if you use the CNN after CNMF

your ImportError is probably because you're in the caiman dir, you shouldn't import packages from within the package's dir.