search

tanaylab / metacells

Metacells - Single-cell RNA Sequencing Analysis
MIT License
92 stars 8 forks source link

pip installation of MC2 can misdirect BLAS library paths #25

Closed tzeitim closed 2 years ago

tzeitim commented 2 years ago

The pip installation of MC2 is enforcing a version of numpy that doesn't point towards a conda-installed version of OpenBLAS. I am not sure this is a MC2 issue itself and I hope it is not a stupid dance between conda and pip.

I've noticed this while trying to solve issue #24. I am still very confused there since I might be hitting a problem like the one reported on any of these: numpy #17856, numpy #19145 and numpy #14474 ... but let's not get into that since that seem to be specific to my system and belongs to issue #24. I opened this new issue as it is somehow unrelated but it might be useful in the future or you might be interested to know that this can happen.

My assumption is that one would like numpy to sort its BLAS dependencies on its own, in a way that one can control. For example, to choose a certain alternative such as MKL or a specific version of BLAS using something like conda.

Here an example using (this yaml file as a conda recipe):


conda clean --all # to remove any pre-fetched tarball and start fresh
conda env create -f mc2b.yaml # linked above
Collecting package metadata (repodata.json): done
Solving environment: done
Downloading and Extracting Packages
gxx_linux-64-11.2.0  | 24 KB     | ################################################################### | 100%
libgcc-devel_linux-6 | 3.3 MB    | ################################################################### | 100%
python_abi-3.10      | 4 KB      | ################################################################### | 100%
libgcc-ng-11.2.0     | 902 KB    | ################################################################### | 100%
python-3.10.4        | 28.6 MB   | ################################################################### | 100%
sqlite-3.38.3        | 1.5 MB    | ################################################################### | 100%
ncurses-6.3          | 1002 KB   | ################################################################### | 100%
gxx_impl_linux-64-11 | 12.4 MB   | ################################################################### | 100%
libgfortran-ng-11.2. | 23 KB     | ################################################################### | 100%
setuptools-62.1.0    | 1.3 MB    | ################################################################### | 100%
libstdcxx-ng-11.2.0  | 4.2 MB    | ################################################################### | 100%
binutils_linux-64-2. | 24 KB     | ################################################################### | 100%
openblas-0.3.20      | 10.8 MB   | ################################################################### | 100%
_libgcc_mutex-0.1    | 3 KB      | ################################################################### | 100%
libopenblas-0.3.20   | 10.1 MB   | ################################################################### | 100%
libblas-3.9.0        | 12 KB     | ################################################################### | 100%
openssl-3.0.3        | 2.9 MB    | ################################################################### | 100%
libcblas-3.9.0       | 12 KB     | ################################################################### | 100%
libstdcxx-devel_linu | 9.8 MB    | ################################################################### | 100%
liblapack-3.9.0      | 12 KB     | ################################################################### | 100%
libsanitizer-11.2.0  | 5.9 MB    | ################################################################### | 100%
libopenblas-ilp64-0. | 9.9 MB    | ################################################################### | 100%
openblas-ilp64-0.3.2 | 10.6 MB   | ################################################################### | 100%
libgfortran5-11.2.0  | 1.7 MB    | ################################################################### | 100%
gcc_impl_linux-64-11 | 49.7 MB   | ################################################################### | 100%
gcc_linux-64-11.2.0  | 25 KB     | ################################################################### | 100%
numpy-1.22.3         | 6.8 MB    | ################################################################### | 100%
_openmp_mutex-4.5    | 23 KB     | ################################################################### | 100%
libgomp-11.2.0       | 428 KB    | ################################################################### | 100%
Preparing transaction: done
Verifying transaction: done
Executing transaction: done

We can now activate the environment and verify numpy's configuration

# Since my system is a bit outdated I need to install some more modern gcc compiler 
export CONDA_BUILD=1 # to enable compilers when the conda env is activated
conda activate mc2b
INFO: activate-binutils_linux-64.sh made the following environmental changes:
+ADDR2LINE=/local/users/polivar/miniconda3/envs/mc2b/bin/x86_64-conda-linux-gnu-addr2line
+AR=/local/users/polivar/miniconda3/envs/mc2b/bin/x86_64-conda-linux-gnu-ar
+AS=/local/users/polivar/miniconda3/envs/mc2b/bin/x86_64-conda-linux-gnu-as
+CXXFILT=/local/users/polivar/miniconda3/envs/mc2b/bin/x86_64-conda-linux-gnu-c++filt
+ELFEDIT=/local/users/polivar/miniconda3/envs/mc2b/bin/x86_64-conda-linux-gnu-elfedit
+GPROF=/local/users/polivar/miniconda3/envs/mc2b/bin/x86_64-conda-linux-gnu-gprof
+LD_GOLD=/local/users/polivar/miniconda3/envs/mc2b/bin/x86_64-conda-linux-gnu-ld.gold
+LD=/local/users/polivar/miniconda3/envs/mc2b/bin/x86_64-conda-linux-gnu-ld
+NM=/local/users/polivar/miniconda3/envs/mc2b/bin/x86_64-conda-linux-gnu-nm
+OBJCOPY=/local/users/polivar/miniconda3/envs/mc2b/bin/x86_64-conda-linux-gnu-objcopy
+OBJDUMP=/local/users/polivar/miniconda3/envs/mc2b/bin/x86_64-conda-linux-gnu-objdump
+RANLIB=/local/users/polivar/miniconda3/envs/mc2b/bin/x86_64-conda-linux-gnu-ranlib
+READELF=/local/users/polivar/miniconda3/envs/mc2b/bin/x86_64-conda-linux-gnu-readelf
+SIZE=/local/users/polivar/miniconda3/envs/mc2b/bin/x86_64-conda-linux-gnu-size
+STRINGS=/local/users/polivar/miniconda3/envs/mc2b/bin/x86_64-conda-linux-gnu-strings
+STRIP=/local/users/polivar/miniconda3/envs/mc2b/bin/x86_64-conda-linux-gnu-strip
INFO: activate-gcc_linux-64.sh made the following environmental changes:
+build_alias=x86_64-conda-linux-gnu
+BUILD=x86_64-conda-linux-gnu
+CC_FOR_BUILD=/local/users/polivar/miniconda3/envs/mc2b/bin/x86_64-conda-linux-gnu-cc
+CC=/local/users/polivar/miniconda3/envs/mc2b/bin/x86_64-conda-linux-gnu-cc
+CFLAGS=-march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /include -fdebug-prefix-map==/usr/local/src/conda/- -fdebug-prefix-map==/usr/local/src/conda-prefix
+CMAKE_ARGS=-DCMAKE_AR=/local/users/polivar/miniconda3/envs/mc2b/bin/x86_64-conda-linux-gnu-ar -DCMAKE_CXX_COMPILER_AR=/local/users/polivar/miniconda3/envs/mc2b/bin/x86_64-conda-linux-gnu-gcc-ar -DCMAKE_C_COMPILER_AR=/local/users/polivar/miniconda3/envs/mc2b/bin/x86_64-conda-linux-gnu-gcc-ar -DCMAKE_RANLIB=/local/users/polivar/miniconda3/envs/mc2b/bin/x86_64-conda-linux-gnu-ranlib -DCMAKE_CXX_COMPILER_RANLIB=/local/users/polivar/miniconda3/envs/mc2b/bin/x86_64-conda-linux-gnu-gcc-ranlib -DCMAKE_C_COMPILER_RANLIB=/local/users/polivar/miniconda3/envs/mc2b/bin/x86_64-conda-linux-gnu-gcc-ranlib -DCMAKE_LINKER=/local/users/polivar/miniconda3/envs/mc2b/bin/x86_64-conda-linux-gnu-ld -DCMAKE_STRIP=/local/users/polivar/miniconda3/envs/mc2b/bin/x86_64-conda-linux-gnu-strip -DCMAKE_FIND_ROOT_PATH_MODE_PROGRAM=NEVER -DCMAKE_FIND_ROOT_PATH_MODE_LIBRARY=ONLY -DCMAKE_FIND_ROOT_PATH_MODE_INCLUDE=ONLY -DCMAKE_FIND_ROOT_PATH=;/x86_64-conda-linux-gnu/sysroot -DCMAKE_INSTALL_PREFIX= -DCMAKE_INSTALL_LIBDIR=lib -DCMAKE_PROGRAM_PATH=/bin;/bin
+CMAKE_PREFIX_PATH=:/local/users/polivar/miniconda3/envs/mc2b/x86_64-conda-linux-gnu/sysroot/usr
+CONDA_BUILD_SYSROOT=/local/users/polivar/miniconda3/envs/mc2b/x86_64-conda-linux-gnu/sysroot
+_CONDA_PYTHON_SYSCONFIGDATA_NAME=_sysconfigdata_x86_64_conda_cos6_linux_gnu
+CONDA_TOOLCHAIN_BUILD=x86_64-conda-linux-gnu
+CONDA_TOOLCHAIN_HOST=x86_64-conda-linux-gnu
+CPPFLAGS=-DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /include
+CPP=/local/users/polivar/miniconda3/envs/mc2b/bin/x86_64-conda-linux-gnu-cpp
+DEBUG_CFLAGS=-march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-all -fno-plt -Og -g -Wall -Wextra -fvar-tracking-assignments -ffunction-sections -pipe -isystem /include -fdebug-prefix-map==/usr/local/src/conda/- -fdebug-prefix-map==/usr/local/src/conda-prefix
+DEBUG_CPPFLAGS=-D_DEBUG -D_FORTIFY_SOURCE=2 -Og -isystem /include
+GCC_AR=/local/users/polivar/miniconda3/envs/mc2b/bin/x86_64-conda-linux-gnu-gcc-ar
+GCC=/local/users/polivar/miniconda3/envs/mc2b/bin/x86_64-conda-linux-gnu-gcc
+GCC_NM=/local/users/polivar/miniconda3/envs/mc2b/bin/x86_64-conda-linux-gnu-gcc-nm
+GCC_RANLIB=/local/users/polivar/miniconda3/envs/mc2b/bin/x86_64-conda-linux-gnu-gcc-ranlib
+host_alias=x86_64-conda-linux-gnu
+HOST=x86_64-conda-linux-gnu
+LDFLAGS=-Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,--allow-shlib-undefined -Wl,-rpath,/lib -Wl,-rpath-link,/lib -L/lib
INFO: activate-gxx_linux-64.sh made the following environmental changes:
+CXXFLAGS=-fvisibility-inlines-hidden -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /include -fdebug-prefix-map==/usr/local/src/conda/- -fdebug-prefix-map==/usr/local/src/conda-prefix
+CXX_FOR_BUILD=/local/users/polivar/miniconda3/envs/mc2b/bin/x86_64-conda-linux-gnu-c++
+CXX=/local/users/polivar/miniconda3/envs/mc2b/bin/x86_64-conda-linux-gnu-c++
+DEBUG_CXXFLAGS=-fvisibility-inlines-hidden -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-all -fno-plt -Og -g -Wall -Wextra -fvar-tracking-assignments -ffunction-sections -pipe -isystem /include -fdebug-prefix-map==/usr/local/src/conda/- -fdebug-prefix-map==/usr/local/src/conda-prefix
+GXX=/local/users/polivar/miniconda3/envs/mc2b/bin/x86_64-conda-linux-gnu-g++

python

Python 3.10.4 | packaged by conda-forge | (main, Mar 24 2022, 17:39:04) [GCC 10.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy as np
>>> np.show_config()
blas_info:
    libraries = ['cblas', 'blas', 'cblas', 'blas']
    library_dirs = ['/local/users/polivar/miniconda3/envs/mc2b/lib']
    include_dirs = ['/local/users/polivar/miniconda3/envs/mc2b/include']
    language = c
    define_macros = [('HAVE_CBLAS', None)]
blas_opt_info:
    define_macros = [('NO_ATLAS_INFO', 1), ('HAVE_CBLAS', None)]
    libraries = ['cblas', 'blas', 'cblas', 'blas']
    library_dirs = ['/local/users/polivar/miniconda3/envs/mc2b/lib']
    include_dirs = ['/local/users/polivar/miniconda3/envs/mc2b/include']
    language = c
lapack_info:
    libraries = ['lapack', 'blas', 'lapack', 'blas']
    library_dirs = ['/local/users/polivar/miniconda3/envs/mc2b/lib']
    language = f77
lapack_opt_info:
    libraries = ['lapack', 'blas', 'lapack', 'blas', 'cblas', 'blas', 'cblas', 'blas']
    library_dirs = ['/local/users/polivar/miniconda3/envs/mc2b/lib']
    language = c
    define_macros = [('NO_ATLAS_INFO', 1), ('HAVE_CBLAS', None)]
    include_dirs = ['/local/users/polivar/miniconda3/envs/mc2b/include']
Supported SIMD extensions in this NumPy install:
    baseline = SSE,SSE2,SSE3
    found = SSSE3,SSE41,POPCNT,SSE42,AVX,F16C,FMA3,AVX2
    not found = AVX512F,AVX512CD,AVX512_KNL,AVX512_KNM,AVX512_SKX,AVX512_CLX,AVX512_CNL,AVX512_ICL
>>>

We can see that this is doing what we would expect: numpy has found the correct libraries installed by conda.

Let's now install MC2 using pip.

pip cache purge # just in case
pip install metacells
Collecting metacells
  Downloading metacells-0.8.0.tar.gz (320 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 320.9/320.9 KB 4.7 MB/s eta 0:00:00
  Preparing metadata (setup.py) ... done
Collecting cvxpy
  Downloading cvxpy-1.2.0.tar.gz (1.3 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.3/1.3 MB 34.0 MB/s eta 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Collecting importlib-metadata
  Downloading importlib_metadata-4.11.3-py3-none-any.whl (18 kB)
Requirement already satisfied: numpy in /local/users/polivar/miniconda3/envs/mc2b/lib/python3.10/site-packages (from metacells) (1.22.3)
Collecting pandas
  Downloading pandas-1.4.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (11.7 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 11.7/11.7 MB 59.1 MB/s eta 0:00:00
Collecting psutil
  Downloading psutil-5.9.0-cp310-cp310-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (281 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 281.4/281.4 KB 48.7 MB/s eta 0:00:00
Collecting python-igraph
  Downloading python_igraph-0.9.10-py3-none-any.whl (9.1 kB)
Collecting pyyaml
  Downloading PyYAML-6.0-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (682 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 682.2/682.2 KB 54.2 MB/s eta 0:00:00
Collecting scanpy
  Downloading scanpy-1.9.1-py3-none-any.whl (2.0 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.0/2.0 MB 64.4 MB/s eta 0:00:00
Collecting scipy
  Using cached scipy-1.8.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (42.3 MB)
Collecting threadpoolctl
  Downloading threadpoolctl-3.1.0-py3-none-any.whl (14 kB)
Collecting umap-learn
  Downloading umap-learn-0.5.3.tar.gz (88 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 88.2/88.2 KB 14.2 MB/s eta 0:00:00
  Preparing metadata (setup.py) ... done
Collecting osqp>=0.4.1
  Downloading osqp-0.6.2.post5-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (298 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 298.1/298.1 KB 49.0 MB/s eta 0:00:00
Collecting ecos>=2
  Downloading ecos-2.0.10-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl (190 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 190.1/190.1 KB 28.5 MB/s eta 0:00:00
Collecting scs>=1.1.6
  Downloading scs-3.2.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (10.7 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 10.7/10.7 MB 72.3 MB/s eta 0:00:00
Collecting zipp>=0.5
  Downloading zipp-3.8.0-py3-none-any.whl (5.4 kB)
Collecting pytz>=2020.1
  Downloading pytz-2022.1-py2.py3-none-any.whl (503 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 503.5/503.5 KB 53.2 MB/s eta 0:00:00
Collecting python-dateutil>=2.8.1
  Downloading python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 247.7/247.7 KB 44.5 MB/s eta 0:00:00
Collecting igraph==0.9.10
  Downloading igraph-0.9.10-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.2 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.2/3.2 MB 65.6 MB/s eta 0:00:00
Collecting texttable>=1.6.2
  Downloading texttable-1.6.4-py2.py3-none-any.whl (10 kB)
Collecting packaging
  Downloading packaging-21.3-py3-none-any.whl (40 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 40.8/40.8 KB 20.3 MB/s eta 0:00:00
Collecting h5py>=3
  Downloading h5py-3.6.0-cp310-cp310-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (4.5 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.5/4.5 MB 65.3 MB/s eta 0:00:00
Collecting networkx>=2.3
  Downloading networkx-2.8-py3-none-any.whl (2.0 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.0/2.0 MB 63.7 MB/s eta 0:00:00
Collecting tqdm
  Downloading tqdm-4.64.0-py2.py3-none-any.whl (78 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 78.4/78.4 KB 21.7 MB/s eta 0:00:00
Collecting numba>=0.41.0
  Downloading numba-0.55.1-1-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (3.3 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.3/3.3 MB 62.1 MB/s eta 0:00:00
Collecting session-info
  Downloading session_info-1.0.0.tar.gz (24 kB)
  Preparing metadata (setup.py) ... done
Collecting seaborn
  Downloading seaborn-0.11.2-py3-none-any.whl (292 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 292.8/292.8 KB 49.2 MB/s eta 0:00:00
Collecting patsy
  Downloading patsy-0.5.2-py2.py3-none-any.whl (233 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 233.7/233.7 KB 39.5 MB/s eta 0:00:00
Collecting joblib
  Downloading joblib-1.1.0-py2.py3-none-any.whl (306 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 307.0/307.0 KB 43.2 MB/s eta 0:00:00
Collecting statsmodels>=0.10.0rc2
  Downloading statsmodels-0.13.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (10.0 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 10.0/10.0 MB 48.9 MB/s eta 0:00:00
Collecting natsort
  Downloading natsort-8.1.0-py3-none-any.whl (37 kB)
Collecting anndata>=0.7.4
  Downloading anndata-0.8.0-py3-none-any.whl (96 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 96.1/96.1 KB 31.8 MB/s eta 0:00:00
Collecting matplotlib>=3.4
  Downloading matplotlib-3.5.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (11.9 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 11.9/11.9 MB 65.8 MB/s eta 0:00:00
Collecting scikit-learn>=0.22
  Downloading scikit_learn-1.0.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (26.5 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 26.5/26.5 MB 48.0 MB/s eta 0:00:00
Collecting pynndescent>=0.5
  Downloading pynndescent-0.5.6.tar.gz (1.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.1/1.1 MB 64.6 MB/s eta 0:00:00
  Preparing metadata (setup.py) ... done
Collecting cycler>=0.10
  Downloading cycler-0.11.0-py3-none-any.whl (6.4 kB)
Collecting pyparsing>=2.2.1
  Downloading pyparsing-3.0.8-py3-none-any.whl (98 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 98.5/98.5 KB 36.7 MB/s eta 0:00:00
Collecting pillow>=6.2.0
  Downloading Pillow-9.1.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.3 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.3/4.3 MB 46.3 MB/s eta 0:00:00
Collecting kiwisolver>=1.0.1
  Downloading kiwisolver-1.4.2-cp310-cp310-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (1.6 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.6/1.6 MB 62.8 MB/s eta 0:00:00
Collecting fonttools>=4.22.0
  Downloading fonttools-4.33.3-py3-none-any.whl (930 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 930.9/930.9 KB 56.2 MB/s eta 0:00:00
Collecting numpy
  Using cached numpy-1.21.6-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (15.9 MB)
Collecting llvmlite<0.39,>=0.38.0rc1
  Downloading llvmlite-0.38.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (34.5 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 34.5/34.5 MB 36.3 MB/s eta 0:00:00
Requirement already satisfied: setuptools in /local/users/polivar/miniconda3/envs/mc2b/lib/python3.10/site-packages (from numba>=0.41.0->scanpy->metacells) (62.1.0)
Collecting qdldl
  Downloading qdldl-0.1.5.post2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.0 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.0/1.0 MB 54.4 MB/s eta 0:00:00
Collecting six>=1.5
  Downloading six-1.16.0-py2.py3-none-any.whl (11 kB)
Collecting stdlib_list
  Downloading stdlib_list-0.8.0-py3-none-any.whl (63 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 63.5/63.5 KB 19.8 MB/s eta 0:00:00
Building wheels for collected packages: metacells, cvxpy, umap-learn, pynndescent, session-info
  Building wheel for metacells (setup.py) ... done
  Created wheel for metacells: filename=metacells-0.8.0-cp310-cp310-linux_x86_64.whl size=4193910 sha256=e125a22ea8eb8dcece27cb1e8ea5708ba3c7470083ee9938538621c2ec2cdd5a
  Stored in directory: /home/polivar/.cache/pip/wheels/90/42/64/c769466188ea53f882e465a330147f71daeba6af17cf1e4aa8
  Building wheel for cvxpy (pyproject.toml) ... done
  Created wheel for cvxpy: filename=cvxpy-1.2.0-cp310-cp310-linux_x86_64.whl size=874821 sha256=0de00af65d2217d7a3f9ec0e46d4e11715b6ed29375d05aa1c10595be61cacf8
  Stored in directory: /home/polivar/.cache/pip/wheels/ab/44/e2/fa919074dc56be927a815b0c8473f7e963ce34a674fff1b795
  Building wheel for umap-learn (setup.py) ... done
  Created wheel for umap-learn: filename=umap_learn-0.5.3-py3-none-any.whl size=82829 sha256=2ce819baeee5ea24836d99b1f2e28706d4d6047a0c10b0b3695fe4c63e38578f
  Stored in directory: /home/polivar/.cache/pip/wheels/a0/e8/c6/a37ea663620bd5200ea1ba0907ab3c217042c1d035ef606acc
  Building wheel for pynndescent (setup.py) ... done
  Created wheel for pynndescent: filename=pynndescent-0.5.6-py3-none-any.whl size=53943 sha256=04874f8f72b08399cdf976116b51310e1a09c6bc32aa254eca7985a092339874
  Stored in directory: /home/polivar/.cache/pip/wheels/03/f6/7a/4d3cf002c42b28ecf83431c8b6563c6a4f7f052d8c64e0dc45
  Building wheel for session-info (setup.py) ... done
  Created wheel for session-info: filename=session_info-1.0.0-py3-none-any.whl size=8048 sha256=0429fb8fa0c12733480e11ba7a53616e1815019e5351d137ce61c69736b33969
  Stored in directory: /home/polivar/.cache/pip/wheels/6a/aa/b9/eb5d4031476ec10802795b97ccf937b9bd998d68a9b268765a
Successfully built metacells cvxpy umap-learn pynndescent session-info
Installing collected packages: texttable, stdlib_list, pytz, zipp, tqdm, threadpoolctl, six, session-info, pyyaml, pyparsing, psutil, pillow, numpy, networkx, natsort, llvmlite, kiwisolver, joblib, igraph, fonttools, cycler, scipy, python-igraph, python-dateutil, patsy, packaging, numba, importlib-metadata, h5py, scs, scikit-learn, qdldl, pandas, matplotlib, ecos, statsmodels, seaborn, pynndescent, osqp, anndata, umap-learn, cvxpy, scanpy, metacells
  Attempting uninstall: numpy
    Found existing installation: numpy 1.22.3
    Uninstalling numpy-1.22.3:
      Successfully uninstalled numpy-1.22.3
Successfully installed anndata-0.8.0 cvxpy-1.2.0 cycler-0.11.0 ecos-2.0.10 fonttools-4.33.3 h5py-3.6.0 igraph-0.9.10 importlib-metadata-4.11.3 joblib-1.1.0 kiwisolver-1.4.2 llvmlite-0.38.0 matplotlib-3.5.2 metacells-0.8.0 natsort-8.1.0 networkx-2.8 numba-0.55.1 numpy-1.21.6 osqp-0.6.2.post5 packaging-21.3 pandas-1.4.2 patsy-0.5.2 pillow-9.1.0 psutil-5.9.0 pynndescent-0.5.6 pyparsing-3.0.8 python-dateutil-2.8.2 python-igraph-0.9.10 pytz-2022.1 pyyaml-6.0 qdldl-0.1.5.post2 scanpy-1.9.1 scikit-learn-1.0.2 scipy-1.8.0 scs-3.2.0 seaborn-0.11.2 session-info-1.0.0 six-1.16.0 statsmodels-0.13.2 stdlib_list-0.8.0 texttable-1.6.4 threadpoolctl-3.1.0 tqdm-4.64.0 umap-learn-0.5.3 zipp-3.8.0

We can observe that numpy was downgraded in this step - which should be fine. The problem is that in this pip transaction numpy lost the communication with the conda environment in terms of where to look for the BLAS libraries that conda kindly installed, and I chose.

python
Python 3.10.4 | packaged by conda-forge | (main, Mar 24 2022, 17:39:04) [GCC 10.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy as np
>>> np.show_config()
blas_mkl_info:
  NOT AVAILABLE
blis_info:
  NOT AVAILABLE
openblas_info:
    libraries = ['openblas', 'openblas']
    library_dirs = ['/usr/local/lib']
    language = c
    define_macros = [('HAVE_CBLAS', None)]
    runtime_library_dirs = ['/usr/local/lib']
blas_opt_info:
    libraries = ['openblas', 'openblas']
    library_dirs = ['/usr/local/lib']
    language = c
    define_macros = [('HAVE_CBLAS', None)]
    runtime_library_dirs = ['/usr/local/lib']
lapack_mkl_info:
  NOT AVAILABLE
openblas_lapack_info:
    libraries = ['openblas', 'openblas']
    library_dirs = ['/usr/local/lib']
    language = c
    define_macros = [('HAVE_CBLAS', None)]
    runtime_library_dirs = ['/usr/local/lib']
lapack_opt_info:
    libraries = ['openblas', 'openblas']
    library_dirs = ['/usr/local/lib']
    language = c
    define_macros = [('HAVE_CBLAS', None)]
    runtime_library_dirs = ['/usr/local/lib']
Supported SIMD extensions in this NumPy install:
    baseline = SSE,SSE2,SSE3
    found = SSSE3,SSE41,POPCNT,SSE42,AVX,F16C,FMA3,AVX2
    not found = AVX512F,AVX512CD,AVX512_KNL,AVX512_KNM,AVX512_SKX,AVX512_CLX,AVX512_CNL,AVX512_ICL

If I repeat all the steps shown above but anticipating the version that metacells enforces via pip (numpy-1.21.6), then the misdirection of the BLAS libraries doesn't happen. (I show now just the end result for simplicity).

Here we see that MC2 could be installed and numpy points to the 'right' (conda's) BLAS libraries.

Python 3.10.4 | packaged by conda-forge | (main, Mar 24 2022, 17:38:57) [GCC 10.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy as np
>>> np.show_config()
blas_info:
    libraries = ['cblas', 'blas', 'cblas', 'blas']
    library_dirs = ['/local/users/polivar/miniconda3/envs/mc2b/lib']
    include_dirs = ['/local/users/polivar/miniconda3/envs/mc2b/include']
    language = c
    define_macros = [('HAVE_CBLAS', None)]
blas_opt_info:
    define_macros = [('NO_ATLAS_INFO', 1), ('HAVE_CBLAS', None)]
    libraries = ['cblas', 'blas', 'cblas', 'blas']
    library_dirs = ['/local/users/polivar/miniconda3/envs/mc2b/lib']
    include_dirs = ['/local/users/polivar/miniconda3/envs/mc2b/include']
    language = c
lapack_info:
    libraries = ['lapack', 'blas', 'lapack', 'blas']
    library_dirs = ['/local/users/polivar/miniconda3/envs/mc2b/lib']
    language = f77
lapack_opt_info:
    libraries = ['lapack', 'blas', 'lapack', 'blas', 'cblas', 'blas', 'cblas', 'blas']
    library_dirs = ['/local/users/polivar/miniconda3/envs/mc2b/lib']
    language = c
    define_macros = [('NO_ATLAS_INFO', 1), ('HAVE_CBLAS', None)]
    include_dirs = ['/local/users/polivar/miniconda3/envs/mc2b/include']
Supported SIMD extensions in this NumPy install:
    baseline = SSE,SSE2,SSE3
    found = SSSE3,SSE41,POPCNT,SSE42,AVX,F16C,FMA3,AVX2
    not found = AVX512F,AVX512CD,AVX512_KNL,AVX512_KNM,AVX512_SKX,AVX512_CLX,AVX512_CNL,AVX512_ICL
>>> import metacells as mc
mc.utilities.parallel.get_processors_count()
56

I guess that for users who depend on conda and a scpecific version of BLAS this will be useful to know.

Unfortunately this has not yet solved issue #24

Thanks for your time

orenbenkiki commented 2 years ago

MC2 just requires "numpy", there's no version restriction anywhere in the MC2 package itself. I'm afraid this is actually "a stupid dance between conda and pip". This has been known to happen :-)