Open rpeys opened 1 year ago
As an update, I got the same error working in a fresh env that I created in the following way conda create -n cuml-scanpy -c rapidsai -c nvidia -c pytorch -c conda-forge scanpy[rapids] cudatoolkit=11.3 pytorch torchvision torchaudio
.
I tried to run conda install -c rapidsai cuml
and got
(cuml-scanpy) rpeyser@hera:~$ conda install -c rapidsai cuml
Retrieving notices: ...working... done
Collecting package metadata (current_repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: /
Found conflicts! Looking for incompatible packages. failed
UnsatisfiableError: The following specifications were found to be incompatible with each other:
Output in format: Requested package -> Available versionsThe following specifications were found to be incompatible with your system:
- feature:/linux-64::__glibc==2.31=0
- cuml -> __glibc[version='>=2.17,<3.0.a0']
- cuml -> cuda-python[version='>=11.7.1,<12'] -> __glibc[version='>=2.17']
- python=3.10 -> libgcc-ng[version='>=10.3.0'] -> __glibc[version='>=2.17']
Your installed version is: 2.31
Note that strict channel priority may have removed packages required for satisfiability.
Current and recent cuML packages are compatible with CUDA 11.3.
I'm able to get conda/mamba to solve your environment creation command (with cuML included) on my system. Can you provide more information about your system? If you clone this repository and run print_env.sh
, we may be able to provide more relevant suggestions.
With that said, I'm not sure if there's a scanpy[rapids]
conda package. If this line is still current and consistent with any conda packages, it looks like it would be using a very old version of RAPIDS packages.
Thanks! You're right, scanpy does seem to use an outdated version of RAPIDS. I'll focus on installing RAPIDS standalone, in that case, which I'm still having trouble with. Here is the output of running print_env.sh
**git*** commit 1181dba55c5e76a36905fa5a7f85bd128bbe37bd (HEAD -> branch-23.04, origin/branch-23.04, origin/HEAD) Author: Corey J. NoletDate: Fri Mar 10 18:39:47 2023 -0500 Removing remaining include of `raft/distance/distance_type.hpp` (#5264) Authors: - Corey J. Nolet (https://github.com/cjnolet) Approvers: - Divye Gala (https://github.com/divyegala) - Dante Gama Dessavre (https://github.com/dantegd) URL: https://github.com/rapidsai/cuml/pull/5264 **git submodules*** ***OS Information*** DISTRIB_ID=Ubuntu DISTRIB_RELEASE=20.04 DISTRIB_CODENAME=focal DISTRIB_DESCRIPTION="Ubuntu 20.04.3 LTS" NAME="Ubuntu" VERSION="20.04.3 LTS (Focal Fossa)" ID=ubuntu ID_LIKE=debian PRETTY_NAME="Ubuntu 20.04.3 LTS" VERSION_ID="20.04" HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" VERSION_CODENAME=focal UBUNTU_CODENAME=focal Linux hera 5.4.0-110-generic #124-Ubuntu SMP Thu Apr 14 19:46:19 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux ***GPU Information*** Mon Mar 13 17:07:58 2023 +-----------------------------------------------------------------------------+ | NVIDIA-SMI 470.161.03 Driver Version: 470.161.03 CUDA Version: 11.4 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |===============================+======================+======================| | 0 NVIDIA GeForce ... Off | 00000000:3B:00.0 Off | N/A | | 23% 30C P8 15W / 250W | 2MiB / 11178MiB | 0% Default | | | | N/A | +-------------------------------+----------------------+----------------------+ | 1 NVIDIA GeForce ... Off | 00000000:5E:00.0 Off | N/A | | 35% 60C P2 121W / 250W | 9313MiB / 11178MiB | 45% Default | | | | N/A | +-------------------------------+----------------------+----------------------+ | 2 NVIDIA GeForce ... Off | 00000000:AF:00.0 Off | N/A | | 23% 33C P8 9W / 250W | 2MiB / 11178MiB | 0% Default | | | | N/A | +-------------------------------+----------------------+----------------------+ | 3 NVIDIA GeForce ... Off | 00000000:D8:00.0 Off | N/A | | 23% 34C P8 8W / 250W | 2MiB / 11178MiB | 0% Default | | | | N/A | +-------------------------------+----------------------+----------------------+ +-----------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=============================================================================| | 1 N/A N/A 984349 C python 9311MiB | +-----------------------------------------------------------------------------+ ***CPU*** Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian Address sizes: 46 bits physical, 48 bits virtual CPU(s): 48 On-line CPU(s) list: 0-47 Thread(s) per core: 2 Core(s) per socket: 12 Socket(s): 2 NUMA node(s): 2 Vendor ID: GenuineIntel CPU family: 6 Model: 85 Model name: Intel(R) Xeon(R) Gold 6126 CPU @ 2.60GHz Stepping: 4 CPU MHz: 1000.236 CPU max MHz: 3700.0000 CPU min MHz: 1000.0000 BogoMIPS: 5200.00 Virtualization: VT-x L1d cache: 768 KiB L1i cache: 768 KiB L2 cache: 24 MiB L3 cache: 38.5 MiB NUMA node0 CPU(s): 0-11,24-35 NUMA node1 CPU(s): 12-23,36-47 Vulnerability Itlb multihit: KVM: Mitigation: Split huge pages Vulnerability L1tf: Mitigation; PTE Inversion; VMX conditional cache flushes, SMT vulnerable Vulnerability Mds: Mitigation; Clear CPU buffers; SMT vulnerable Vulnerability Meltdown: Mitigation; PTI Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Vulnerability Spectre v2: Mitigation; Retpolines, IBPB conditional, IBRS_FW, STIBP conditional, RSB filling Vulnerability Srbds: Not affected Vulnerability Tsx async abort: Mitigation; Clear CPU buffers; SMT vulnerable Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single pti intel_ppin ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req pku ospke md_clear flush_l1d ***CMake*** /usr/bin/cmake cmake version 3.16.3 CMake suite maintained and supported by Kitware (kitware.com/cmake). ***g++*** /usr/bin/g++ g++ (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0 Copyright (C) 2019 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. ***nvcc*** /opt/conda/rpeyser/envs/cuml-scanpy/bin/nvcc nvcc: NVIDIA (R) Cuda compiler driver Copyright (c) 2005-2022 NVIDIA Corporation Built on Wed_Jun__8_16:49:14_PDT_2022 Cuda compilation tools, release 11.7, V11.7.99 Build cuda_11.7.r11.7/compiler.31442593_0 ***Python*** /opt/conda/rpeyser/envs/cuml-scanpy/bin/python Python 3.10.9 ***Environment Variables*** PATH : /usr/local/cuda-11.4/bin:/home/rpeyser/bin:/home/rpeyser/.local/bin:/home/rpeyser/google-cloud-sdk/bin:/opt/conda/rpeyser/envs/cuml-scanpy/bin:/opt/conda/rpeyser/condabin:/usr/local/cuda-11.4/bin:/usr/local/csail/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:.:/software/jdk1.8.0_191/bin:/home/rpeyser/software/hadoop-3.1.1/bin:/home/rpeyser/software/apache-hive-3.1.1-bin/bin:/software/jdk1.8.0_191/bin:/home/rpeyser/software/hadoop-3.1.1/bin:/home/rpeyser/software/apache-hive-3.1.1-bin/bin LD_LIBRARY_PATH : /usr/local/cuda-11.4/lib64:/usr/local/cuda-11.4/lib64: NUMBAPRO_NVVM : NUMBAPRO_LIBDEVICE : CONDA_PREFIX : /opt/conda/rpeyser/envs/cuml-scanpy PYTHON_PATH : ***conda packages*** /opt/conda/rpeyser/condabin/conda # packages in environment at /opt/conda/rpeyser/envs/cuml-scanpy: # # Name Version Build Channel _libgcc_mutex 0.1 conda_forge conda-forge _openmp_mutex 4.5 2_kmp_llvm conda-forge anndata 0.8.0 pyhd8ed1ab_1 conda-forge asttokens 2.2.1 pypi_0 pypi backcall 0.2.0 pypi_0 pypi blas 2.116 mkl conda-forge blas-devel 3.9.0 16_linux64_mkl conda-forge brotli 1.0.9 h166bdaf_8 conda-forge brotli-bin 1.0.9 h166bdaf_8 conda-forge brotlipy 0.7.0 py310h5764c6d_1005 conda-forge bzip2 1.0.8 h7f98852_4 conda-forge c-ares 1.18.1 h7f98852_0 conda-forge ca-certificates 2022.12.7 ha878542_0 conda-forge cached-property 1.5.2 hd8ed1ab_1 conda-forge cached_property 1.5.2 pyha770c72_1 conda-forge certifi 2022.12.7 pyhd8ed1ab_0 conda-forge cffi 1.15.1 py310h255011f_3 conda-forge charset-normalizer 2.1.1 pyhd8ed1ab_0 conda-forge colorama 0.4.6 pyhd8ed1ab_0 conda-forge comm 0.1.2 pypi_0 pypi contourpy 1.0.7 py310hdf3cbec_0 conda-forge cryptography 39.0.2 py310h34c0648_0 conda-forge cuda 11.7.1 0 nvidia cuda-cccl 11.7.91 0 nvidia cuda-command-line-tools 11.7.1 0 nvidia cuda-compiler 11.7.1 0 nvidia cuda-cudart 11.7.99 0 nvidia cuda-cudart-dev 11.7.99 0 nvidia cuda-cuobjdump 11.7.91 0 nvidia cuda-cupti 11.7.101 0 nvidia cuda-cuxxfilt 11.7.91 0 nvidia cuda-demo-suite 12.1.55 0 nvidia cuda-documentation 12.1.55 0 nvidia cuda-driver-dev 11.7.99 0 nvidia cuda-gdb 12.1.55 0 nvidia cuda-libraries 11.7.1 0 nvidia cuda-libraries-dev 11.7.1 0 nvidia cuda-memcheck 11.8.86 0 nvidia cuda-nsight 12.1.55 0 nvidia cuda-nsight-compute 12.1.0 0 nvidia cuda-nvcc 11.7.99 0 nvidia cuda-nvdisasm 12.1.55 0 nvidia cuda-nvml-dev 11.7.91 0 nvidia cuda-nvprof 12.1.55 0 nvidia cuda-nvprune 11.7.91 0 nvidia cuda-nvrtc 11.7.99 0 nvidia cuda-nvrtc-dev 11.7.99 0 nvidia cuda-nvtx 11.7.91 0 nvidia cuda-nvvp 12.1.55 0 nvidia cuda-runtime 11.7.1 0 nvidia cuda-sanitizer-api 12.1.55 0 nvidia cuda-toolkit 11.7.1 0 nvidia cuda-tools 11.7.1 0 nvidia cuda-visual-tools 11.7.1 0 nvidia cudatoolkit 11.3.1 ha36c431_9 nvidia cycler 0.11.0 pyhd8ed1ab_0 conda-forge debugpy 1.6.6 pypi_0 pypi decorator 5.1.1 pypi_0 pypi executing 1.2.0 pypi_0 pypi ffmpeg 4.3 hf484d3e_0 pytorch fonttools 4.39.0 py310h1fa729e_0 conda-forge freetype 2.12.1 hca18f0e_1 conda-forge gds-tools 1.6.0.25 0 nvidia gmp 6.2.1 h58526e2_0 conda-forge gnutls 3.6.13 h85f3911_1 conda-forge h5py 3.8.0 nompi_py310ha66b2ad_101 conda-forge hdf5 1.14.0 nompi_hb72d44e_103 conda-forge icu 70.1 h27087fc_0 conda-forge idna 3.4 pyhd8ed1ab_0 conda-forge importlib-metadata 6.0.0 pyha770c72_0 conda-forge importlib_metadata 6.0.0 hd8ed1ab_0 conda-forge ipykernel 6.21.3 pypi_0 pypi ipython 8.11.0 pypi_0 pypi jedi 0.18.2 pypi_0 pypi joblib 1.2.0 pyhd8ed1ab_0 conda-forge jpeg 9e h0b41bf4_3 conda-forge jupyter-client 8.0.3 pypi_0 pypi jupyter-core 5.2.0 pypi_0 pypi keyutils 1.6.1 h166bdaf_0 conda-forge kiwisolver 1.4.4 py310hbf28c38_1 conda-forge krb5 1.20.1 h81ceb04_0 conda-forge lame 3.100 h166bdaf_1003 conda-forge lcms2 2.15 hfd0df8a_0 conda-forge ld_impl_linux-64 2.40 h41732ed_0 conda-forge lerc 4.0.0 h27087fc_0 conda-forge libaec 1.0.6 hcb278e6_1 conda-forge libblas 3.9.0 16_linux64_mkl conda-forge libbrotlicommon 1.0.9 h166bdaf_8 conda-forge libbrotlidec 1.0.9 h166bdaf_8 conda-forge libbrotlienc 1.0.9 h166bdaf_8 conda-forge libcblas 3.9.0 16_linux64_mkl conda-forge libcublas 12.1.0.26 0 nvidia libcublas-dev 12.1.0.26 0 nvidia libcufft 11.0.2.4 0 nvidia libcufft-dev 11.0.2.4 0 nvidia libcufile 1.6.0.25 0 nvidia libcufile-dev 1.6.0.25 0 nvidia libcurand 10.3.2.56 0 nvidia libcurand-dev 10.3.2.56 0 nvidia libcurl 7.88.1 hdc1c0ab_0 conda-forge libcusolver 11.4.4.55 0 nvidia libcusolver-dev 11.4.4.55 0 nvidia libcusparse 12.0.2.55 0 nvidia libcusparse-dev 12.0.2.55 0 nvidia libdeflate 1.17 h0b41bf4_0 conda-forge libedit 3.1.20191231 he28a2e2_2 conda-forge libev 4.33 h516909a_1 conda-forge libffi 3.4.2 h7f98852_5 conda-forge libgcc-ng 12.2.0 h65d4601_19 conda-forge libgfortran-ng 12.2.0 h69a702a_19 conda-forge libgfortran5 12.2.0 h337968e_19 conda-forge libhwloc 2.9.0 hd6dc26d_0 conda-forge libiconv 1.17 h166bdaf_0 conda-forge liblapack 3.9.0 16_linux64_mkl conda-forge liblapacke 3.9.0 16_linux64_mkl conda-forge libllvm11 11.1.0 he0ac6c6_5 conda-forge libnghttp2 1.52.0 h61bc06f_0 conda-forge libnpp 12.0.2.50 0 nvidia libnpp-dev 12.0.2.50 0 nvidia libnsl 2.0.0 h7f98852_0 conda-forge libnvjpeg 12.1.0.39 0 nvidia libnvjpeg-dev 12.1.0.39 0 nvidia libpng 1.6.39 h753d276_0 conda-forge libsqlite 3.40.0 h753d276_0 conda-forge libssh2 1.10.0 hf14f497_3 conda-forge libstdcxx-ng 12.2.0 h46fd767_19 conda-forge libtiff 4.5.0 h6adf6a1_2 conda-forge libuuid 2.32.1 h7f98852_1000 conda-forge libwebp-base 1.2.4 h166bdaf_0 conda-forge libxcb 1.13 h7f98852_1004 conda-forge libxml2 2.10.3 h7463322_0 conda-forge libzlib 1.2.13 h166bdaf_4 conda-forge llvm-openmp 15.0.7 h0cdce71_0 conda-forge llvmlite 0.39.1 py310h58363a5_1 conda-forge matplotlib-base 3.7.1 py310he60537e_0 conda-forge matplotlib-inline 0.1.6 pypi_0 pypi mkl 2022.1.0 h84fe81f_915 conda-forge mkl-devel 2022.1.0 ha770c72_916 conda-forge mkl-include 2022.1.0 h84fe81f_915 conda-forge munkres 1.1.4 pyh9f0ad1d_0 conda-forge natsort 8.3.1 pyhd8ed1ab_0 conda-forge ncurses 6.3 h27087fc_1 conda-forge nest-asyncio 1.5.6 pypi_0 pypi nettle 3.6 he412f7d_0 conda-forge networkx 3.0 pyhd8ed1ab_0 conda-forge nsight-compute 2023.1.0.15 0 nvidia numba 0.56.4 py310ha5257ce_0 conda-forge numpy 1.23.5 py310h53a5b5f_0 conda-forge openh264 2.1.1 h780b84a_0 conda-forge openjpeg 2.5.0 hfec8fc6_2 conda-forge openssl 3.0.8 h0b41bf4_0 conda-forge packaging 23.0 pyhd8ed1ab_0 conda-forge pandas 1.5.3 py310h9b08913_0 conda-forge parso 0.8.3 pypi_0 pypi patsy 0.5.3 pyhd8ed1ab_0 conda-forge pexpect 4.8.0 pypi_0 pypi pickleshare 0.7.5 pypi_0 pypi pillow 9.4.0 py310h023d228_1 conda-forge pip 23.0.1 pyhd8ed1ab_0 conda-forge platformdirs 3.1.0 pyhd8ed1ab_0 conda-forge pooch 1.7.0 pyhd8ed1ab_0 conda-forge prompt-toolkit 3.0.38 pypi_0 pypi psutil 5.9.4 pypi_0 pypi pthread-stubs 0.4 h36c2ea0_1001 conda-forge ptyprocess 0.7.0 pypi_0 pypi pure-eval 0.2.2 pypi_0 pypi pycparser 2.21 pyhd8ed1ab_0 conda-forge pygments 2.14.0 pypi_0 pypi pynndescent 0.5.8 pyh1a96a4e_0 conda-forge pyopenssl 23.0.0 pyhd8ed1ab_0 conda-forge pyparsing 3.0.9 pyhd8ed1ab_0 conda-forge pysocks 1.7.1 pyha2e5f31_6 conda-forge python 3.10.9 he550d4f_0_cpython conda-forge python-dateutil 2.8.2 pyhd8ed1ab_0 conda-forge python_abi 3.10 3_cp310 conda-forge pytorch 1.13.1 py3.10_cuda11.7_cudnn8.5.0_0 pytorch pytorch-cuda 11.7 h67b0de4_0 pytorch pytorch-mutex 1.0 cuda pytorch pytz 2022.7.1 pyhd8ed1ab_0 conda-forge pyzmq 25.0.1 pypi_0 pypi readline 8.1.2 h0f457ee_0 conda-forge requests 2.28.2 pyhd8ed1ab_0 conda-forge scanpy 1.9.3 pyhd8ed1ab_0 conda-forge scikit-learn 1.2.2 py310h209a8ca_0 conda-forge scipy 1.10.1 py310h8deb116_0 conda-forge seaborn 0.12.2 hd8ed1ab_0 conda-forge seaborn-base 0.12.2 pyhd8ed1ab_0 conda-forge session-info 1.0.0 pyhd8ed1ab_0 conda-forge setuptools 67.6.0 pyhd8ed1ab_0 conda-forge six 1.16.0 pyh6c4a22f_0 conda-forge stack-data 0.6.2 pypi_0 pypi statsmodels 0.13.5 py310hde88566_2 conda-forge stdlib-list 0.8.0 pyhd8ed1ab_0 conda-forge tbb 2021.8.0 hf52228f_0 conda-forge threadpoolctl 3.1.0 pyh8a188c0_0 conda-forge tk 8.6.12 h27826a3_0 conda-forge torchaudio 0.13.1 py310_cu117 pytorch torchvision 0.14.1 py310_cu117 pytorch tornado 6.2 pypi_0 pypi tqdm 4.65.0 pyhd8ed1ab_1 conda-forge traitlets 5.9.0 pypi_0 pypi typing-extensions 4.5.0 hd8ed1ab_0 conda-forge typing_extensions 4.5.0 pyha770c72_0 conda-forge tzdata 2022g h191b570_0 conda-forge umap-learn 0.5.3 py310hff52083_0 conda-forge unicodedata2 15.0.0 py310h5764c6d_0 conda-forge urllib3 1.26.15 pyhd8ed1ab_0 conda-forge wcwidth 0.2.6 pypi_0 pypi wheel 0.38.4 pyhd8ed1ab_0 conda-forge xorg-libxau 1.0.9 h7f98852_0 conda-forge xorg-libxdmcp 1.1.3 h7f98852_0 conda-forge xz 5.2.6 h166bdaf_0 conda-forge zipp 3.15.0 pyhd8ed1ab_0 conda-forge zlib 1.2.13 h166bdaf_4 conda-forge zstd 1.5.2 h3eb15da_6 conda-forge
I ended up successfully installing cuml into a different conda env using pip install cuml-cu11 --extra-index-url=https://pypi.nvidia.com
. However, when I run import cuml
in python, I get the following error: cudf.errors.UnsupportedCUDAError: Detected CUDA Runtime version is 10.0. Please update your CUDA Runtime to 11.0 or above.
It is true that my server has cuda 10 installed in /usr/local/cuda-10.1/
, however, my conda environment shows that I have cuda-python version 11.8 installed. Do you know how I can get cuml to recognize the cuda toolkit version installed in my conda env, and avoid this error? Thanks!
Printing out updated output from print_env.sh
:
**git*** Not inside a git repository ***OS Information*** DISTRIB_ID=Ubuntu DISTRIB_RELEASE=20.04 DISTRIB_CODENAME=focal DISTRIB_DESCRIPTION="Ubuntu 20.04.3 LTS" NAME="Ubuntu" VERSION="20.04.3 LTS (Focal Fossa)" ID=ubuntu ID_LIKE=debian PRETTY_NAME="Ubuntu 20.04.3 LTS" VERSION_ID="20.04" HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" VERSION_CODENAME=focal UBUNTU_CODENAME=focal Linux hera 5.4.0-110-generic #124-Ubuntu SMP Thu Apr 14 19:46:19 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux ***GPU Information*** Sat Mar 18 23:31:01 2023 +-----------------------------------------------------------------------------+ | NVIDIA-SMI 470.161.03 Driver Version: 470.161.03 CUDA Version: 11.4 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |===============================+======================+======================| | 0 NVIDIA GeForce ... Off | 00000000:3B:00.0 Off | N/A | | 23% 30C P8 15W / 250W | 1353MiB / 11178MiB | 0% Default | | | | N/A | +-------------------------------+----------------------+----------------------+ | 1 NVIDIA GeForce ... Off | 00000000:5E:00.0 Off | N/A | | 34% 58C P2 115W / 250W | 9313MiB / 11178MiB | 43% Default | | | | N/A | +-------------------------------+----------------------+----------------------+ | 2 NVIDIA GeForce ... Off | 00000000:AF:00.0 Off | N/A | | 23% 33C P8 9W / 250W | 2MiB / 11178MiB | 0% Default | | | | N/A | +-------------------------------+----------------------+----------------------+ | 3 NVIDIA GeForce ... Off | 00000000:D8:00.0 Off | N/A | | 23% 33C P8 8W / 250W | 2MiB / 11178MiB | 0% Default | | | | N/A | +-------------------------------+----------------------+----------------------+ +-----------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=============================================================================| | 0 N/A N/A 462567 C ...vs/cuml-scanpy/bin/python 1351MiB | | 1 N/A N/A 984349 C python 9311MiB | +-----------------------------------------------------------------------------+ ***CPU*** Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian Address sizes: 46 bits physical, 48 bits virtual CPU(s): 48 On-line CPU(s) list: 0-47 Thread(s) per core: 2 Core(s) per socket: 12 Socket(s): 2 NUMA node(s): 2 Vendor ID: GenuineIntel CPU family: 6 Model: 85 Model name: Intel(R) Xeon(R) Gold 6126 CPU @ 2.60GHz Stepping: 4 CPU MHz: 1000.066 CPU max MHz: 3700.0000 CPU min MHz: 1000.0000 BogoMIPS: 5200.00 Virtualization: VT-x L1d cache: 768 KiB L1i cache: 768 KiB L2 cache: 24 MiB L3 cache: 38.5 MiB NUMA node0 CPU(s): 0-11,24-35 NUMA node1 CPU(s): 12-23,36-47 Vulnerability Itlb multihit: KVM: Mitigation: Split huge pages Vulnerability L1tf: Mitigation; PTE Inversion; VMX conditional cache flushes, SMT vulnerable Vulnerability Mds: Mitigation; Clear CPU buffers; SMT vulnerable Vulnerability Meltdown: Mitigation; PTI Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Vulnerability Spectre v2: Mitigation; Retpolines, IBPB conditional, IBRS_FW, STIBP conditional, RSB filling Vulnerability Srbds: Not affected Vulnerability Tsx async abort: Mitigation; Clear CPU buffers; SMT vulnerable Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single pti intel_ppin ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req pku ospke md_clear flush_l1d ***CMake*** /usr/bin/cmake cmake version 3.16.3 CMake suite maintained and supported by Kitware (kitware.com/cmake). ***g++*** /usr/bin/g++ g++ (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0 Copyright (C) 2019 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. ***nvcc*** ***Python*** /opt/conda/rpeyser/envs/scset3/bin/python Python 3.8.15 ***Environment Variables*** PATH : /home/rpeyser/bin:/home/rpeyser/.local/bin:/home/rpeyser/google-cloud-sdk/bin:/opt/conda/rpeyser/envs/scset3/bin:/opt/conda/rpeyser/condabin:/usr/local/cuda-11.4/bin:/usr/local/csail/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:.:/software/jdk1.8.0_191/bin:/home/rpeyser/software/hadoop-3.1.1/bin:/home/rpeyser/software/apache-hive-3.1.1-bin/bin LD_LIBRARY_PATH : /usr/local/cuda-11.4/lib64: NUMBAPRO_NVVM : NUMBAPRO_LIBDEVICE : CONDA_PREFIX : /opt/conda/rpeyser/envs/scset3 PYTHON_PATH : ***conda packages*** /opt/conda/rpeyser/condabin/conda # packages in environment at /opt/conda/rpeyser/envs/scset3: # # Name Version Build Channel _libgcc_mutex 0.1 conda_forge conda-forge _openmp_mutex 4.5 2_gnu conda-forge absl-py 1.3.0 pypi_0 pypi addict 2.4.0 pypi_0 pypi anndata 0.8.0 pypi_0 pypi asttokens 2.2.1 pyhd8ed1ab_0 conda-forge attrs 22.2.0 pypi_0 pypi backcall 0.2.0 pyh9f0ad1d_0 conda-forge backports 1.0 pyhd8ed1ab_3 conda-forge backports.functools_lru_cache 1.6.4 pyhd8ed1ab_0 conda-forge bzip2 1.0.8 h7f98852_4 conda-forge ca-certificates 2022.12.7 ha878542_0 conda-forge cachetools 4.2.4 pypi_0 pypi certifi 2022.12.7 pypi_0 pypi chardet 3.0.4 pypi_0 pypi charset-normalizer 2.1.1 pypi_0 pypi click 8.1.3 pypi_0 pypi cloudpickle 2.2.1 pypi_0 pypi comm 0.1.2 pyhd8ed1ab_0 conda-forge configargparse 1.5.3 pypi_0 pypi contourpy 1.0.6 pypi_0 pypi cubinlinker-cu11 0.3.0.post1 pypi_0 pypi cuda-python 11.8.1 pypi_0 pypi cudf-cu11 23.2.0 pypi_0 pypi cuml-cu11 23.2.0 pypi_0 pypi cupy-cuda11x 11.6.0 pypi_0 pypi cycler 0.11.0 pypi_0 pypi cython 0.29.21 pypi_0 pypi dash 2.8.1 pypi_0 pypi dash-core-components 2.0.0 pypi_0 pypi dash-html-components 2.0.0 pypi_0 pypi dash-table 5.0.0 pypi_0 pypi dask 2023.1.1 pypi_0 pypi dask-cuda 23.2.1 pypi_0 pypi dask-cudf-cu11 23.2.0 pypi_0 pypi debugpy 1.6.5 py38h8dc9893_0 conda-forge decorator 5.1.1 pyhd8ed1ab_0 conda-forge deepspeed 0.3.13 pypi_0 pypi distributed 2023.1.1 pypi_0 pypi entrypoints 0.4 pyhd8ed1ab_0 conda-forge executing 1.2.0 pyhd8ed1ab_0 conda-forge fastjsonschema 2.16.2 pypi_0 pypi fastrlock 0.8.1 pypi_0 pypi flask 2.2.2 pypi_0 pypi fonttools 4.38.0 pypi_0 pypi fsspec 2023.3.0 pypi_0 pypi future 0.18.2 pypi_0 pypi google-auth 1.35.0 pypi_0 pypi google-auth-oauthlib 0.4.6 pypi_0 pypi grpcio 1.51.1 pypi_0 pypi h5py 3.7.0 pypi_0 pypi heapdict 1.0.1 pypi_0 pypi idna 2.10 pypi_0 pypi igraph 0.10.4 pypi_0 pypi imageio 2.9.0 pypi_0 pypi imgaug 0.4.0 pypi_0 pypi importlib-metadata 6.0.0 pypi_0 pypi importlib-resources 5.10.2 pypi_0 pypi ipykernel 6.20.1 pyh210e3f2_0 conda-forge ipython 8.8.0 pyh41d4057_0 conda-forge ipywidgets 8.0.4 pypi_0 pypi itsdangerous 2.1.2 pypi_0 pypi jedi 0.18.2 pyhd8ed1ab_0 conda-forge jinja2 3.1.2 pypi_0 pypi joblib 1.2.0 pypi_0 pypi jsonschema 4.17.3 pypi_0 pypi jupyter_client 7.4.9 pyhd8ed1ab_0 conda-forge jupyter_core 5.1.3 py38h578d9bd_0 conda-forge jupyterlab-widgets 3.0.5 pypi_0 pypi kiwisolver 1.4.4 pypi_0 pypi ld_impl_linux-64 2.39 hcc3a1bd_1 conda-forge leidenalg 0.9.1 pypi_0 pypi libffi 3.4.2 h7f98852_5 conda-forge libgcc-ng 12.2.0 h65d4601_19 conda-forge libgomp 12.2.0 h65d4601_19 conda-forge libnsl 2.0.0 h7f98852_0 conda-forge libsodium 1.0.18 h36c2ea0_1 conda-forge libsqlite 3.40.0 h753d276_0 conda-forge libstdcxx-ng 12.2.0 h46fd767_19 conda-forge libuuid 2.32.1 h7f98852_1000 conda-forge libzlib 1.2.13 h166bdaf_4 conda-forge llvmlite 0.39.1 pypi_0 pypi locket 1.0.0 pypi_0 pypi markdown 3.4.1 pypi_0 pypi markupsafe 2.1.1 pypi_0 pypi matplotlib 3.6.2 pypi_0 pypi matplotlib-inline 0.1.6 pyhd8ed1ab_0 conda-forge msgpack 1.0.5 pypi_0 pypi natsort 8.2.0 pypi_0 pypi nbformat 5.5.0 pypi_0 pypi ncurses 6.3 h27087fc_1 conda-forge nest-asyncio 1.5.6 pyhd8ed1ab_0 conda-forge networkx 2.8.8 pypi_0 pypi ninja 1.10.0.post2 pypi_0 pypi numba 0.56.4 pypi_0 pypi numpy 1.23.5 pypi_0 pypi nvtx 0.2.5 pypi_0 pypi oauthlib 3.2.2 pypi_0 pypi open3d 0.16.0 pypi_0 pypi opencv-python 4.3.0.36 pypi_0 pypi openssl 3.0.7 h0b41bf4_1 conda-forge ordered-set 4.0.2 pypi_0 pypi packaging 22.0 pypi_0 pypi pandas 1.5.2 pypi_0 pypi parso 0.8.3 pyhd8ed1ab_0 conda-forge partd 1.3.0 pypi_0 pypi patsy 0.5.3 pypi_0 pypi pexpect 4.8.0 pyh1a96a4e_2 conda-forge pickleshare 0.7.5 py_1003 conda-forge pillow 9.4.0 pypi_0 pypi pip 22.3.1 pyhd8ed1ab_0 conda-forge pkgutil-resolve-name 1.3.10 pypi_0 pypi platformdirs 2.6.2 pyhd8ed1ab_0 conda-forge plotly 5.13.0 pypi_0 pypi prompt-toolkit 3.0.36 pyha770c72_0 conda-forge protobuf 4.21.0 pypi_0 pypi psutil 5.9.4 py38h0a891b7_0 conda-forge ptxcompiler-cu11 0.7.0.post1 pypi_0 pypi ptyprocess 0.7.0 pyhd3deb0d_0 conda-forge pure_eval 0.2.2 pyhd8ed1ab_0 conda-forge pyarrow 10.0.0 pypi_0 pypi pyasn1 0.4.8 pypi_0 pypi pyasn1-modules 0.2.8 pypi_0 pypi pygments 2.14.0 pyhd8ed1ab_0 conda-forge pylibraft-cu11 23.2.0 pypi_0 pypi pynndescent 0.5.8 pypi_0 pypi pynvml 11.4.1 pypi_0 pypi pyparsing 3.0.9 pypi_0 pypi pyquaternion 0.9.9 pypi_0 pypi pyrsistent 0.19.3 pypi_0 pypi python 3.8.15 h4a9ceb5_0_cpython conda-forge python-dateutil 2.8.2 pyhd8ed1ab_0 conda-forge python_abi 3.8 3_cp38 conda-forge pytz 2022.7 pypi_0 pypi pywavelets 1.4.1 pypi_0 pypi pyyaml 6.0 pypi_0 pypi pyzmq 25.0.0 py38he24dcef_0 conda-forge raft-dask-cu11 23.2.0 pypi_0 pypi readline 8.1.2 h0f457ee_0 conda-forge requests 2.23.0 pypi_0 pypi requests-oauthlib 1.3.0 pypi_0 pypi rmm-cu11 23.2.0 pypi_0 pypi rsa 4.9 pypi_0 pypi scanpy 1.9.2 pypi_0 pypi scikit-image 0.17.2 pypi_0 pypi scikit-learn 0.23.2 pypi_0 pypi scipy 1.10.0 pypi_0 pypi seaborn 0.12.2 pypi_0 pypi session-info 1.0.0 pypi_0 pypi setuptools 65.6.3 pyhd8ed1ab_0 conda-forge shapely 2.0.0 pypi_0 pypi six 1.14.0 pypi_0 pypi sortedcontainers 2.4.0 pypi_0 pypi stack_data 0.6.2 pyhd8ed1ab_0 conda-forge statsmodels 0.13.5 pypi_0 pypi stdlib-list 0.8.0 pypi_0 pypi tblib 1.7.0 pypi_0 pypi tenacity 8.1.0 pypi_0 pypi tensorboard 2.11.2 pypi_0 pypi tensorboard-data-server 0.6.1 pypi_0 pypi tensorboard-plugin-wit 1.8.1 pypi_0 pypi tensorboardx 1.8 pypi_0 pypi texttable 1.6.7 pypi_0 pypi threadpoolctl 3.1.0 pypi_0 pypi tifffile 2022.10.10 pypi_0 pypi tk 8.6.12 h27826a3_0 conda-forge toolz 0.12.0 pypi_0 pypi torch 1.12.1+cu113 pypi_0 pypi torchaudio 0.12.1+cu113 pypi_0 pypi torchvision 0.13.1+cu113 pypi_0 pypi tornado 6.2 py38h0a891b7_1 conda-forge tqdm 4.43.0 pypi_0 pypi traitlets 5.8.1 pyhd8ed1ab_0 conda-forge treelite 3.1.0 pypi_0 pypi treelite-runtime 3.1.0 pypi_0 pypi typing-extensions 4.4.0 hd8ed1ab_0 conda-forge typing_extensions 4.4.0 pyha770c72_0 conda-forge ucx-py-cu11 0.30.0 pypi_0 pypi umap-learn 0.5.3 pypi_0 pypi urllib3 1.25.8 pypi_0 pypi wcwidth 0.2.5 pyh9f0ad1d_2 conda-forge werkzeug 2.2.2 pypi_0 pypi wheel 0.38.4 pyhd8ed1ab_0 conda-forge widgetsnbextension 4.0.5 pypi_0 pypi xz 5.2.6 h166bdaf_0 conda-forge zeromq 4.3.4 h9c3ff4c_1 conda-forge zict 2.2.0 pypi_0 pypi zipp 3.11.0 pypi_0 pypi
Hi Rpeys,
I came across the same issue just now. Maybe you could check your CUDA runtime version by
nvcc -V
and since my cuda runtime version is indeed 10.0.
I created a new environment with python=3.10(this way, it will not mess up my system CUDA) and installed the cuda-toolkit using conda by
conda install -c "nvidia/label/cuda-{version}" cuda-toolkit
I got stuck in installing cuda toolkit for quite a while because there's another package called cudatoolkit(without the hyphen), I installed it at first, but nvcc was not installed in conda env's bin, so please do use cuda-toolkit for installation.
As for the version, I chose the same version as that shown from nvidia-smi.
afterwards, I export PATH={my conda environment path}, and the problem was solved.
I think the issue may stem from the cuda-toolkit version and hope this could help you.
I tried to install cuml using conda, but received an error. I am in a conda environment with cuda = 11.3 and glibc = 2.31. Any help diagnosing the problem would be appreciated. Is cuml compatible with cuda 11.3? It is hard for me to tell from the documentation. Thanks!
Here is the command I ran from a jupyter notebook, to conda install cuml:
Here is the output, including the error:
Here is a full description of the packages in my conda env: