CUDA Setup failed despite GPU being available.

[bryanhughes]

System Info

$ uname -a
Linux mimzy-jetson 5.15.122-tegra #1 SMP PREEMPT Mon Dec 18 21:24:25 PST 2023 aarch64 aarch64 aarch64 GNU/Linux

$ python --version
Python 3.10.12

Hardware: Nvidia Jetson Orin AGX with Jetpack 6 and cuda 12.2

Reproduction

$ pip install bitsandbytes

Expected behavior

To install, but go the following:

$ pip install bitsandbytes
Collecting bitsandbytes
  Using cached bitsandbytes-0.42.0-py3-none-any.whl.metadata (9.9 kB)
Requirement already satisfied: scipy in ./mimzy-env/lib/python3.10/site-packages (from bitsandbytes) (1.12.0)
Requirement already satisfied: numpy<1.29.0,>=1.22.4 in /home/bryan/.local/lib/python3.10/site-packages (from scipy->bitsandbytes) (1.26.4)
Using cached bitsandbytes-0.42.0-py3-none-any.whl (105.0 MB)
Installing collected packages: bitsandbytes
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
llava 1.2.2.post1 requires scikit-learn==1.2.2, but you have scikit-learn 1.4.2 which is incompatible.
llava 1.2.2.post1 requires torch==2.1.2, but you have torch 2.2.0a0+6a974be which is incompatible.
Successfully installed bitsandbytes-0.42.0

(mimzy-env) bryan@mimzy-jetson:~/git/mimzy$ python -m bitsandbytes
False

===================================BUG REPORT===================================
/mnt/nvme/git/mimzy/mimzy-env/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:167: UserWarning: Welcome to bitsandbytes. For bug reports, please run

python -m bitsandbytes

  warn(msg)
================================================================================
/mnt/nvme/git/mimzy/mimzy-env/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:167: UserWarning: /usr/local/lib did not contain ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] as expected! Searching further paths...
  warn(msg)
CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching in backup paths...
DEBUG: Possible options found for libcudart.so: {PosixPath('/usr/local/cuda/lib64/libcudart.so')}
CUDA SETUP: PyTorch settings found: CUDA_VERSION=122, Highest Compute Capability: 8.7.
CUDA SETUP: To manually override the PyTorch CUDA version please see:https://github.com/TimDettmers/bitsandbytes/blob/main/how_to_use_nonpytorch_cuda.md
CUDA SETUP: Loading binary /mnt/nvme/git/mimzy/mimzy-env/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cuda122.so...
/mnt/nvme/git/mimzy/mimzy-env/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cuda122.so: cannot open shared object file: No such file or directory
CUDA SETUP: Something unexpected happened. Please compile from source:
git clone https://github.com/TimDettmers/bitsandbytes.git
cd bitsandbytes
CUDA_VERSION=122
python setup.py install
Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 187, in _run_module_as_main
    mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
  File "/usr/lib/python3.10/runpy.py", line 146, in _get_module_details
    return _get_module_details(pkg_main_name, error)
  File "/usr/lib/python3.10/runpy.py", line 110, in _get_module_details
    __import__(pkg_name)
  File "/mnt/nvme/git/mimzy/mimzy-env/lib/python3.10/site-packages/bitsandbytes/__init__.py", line 6, in <module>
    from . import cuda_setup, utils, research
  File "/mnt/nvme/git/mimzy/mimzy-env/lib/python3.10/site-packages/bitsandbytes/research/__init__.py", line 1, in <module>
    from . import nn
  File "/mnt/nvme/git/mimzy/mimzy-env/lib/python3.10/site-packages/bitsandbytes/research/nn/__init__.py", line 1, in <module>
    from .modules import LinearFP8Mixed, LinearFP8Global
  File "/mnt/nvme/git/mimzy/mimzy-env/lib/python3.10/site-packages/bitsandbytes/research/nn/modules.py", line 8, in <module>
    from bitsandbytes.optim import GlobalOptimManager
  File "/mnt/nvme/git/mimzy/mimzy-env/lib/python3.10/site-packages/bitsandbytes/optim/__init__.py", line 6, in <module>
    from bitsandbytes.cextension import COMPILED_WITH_CUDA
  File "/mnt/nvme/git/mimzy/mimzy-env/lib/python3.10/site-packages/bitsandbytes/cextension.py", line 20, in <module>
    raise RuntimeError('''
RuntimeError:
        CUDA Setup failed despite GPU being available. Please run the following command to get more information:

        python -m bitsandbytes

        Inspect the output of the command and see if you can locate CUDA libraries. You might need to add them
        to your LD_LIBRARY_PATH. If you suspect a bug, please take the information from python -m bitsandbytes
        and open an issue at: https://github.com/TimDettmers/bitsandbytes/issues

Here is the path to he shared library in question:

$ ll /usr/local/cuda/lib64/libcudart.so
lrwxrwxrwx 1 root root 15 Aug 15  2023 /usr/local/cuda/lib64/libcudart.so -> libcudart.so.12

Here are my environment variables:

$ echo $LD_LIBRARY_PATH
/usr/local/lib:/usr/local/cuda/lib64:/usr/local/lib
$ echo $CUDA_HOME
/usr/local/cuda

[matthewdouglas]

Hi @bryanhughes,

Despite the wheel's tag, bitsandbytes-0.42.0-py3-none-any.whl, it is only built for x86-64 platforms. There has not been an aarch64 release (yet).

With that said, I'm personally interested in supporting an aarch64 build, but have limited hardware access for testing. I can keep you posted on this issue if you're interested in helping to test an aarch64/sm87 build.

[bryanhughes]

Yes please. Is there anything special about getting it built on the aarch64 device. I obviously have one and it you can give me direction, I am happy to try to get it building and contribute back a PR.

Otherwise I am happy to wait. In the meantime, can I just build the project from scratch on my system?

[yurisalesc]

Same problem here:

Copyright (c) 2005-2024 NVIDIA Corporation
Built on Thu_Mar_28_02:18:24_PDT_2024
Cuda compilation tools, release 12.4, V12.4.131
Build cuda_12.4.r12.4/compiler.34097967_0

NVIDIA Corporation GP107M [GeForce MX350]

[matthewdouglas]

@yurisalesc I'm not sure that's the same issue; as far as I know MX350 isn't found in aarch64 devices. Can you open a separate issue with more information?

@bryanhughes It's mostly some SIMD code and build/CI/distribution changes needed for CUDA+aarch64. I suspect that it may build and run just fine, but if not there might be some changes needed in include/Portable.h and/or include/SIMD.h.