bitsandbytes doesn't work

[xpang-sf]

Hi, I am trying to install: https://github.com/tloen/alpaca-lora and when I run: CUDA_VERSION=110 make cuda110 /bitsandbytes/csrc/kernels.cu:20:10: fatal error: cuda/pipeline: No such file or directory

include <cuda/pipeline>

      ^~~~~~~~~~~~~~~

compilation terminated.

Can you please let me know how to fix this? Maybe it is a simple question for you . Thank you and look forward to hearing from you soon.

XL

[xpang-sf]

add more information:

sh-4.2$ python -m bitsandbytes

===================================BUG REPORT=================================== Welcome to bitsandbytes. For bug reports, please run

python -m bitsandbytes

and submit this information together with your error trace to: https://github.com/TimDettmers/bitsandbytes/issues

bin /home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cuda110.so False /home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: /home/ec2-user/anaconda3/envs/JupyterSystemEnv did not contain ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] as expected! Searching further paths... warn(msg) /home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: /opt/amazon/efa/lib64:/opt/amazon/openmpi/lib64:/usr/local/lib:/usr/lib:/lib:/opt/amazon/efa/lib64:/opt/amazon/openmpi/lib64:/usr/local/lib:/usr/lib:/lib: did not contain ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] as expected! Searching further paths... warn(msg) /home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('8443'), PosixPath('//ip-172-16-79-80.us-west-2.compute.internal'), PosixPath('https')} warn(msg) CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching in backup paths... /home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: Found duplicate ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] files: {PosixPath('/usr/local/cuda/lib64/libcudart.so.11.0'), PosixPath('/usr/local/cuda/lib64/libcudart.so')}.. We'll flip a coin and try one of these, in order to fail forward. Either way, this might cause trouble in the future: If you get CUDA error: invalid device function errors, the above might be the cause and the solution is to make sure only one ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] in the paths that we search based on your env. warn(msg) CUDA SETUP: CUDA runtime path found: /usr/local/cuda/lib64/libcudart.so.11.0 CUDA SETUP: Highest compute capability among GPUs detected: 8.6 CUDA SETUP: Detected CUDA version 110 CUDA SETUP: Required library version not found: libbitsandbytes_cuda110.so. Maybe you need to compile it from source? CUDA SETUP: Defaulting to libbitsandbytes_cpu.so...

================================================ERROR===================================== CUDA SETUP: CUDA detection failed! Possible reasons:

1. CUDA driver not installed
2. CUDA not installed
3. You have multiple conflicting CUDA libraries

4. Required library not pre-compiled for this bitsandbytes release! CUDA SETUP: If you compiled from source, try again with make CUDA_VERSION=DETECTED_CUDA_VERSION for example, make CUDA_VERSION=113. CUDA SETUP: The CUDA version for the compile might depend on your conda install. Inspect CUDA version via conda list | grep cuda.

CUDA SETUP: Something unexpected happened. Please compile from source: git clone git@github.com:TimDettmers/bitsandbytes.git cd bitsandbytes CUDA_VERSION=110 make cuda110 python setup.py install CUDA SETUP: Setup Failed! Traceback (most recent call last): File "/home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.10/runpy.py", line 187, in _run_module_as_main mod_name, mod_spec, code = _get_module_details(mod_name, _Error) File "/home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.10/runpy.py", line 146, in _get_module_details return _get_module_details(pkg_main_name, error) File "/home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.10/runpy.py", line 110, in _get_module_details import(pkg_name) File "/home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.10/site-packages/bitsandbytes/init.py", line 6, in from . import cuda_setup, utils, research File "/home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.10/site-packages/bitsandbytes/research/init.py", line 1, in from . import nn File "/home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.10/site-packages/bitsandbytes/research/nn/init.py", line 1, in from .modules import LinearFP8Mixed, LinearFP8Global File "/home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.10/site-packages/bitsandbytes/research/nn/modules.py", line 8, in from bitsandbytes.optim import GlobalOptimManager File "/home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.10/site-packages/bitsandbytes/optim/init.py", line 6, in from bitsandbytes.cextension import COMPILED_WITH_CUDA File "/home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.10/site-packages/bitsandbytes/cextension.py", line 20, in raise RuntimeError(''' RuntimeError: CUDA Setup failed despite GPU being available. Please run the following command to get more information:

    python -m bitsandbytes

    Inspect the output of the command and see if you can locate CUDA libraries. You might need to add them
    to your LD_LIBRARY_PATH. If you suspect a bug, please take the information from python -m bitsandbytes
    and open an issue at: https://github.com/TimDettmers/bitsandbytes/issues

[jllllll]

Wheel does not contain a binary for CUDA 11.0. This presumably due to it being incompatible with new code.

[xpang-sf]

Thank you. Can please let me know what steps I can do to make the code compatible with the environment? For instance, how to upgrade cuda or how to change code to make it compatible? thank you so much.

[cwpeng-cn]

I also encountered the same problem with cuda11.0, and it seems to be caused by a bug in the CUDA library (https://github.com/NVIDIA/libcudacxx/issues/22). I guess the only solution is to upgrade the CUDA version.

[jllllll]

Thank you. Can please let me know what steps I can do to make the code compatible with the environment? For instance, how to upgrade cuda or how to change code to make it compatible? thank you so much.

The new CUDA code is used fairly extensively in the new bitsandbytes version. If all you are wanting to do is run alpaca-lora, then it should be fine to use the previous version: pip install bitsandbytes==0.38.1

[xpang-sf]

Thank you so much for all of these useful information, I will take a look and see whether I can run it.

[AlionSSS]

Thank you. Can please let me know what steps I can do to make the code compatible with the environment? For instance, how to upgrade cuda or how to change code to make it compatible? thank you so much.

The new CUDA code is used fairly extensively in the new bitsandbytes version. If all you are wanting to do is run alpaca-lora, then it should be fine to use the previous version: pip install bitsandbytes==0.38.1

There is a new OOM bug about bitsandbytes 0.38.x，so it may not work. New OOM bug introduced in bitsandbytes 0.38.x? #324

I've tried switching to bitsandbytes 0.38.1, but it gets Cuda OOM.

[xpang-sf]

Hi, just now I tried pip install bitsandbytes==0.38.1 (uninstalled bitsandbytes-0.39.0) , it works

[AlexiaJM]

I have the same bug, only 0.38.1 works, but I need 0.39.0 to be able to use https://github.com/artidoro/qlora 4 bit quantization.

In my case I have cuda 11.4.

[xpang-sf]

may i ask: what's the reason that you need to use https://github.com/artidoro/qlora 4 bit quantization because of resource limitation? thanks

[AlexiaJM]

@xpang-sf it allows fine-tuning LLaMA-65B on a single gpu in under 24h and I want to experiment with it. Have you been using https://github.com/tloen/alpaca-lora? I didn't know this one, it's my first time with LLMs. Does it have automated metrics calculation and is performance close to QLoRA LLaMA-65B?

[xpang-sf]

Hi, Alexia, I see, thanks for letting me know. yes, i started to use this: https://github.com/tloen/alpaca-lora, it can fit for 24GB GPU memory. Alpaca is fine-tuned on 7B LLaMA model , not 65B. I don't think it can has performance close to QLoRA LLaMA-65B . Once i do some tests, I let you know. Best for your project.

[TimDettmers]

Upgrading via pip install -U bitsandbytes should resolve this issue. I will close this issue. Please reopen if upgrading does not resolve the error.

[swumagic]

Bitsandbytes was not supported windows before, but my method can support windows.（yuhuang） 1 open folder J:\StableDiffusion\sdwebui，Click the address bar of the folder and enter CMD or WIN+R, CMD 。enter，cd /d J:\StableDiffusion\sdwebui 2 J:\StableDiffusion\sdwebui\py310\python.exe -m pip uninstall bitsandbytes

3 J:\StableDiffusion\sdwebui\py310\python.exe -m pip uninstall bitsandbytes-windows

4 J:\StableDiffusion\sdwebui\py310\python.exe -m pip install https://github.com/jllllll/bitsandbytes-windows-webui/releases/download/wheels/bitsandbytes-0.41.1-py3-none-win_amd64.whl

Replace your SD venv directory file（python.exe Folder） here（J:\StableDiffusion\sdwebui\py310）