Module 'bitsandbytes' has no attribute 'nn'

[Dygitz]

I installed bitsandbytes-windows with pip install git+https://github.com/Keith-Hon/bitsandbytes-windows.git and am using transformers by HuggingFace and got the below error. I am not sure if the error is caused by my CUDA version or something else. Any help appreciated!

My code

import os
os.environ["CUDA_VISIBLE_DEVICES"]="0"
import torch
import torch.nn as nn
import bitsandbytes as bnb
from transformers import AutoTokenizer, AutoConfig, AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained(
    "bigscience/bloom-7b1",
    load_in_8bit=True,
    device_map='auto',
)

tokenizer = AutoTokenizer.from_pretrained("bigscience/bloom-7b1")

Error

module 'bitsandbytes' has no attribute 'nn'

python -m bitsandbytes results

binary_path: C:\Users\golde\anaconda3\lib\site-packages\bitsandbytes\cuda_setup\libbitsandbytes_cuda116.dll CUDA SETUP: Loading binary C:\Users\golde\anaconda3\lib\site-packages\bitsandbytes\cuda_setup\libbitsandbytes_cuda116.dll... ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ ++++++++++++++++++++ DEBUG INFORMATION +++++++++++++++++++++ ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

++++++++++ POTENTIALLY LIBRARY-PATH-LIKE ENV VARS ++++++++++ ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

WARNING: Please be sure to sanitize sensible info from any such env vars!

++++++++++++++++++++++++++ OTHER +++++++++++++++++++++++++++ COMPILED_WITH_CUDA = True Traceback (most recent call last): File "C:\Users\golde\anaconda3\lib\runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "C:\Users\golde\anaconda3\lib\runpy.py", line 86, in _run_code exec(code, run_globals) File "C:\Users\golde\anaconda3\lib\site-packages\bitsandbytes__main__.py", line 47, in print(f"COMPUTE_CAPABILITIES_PER_GPU = {get_compute_capabilities(cuda)}") File "C:\Users\golde\anaconda3\lib\site-packages\bitsandbytes\cuda_setup\main.py", line 336, in get_compute_capabilities check_cuda_result(cuda, cuda.cuDeviceGetCount(ct.byref(nGpus))) AttributeError: 'NoneType' object has no attribute 'cuDeviceGetCount'

nvidia-smi results

Sat Jun 10 16:23:25 2023 +---------------------------------------------------------------------------------------+ | NVIDIA-SMI 535.98 Driver Version: 535.98 CUDA Version: 12.2 | |-----------------------------------------+----------------------+----------------------+ | GPU Name TCC/WDDM | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+======================+======================| | 0 NVIDIA GeForce GTX 1080 WDDM | 00000000:01:00.0 On | N/A | | 26% 41C P5 12W / 180W | 2118MiB / 8192MiB | 1% Default | | | | N/A | +-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=======================================================================================| | 0 N/A N/A 2072 C+G ....5.48.0_x64__htrsf667h5kn2\AWCC.exe N/A | | 0 N/A N/A 2560 C+G ...2txyewy\StartMenuExperienceHost.exe N/A | | 0 N/A N/A 6460 C+G ...h-0\231.9011.36\bin\dataspell64.exe N/A | | 0 N/A N/A 6524 C+G ...GeForce Experience\NVIDIA Share.exe N/A | | 0 N/A N/A 10816 C+G ...AppData\Roaming\Spotify\Spotify.exe N/A | | 0 N/A N/A 12196 C+G ...on\wallpaper_engine\wallpaper32.exe N/A | | 0 N/A N/A 13316 C+G ...5n1h2txyewy\ShellExperienceHost.exe N/A | | 0 N/A N/A 14240 C+G ...oogle\Chrome\Application\chrome.exe N/A | | 0 N/A N/A 16136 C+G ...m Files (x86)\Overwolf\Overwolf.exe N/A | | 0 N/A N/A 17464 C+G ...CBS_cw5n1h2txyewy\TextInputHost.exe N/A | | 0 N/A N/A 18088 C+G ...\Local\slack\app-4.32.122\slack.exe N/A | | 0 N/A N/A 18240 C+G ...pIntegrations\Grammarly.Desktop.exe N/A | | 0 N/A N/A 19300 C+G ...ne\Binaries\Win64\EpicWebHelper.exe N/A | | 0 N/A N/A 20252 C+G ...al\Discord\app-1.0.9013\Discord.exe N/A | | 0 N/A N/A 21688 C+G ....Search_cw5n1h2txyewy\SearchApp.exe N/A | | 0 N/A N/A 21752 C+G C:\Windows\explorer.exe N/A | | 0 N/A N/A 22304 C+G ...inaries\Win64\EpicGamesLauncher.exe N/A | | 0 N/A N/A 23652 C+G ...wolf\0.224.0.25\OverwolfBrowser.exe N/A | +---------------------------------------------------------------------------------------+

[TimDettmers]

Windows is currently not supported. You can find more information here: #30

[ihongxx]

$ sudo apt install gcc-10 g++-10 $ export CC=/usr/bin/gcc-10 $ export CXX=/usr/bin/g++-10 $ export CUDA_ROOT=/usr/local/cuda $ ln -s /usr/bin/gcc-10 $CUDA_ROOT/bin/gcc $ ln -s /usr/bin/g++-10 $CUDA_ROOT/bin/g++ $ git clone https://github.com/timdettmers/bitsandbytes.git $ cd bitsandbytes $ CUDA_HOME=/usr/local/cuda-11.6 CUDA_VERSION=116 make cuda11x $ python setup.py install I succeeded！