search

bitsandbytes-foundation / bitsandbytes

Accessible large language models via k-bit quantization for PyTorch.
https://huggingface.co/docs/bitsandbytes/main/en/index
MIT License
6.3k stars 630 forks source link

"You have a version of `bitsandbytes` that is not compatible with 4bit inference and training" #1246

Open entzyeung opened 5 months ago

entzyeung commented 5 months ago

I am now trying to finetune a llama3 model. I am using unsloth,

from unsloth import FastLanguageModel

Then I load Llama3 model.

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name = "unsloth/llama-3-8b-bnb-4bit",
    max_seq_length = max_seq_length,
    dtype = None,
    load_in_4bit = True,
)

I am running my script on CS Code, and my python and script are on WSL. My system info is as below:


==((====))==  Unsloth: Fast Llama patching release 2024.5
   \\   /|    GPU: NVIDIA GeForce RTX 4090. Max memory: 23.988 GB. Platform = Linux.
O^O/ \_/ \    Pytorch: 2.1.0+cu121. CUDA = 8.9. CUDA Toolkit = 12.1.
\        /    Bfloat16 = TRUE. Xformers = 0.0.22.post7. FA = False.
 "-____-"     Free Apache license: http://github.com/unslothai/unsloth

Now I run into this error:


---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[7], line 2
      1 # 2. Load Llama3 model
----> 2 model, tokenizer = FastLanguageModel.from_pretrained(
      3     model_name = "unsloth/llama-3-8b-bnb-4bit",
      4     max_seq_length = max_seq_length,
      5     dtype = None,
      6     load_in_4bit = True,
      7 )

File ~/miniconda/envs/llama3/lib/python3.9/site-packages/unsloth/models/loader.py:142, in FastLanguageModel.from_pretrained(model_name, max_seq_length, dtype, load_in_4bit, token, device_map, rope_scaling, fix_tokenizer, trust_remote_code, use_gradient_checkpointing, resize_model_vocab, *args, **kwargs)
    139     tokenizer_name = None
    140 pass
--> 142 model, tokenizer = dispatch_model.from_pretrained(
    143     model_name     = model_name,
    144     max_seq_length = max_seq_length,
    145     dtype          = dtype,
    146     load_in_4bit   = load_in_4bit,
    147     token          = token,
    148     device_map     = device_map,
    149     rope_scaling   = rope_scaling,
    150     fix_tokenizer  = fix_tokenizer,
    151     model_patcher  = dispatch_model,
    152     tokenizer_name = tokenizer_name,
...
     96         "You have a version of `bitsandbytes` that is not compatible with 4bit inference and training"
     97         " make sure you have the latest version of `bitsandbytes` installed"
     98     )

I tried to update my bitsandbytes, with pip install bitsandbytes, it says all the requirements are satisfied.

Would anyone please help?

matthewdouglas commented 5 months ago

@entzyeung You can upgrade to the latest version with pip install -U bitsandbytes.

entzyeung commented 5 months ago

I tried pip install -U bitsandbytes too, it basically returns the same things,

(llama3) ubuntu2022@alien:~$ pip install -U bitsandbytes
Requirement already satisfied: bitsandbytes in ./miniconda/envs/llama3/lib/python3.9/site-packages (0.43.1)
Requirement already satisfied: torch in ./miniconda/envs/llama3/lib/python3.9/site-packages (from bitsandbytes) (2.1.0)
Requirement already satisfied: numpy in ./miniconda/envs/llama3/lib/python3.9/site-packages (from bitsandbytes) (1.26.4)
Requirement already satisfied: filelock in ./miniconda/envs/llama3/lib/python3.9/site-packages (from torch->bitsandbytes) (3.14.0)
Requirement already satisfied: typing-extensions in ./miniconda/envs/llama3/lib/python3.9/site-packages (from torch->bitsandbytes) (4.12.1)
Requirement already satisfied: sympy in ./miniconda/envs/llama3/lib/python3.9/site-packages (from torch->bitsandbytes) (1.12.1)
Requirement already satisfied: networkx in ./miniconda/envs/llama3/lib/python3.9/site-packages (from torch->bitsandbytes) (3.2.1)
Requirement already satisfied: jinja2 in ./miniconda/envs/llama3/lib/python3.9/site-packages (from torch->bitsandbytes) (3.1.4)
Requirement already satisfied: fsspec in ./miniconda/envs/llama3/lib/python3.9/site-packages (from torch->bitsandbytes) (2024.3.1)
Requirement already satisfied: nvidia-cuda-nvrtc-cu12==12.1.105 in ./miniconda/envs/llama3/lib/python3.9/site-packages (from torch->bitsandbytes) (12.1.105)
Requirement already satisfied: nvidia-cuda-runtime-cu12==12.1.105 in ./miniconda/envs/llama3/lib/python3.9/site-packages (from torch->bitsandbytes) (12.1.105)
Requirement already satisfied: nvidia-cuda-cupti-cu12==12.1.105 in ./miniconda/envs/llama3/lib/python3.9/site-packages (from torch->bitsandbytes) (12.1.105)
Requirement already satisfied: nvidia-cudnn-cu12==8.9.2.26 in ./miniconda/envs/llama3/lib/python3.9/site-packages (from torch->bitsandbytes) (8.9.2.26)
Requirement already satisfied: nvidia-cublas-cu12==12.1.3.1 in ./miniconda/envs/llama3/lib/python3.9/site-packages (from torch->bitsandbytes) (12.1.3.1)
Requirement already satisfied: nvidia-cufft-cu12==11.0.2.54 in ./miniconda/envs/llama3/lib/python3.9/site-packages (from torch->bitsandbytes) (11.0.2.54)
Requirement already satisfied: nvidia-curand-cu12==10.3.2.106 in ./miniconda/envs/llama3/lib/python3.9/site-packages (from torch->bitsandbytes) (10.3.2.106)
Requirement already satisfied: nvidia-cusolver-cu12==11.4.5.107 in ./miniconda/envs/llama3/lib/python3.9/site-packages (from torch->bitsandbytes) (11.4.5.107)
Requirement already satisfied: nvidia-cusparse-cu12==12.1.0.106 in ./miniconda/envs/llama3/lib/python3.9/site-packages (from torch->bitsandbytes) (12.1.0.106)
Requirement already satisfied: nvidia-nccl-cu12==2.18.1 in ./miniconda/envs/llama3/lib/python3.9/site-packages (from torch->bitsandbytes) (2.18.1)
Requirement already satisfied: nvidia-nvtx-cu12==12.1.105 in ./miniconda/envs/llama3/lib/python3.9/site-packages (from torch->bitsandbytes) (12.1.105)
Requirement already satisfied: triton==2.1.0 in ./miniconda/envs/llama3/lib/python3.9/site-packages (from torch->bitsandbytes) (2.1.0)
Requirement already satisfied: nvidia-nvjitlink-cu12 in ./miniconda/envs/llama3/lib/python3.9/site-packages (from nvidia-cusolver-cu12==11.4.5.107->torch->bitsandbytes) (12.5.40)
Requirement already satisfied: MarkupSafe>=2.0 in ./miniconda/envs/llama3/lib/python3.9/site-packages (from jinja2->torch->bitsandbytes) (2.1.5)
matthewdouglas commented 5 months ago

That message is coming from transformers where the version of bitsandbytes needs to be validated. For some reason it seems unable to determine that you have 0.43.1 installed.

Can you check to make sure the dist-info for the package is there? You should see it somewhere like ./miniconda/envs/llama3/lib/python3.9/site-packages/bitsandbytes-0.43.1.dist-info/*.

Is there any chance this was removed at some point?

Related: python/cpython#91216

entzyeung commented 5 months ago

Thank you @matthewdouglas

image The dist-info folder is there (see the pic above).

My transformers version is as below:

import transformers
print(transformers.__version__)

returns:
4.41.2

and I can confirm that my bitsandbytes is 0.43.1

(llama3) ubuntu2022@alien:~$ pip list
Package                  Version
------------------------ ------------
accelerate               0.31.0
aiohttp                  3.9.5
aiosignal                1.3.1
asttokens                2.4.1
async-timeout            4.0.3
attrs                    23.2.0
bitsandbytes             0.43.1

my unsloth is 2024.5

entzyeung commented 5 months ago

The entire list from pip list

Package                  Version
------------------------ ------------
accelerate               0.31.0
aiohttp                  3.9.5
aiosignal                1.3.1
asttokens                2.4.1
async-timeout            4.0.3
attrs                    23.2.0
bitsandbytes             0.43.1
Bottleneck               1.3.7
certifi                  2024.6.2
charset-normalizer       3.3.2
click                    8.1.7
comm                     0.2.2
contourpy                1.2.0
cycler                   0.11.0
datasets                 2.19.2
debugpy                  1.6.7
decorator                5.1.1
dill                     0.3.8
docker-pycreds           0.4.0
docstring_parser         0.16
eval_type_backport       0.2.0
exceptiongroup           1.2.0
executing                2.0.1
filelock                 3.14.0
fonttools                4.51.0
frozenlist               1.4.1
fsspec                   2024.3.1
gitdb                    4.0.11
GitPython                3.1.43
huggingface-hub          0.23.3
idna                     3.7
importlib_metadata       7.1.0
importlib-resources      6.1.1
ipykernel                6.29.4
ipython                  8.18.1
ipywidgets               8.1.3
jedi                     0.19.1
Jinja2                   3.1.4
jupyter_client           8.6.2
jupyter_core             5.7.2
jupyterlab_widgets       3.0.11
kiwisolver               1.4.4
markdown-it-py           3.0.0
MarkupSafe               2.1.5
matplotlib               3.8.4
matplotlib-inline        0.1.7
mdurl                    0.1.2
mkl-fft                  1.3.8
mkl-random               1.2.4
mkl-service              2.4.0
mpmath                   1.3.0
multidict                6.0.5
multiprocess             0.70.16
nest_asyncio             1.6.0
networkx                 3.2.1
numexpr                  2.8.7
numpy                    1.26.4
nvidia-cublas-cu12       12.1.3.1
nvidia-cuda-cupti-cu12   12.1.105
nvidia-cuda-nvrtc-cu12   12.1.105
nvidia-cuda-runtime-cu12 12.1.105
nvidia-cudnn-cu12        8.9.2.26
nvidia-cufft-cu12        11.0.2.54
nvidia-curand-cu12       10.3.2.106
nvidia-cusolver-cu12     11.4.5.107
nvidia-cusparse-cu12     12.1.0.106
nvidia-nccl-cu12         2.18.1
nvidia-nvjitlink-cu12    12.5.40
nvidia-nvtx-cu12         12.1.105
packaging                23.2
pandas                   2.2.1
parso                    0.8.4
peft                     0.11.1
pexpect                  4.9.0
pickleshare              0.7.5
pillow                   10.3.0
pip                      24.0
platformdirs             4.2.2
ply                      3.11
prompt_toolkit           3.0.46
protobuf                 3.20.3
psutil                   5.9.8
ptyprocess               0.7.0
pure-eval                0.2.2
pyarrow                  16.1.0
pyarrow-hotfix           0.6
Pygments                 2.18.0
pyparsing                3.0.9
PyQt5                    5.15.10
PyQt5-sip                12.13.0
python-dateutil          2.9.0.post0
pytz                     2024.1
PyYAML                   6.0.1
pyzmq                    25.1.2
regex                    2024.5.15
requests                 2.32.3
rich                     13.7.1
safetensors              0.4.3
sentencepiece            0.2.0
sentry-sdk               2.5.0
setproctitle             1.3.3
setuptools               69.5.1
shtab                    1.7.1
sip                      6.7.12
six                      1.16.0
smmap                    5.0.1
stack-data               0.6.2
sympy                    1.12.1
tokenizers               0.19.1
tomli                    2.0.1
torch                    2.1.0
tornado                  6.3.3
tqdm                     4.66.4
traitlets                5.14.3
transformers             4.41.2
triton                   2.1.0
trl                      0.8.6
typing_extensions        4.12.1
tyro                     0.8.4
tzdata                   2023.3
unicodedata2             15.1.0
unsloth                  2024.5
urllib3                  2.2.1
wandb                    0.17.1
wcwidth                  0.2.13
wheel                    0.43.0
widgetsnbextension       4.0.11
xformers                 0.0.22.post7
xxhash                   3.4.1
yarl                     1.9.4
zipp                     3.17.0
matthewdouglas commented 5 months ago

@entzyeung I would consider reinstalling with pip install --force-reinstall bitsandbytes. cc: @younesbelkada Any other ideas on this one?

entzyeung commented 5 months ago

thank you @matthewdouglas pip install --force-reinstall bitsandbytes works! I just tried -U before, and have never tired --force.