libbitsandbytes_cpu.so: undefined symbol: cget_col_row_stats

[jaimeromoreno]

Hello I'm trying to run the generate.py example but I get the following error ("decapoda-research/llama-7b-hf" was previously downloaded) :

===================================BUG REPORT===================================
Welcome to bitsandbytes. For bug reports, please submit your error trace to: https://github.com/TimDettmers/bitsandbytes/issues
================================================================================
/root/anaconda3/envs/alpaca-lora/lib/python3.9/site-packages/bitsandbytes/cuda_setup/main.py:136: UserWarning: /root/anaconda3/envs/alpaca-lora did not contain libcudart.so as expected! Searching further paths...
  warn(msg)
/root/anaconda3/envs/alpaca-lora/lib/python3.9/site-packages/bitsandbytes/cuda_setup/main.py:136: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/usr/local/cuda/lib64')}
  warn(msg)
/root/anaconda3/envs/alpaca-lora/lib/python3.9/site-packages/bitsandbytes/cuda_setup/main.py:136: UserWarning: :/usr/local/cuda/lib64/ did not contain libcudart.so as expected! Searching further paths...
  warn(msg)
CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching /usr/local/cuda/lib64...
ERROR: python: undefined symbol: cudaRuntimeGetVersion
CUDA SETUP: libcudart.so path is None
CUDA SETUP: Is seems that your cuda installation is not in your path. See https://github.com/TimDettmers/bitsandbytes/issues/85 for more information.
CUDA SETUP: CUDA version lower than 11 are currently not supported for LLM.int8(). You will be only to use 8-bit optimizers and quantization routines!!
/root/anaconda3/envs/alpaca-lora/lib/python3.9/site-packages/bitsandbytes/cuda_setup/main.py:136: UserWarning: WARNING: No libcudart.so found! Install CUDA or the cudatoolkit package (anaconda)!
  warn(msg)
CUDA SETUP: Highest compute capability among GPUs detected: 7.5
CUDA SETUP: Detected CUDA version 00
CUDA SETUP: Loading binary /root/anaconda3/envs/alpaca-lora/lib/python3.9/site-packages/bitsandbytes/libbitsandbytes_cpu.so...
/root/anaconda3/envs/alpaca-lora/lib/python3.9/site-packages/bitsandbytes/cextension.py:31: UserWarning: The installed version of bitsandbytes was compiled without GPU support. 8-bit optimizers and GPU quantization are unavailable.
  warn("The installed version of bitsandbytes was compiled without GPU support. "
The tokenizer class you load from this checkpoint is not the same type as the class this function is called from. It may result in unexpected tokenization. 
The tokenizer class you load from this checkpoint is 'LLaMATokenizer'. 
The class this function is called from is 'LlamaTokenizer'.
Loading checkpoint shards:   0%|                                                                                                                   | 0/33 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "/root/projects/alpaca-lora/generate.py", line 13, in <module>
    model = LlamaForCausalLM.from_pretrained(
  File "/root/anaconda3/envs/alpaca-lora/lib/python3.9/site-packages/transformers/modeling_utils.py", line 2646, in from_pretrained
    ) = cls._load_pretrained_model(
  File "/root/anaconda3/envs/alpaca-lora/lib/python3.9/site-packages/transformers/modeling_utils.py", line 2969, in _load_pretrained_model
    new_error_msgs, offload_index, state_dict_index = _load_state_dict_into_meta_model(
  File "/root/anaconda3/envs/alpaca-lora/lib/python3.9/site-packages/transformers/modeling_utils.py", line 676, in _load_state_dict_into_meta_model
    set_module_8bit_tensor_to_device(model, param_name, param_device, value=param)
  File "/root/anaconda3/envs/alpaca-lora/lib/python3.9/site-packages/transformers/utils/bitsandbytes.py", line 70, in set_module_8bit_tensor_to_device
    new_value = bnb.nn.Int8Params(new_value, requires_grad=False, has_fp16_weights=has_fp16_weights).to(device)
  File "/root/anaconda3/envs/alpaca-lora/lib/python3.9/site-packages/bitsandbytes/nn/modules.py", line 196, in to
    return self.cuda(device)
  File "/root/anaconda3/envs/alpaca-lora/lib/python3.9/site-packages/bitsandbytes/nn/modules.py", line 160, in cuda
    CB, CBt, SCB, SCBt, coo_tensorB = bnb.functional.double_quant(B)
  File "/root/anaconda3/envs/alpaca-lora/lib/python3.9/site-packages/bitsandbytes/functional.py", line 1616, in double_quant
    row_stats, col_stats, nnz_row_ptr = get_colrow_absmax(
  File "/root/anaconda3/envs/alpaca-lora/lib/python3.9/site-packages/bitsandbytes/functional.py", line 1505, in get_colrow_absmax
    lib.cget_col_row_stats(ptrA, ptrRowStats, ptrColStats, ptrNnzrows, ct.c_float(threshold), rows, cols)
  File "/root/anaconda3/envs/alpaca-lora/lib/python3.9/ctypes/__init__.py", line 395, in __getattr__
    func = self.__getitem__(name)
  File "/root/anaconda3/envs/alpaca-lora/lib/python3.9/ctypes/__init__.py", line 400, in __getitem__
    func = self._FuncPtr((name_or_ordinal, self))
AttributeError: /root/anaconda3/envs/alpaca-lora/lib/python3.9/site-packages/bitsandbytes/libbitsandbytes_cpu.so: undefined symbol: cget_col_row_stats

I'm running into a conda environment based on python 3.9 and I installed the requirements with pip. Also I compiled the bitsandbytes package from source by running

git clone https://github.com/TimDettmers/bitsandbytes.git
cd bitsandbytes/
make cuda11x
CUDA_VERSION=112
python setup.py install

Here are the nvidia-smi and nvcc --version prints:

Fri Mar 17 13:53:10 2023       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.161.03   Driver Version: 470.161.03   CUDA Version: 11.4     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Quadro RTX 4000     On   | 00000000:00:10.0 Off |                  N/A |
| 30%   34C    P8     6W / 125W |      1MiB /  7982MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2021 NVIDIA Corporation
Built on Sun_Feb_14_21:12:58_PST_2021
Cuda compilation tools, release 11.2, V11.2.152
Build cuda_11.2.r11.2/compiler.29618528_0

which nvcc:
/usr/bin/nvcc
which nvidia-smi:
/usr/bin/nvidia-smi

Any idea what the problem might be? Thank you in advance.

[moorehousew]

I had the same issue, executing cp libbitsandbytes_cuda117.so libbitsandbytes_cpu.so in the bitsandbytes package folder worked for me- should be in the installation dir for your conda env.

[Jeffwan]

ubuntu@192-9-155-93:~/alpaca-lora$ cp /home/ubuntu/.local/lib/python3.8/site-packages/bitsandbytes/libbitsandbytes_cuda117.so /home/ubuntu/.local/lib/python3.8/site-packages/bitsandbytes/libbitsandbytes_cpu.so

I meet the exact same problem and this works for me

[iBibek]

I had the same issue, executing cp libbitsandbytes_cuda117.so libbitsandbytes_cpu.so in the bitsandbytes package folder worked for me- should be in the installation dir for your conda env.

Thank you so much

[LongZoz]

I ran into a similar problem, but after executing cp libbitsandbytes_cuda117.so libbitsandbytes_cpu.so, and then compiling the model, there was a new problem: CUDA Setup failed despite GPU being available. Inspect the CUDA SETUP outputs above to fix your environment!