Unable to get the output due to bitsandbytes library error

unslothai / unsloth

Finetune Llama 3.2, Mistral, Phi, Qwen 2.5 & Gemma LLMs 2-5x faster with 80% less memory

https://unsloth.ai

Apache License 2.0

18.53k stars 1.3k forks source link

Unable to get the output due to bitsandbytes library error #1308

Open iamnikhildogra opened 5 days ago

iamnikhildogra commented 5 days ago

Hi,

I am currently trying to run unsloth/llama-3-8b-Instruct-bnb-4bit model on K80 GPU machine with cuda version: cuda_12.6.r12.6 and my facing this issue while trying to generate the model:

AttributeError: /root/venv/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cpu.so: undefined symbol: cdequantize_blockwise_fp32

Please anyone can help me quick if possible

danielhanchen commented 3 days ago

Oh K80s are a bit old :( I'm unsure if we support that sorry - maybe an older version of bitsandbytes and pytorch and unsloth might work, but unsure - sorry