CUDA Setup failed despite GPU being available. Please run the following command to get more information:

[WuRui-Ella]

配置： 系统：CentOS Linux release 7.9.2009 CUDA版本： 12.4 python版本： 3.11 NVIDIA-SMI ：550.54.14 我在使用hello_world推理时遇到了这个问题，已经提示重新编译了bitsandbytes，但还是出现了这个问题。

问题如下： CUDA SETUP: Something unexpected happened. Please compile from source: git clone git@github.com:TimDettmers/bitsandbytes.git cd bitsandbytes CUDA_VERSION=124 python setup.py install CUDA SETUP: Setup Failed! CUDA SETUP: Something unexpected happened. Please compile from source: git clone git@github.com:TimDettmers/bitsandbytes.git cd bitsandbytes CUDA_VERSION=124 python setup.py install ╭───────────────────── Traceback (most recent call last) ──────────────────────╮ │ /home/ToolBench-master/toolbench/inference/qa_pipeline.py:13 in │ │ │ │ 10 os.environ['CUDA_VERSION'] = "12.4" │ │ 11 os.environ['NVCC'] = '/usr/local/cuda-12.4/bin/nvcc' │ │ 12 │ │ ❱ 13 from toolbench.inference.Downstream_tasks.rapidapi import pipeline_runn │ │ 14 │ │ 15 │ │ 16 if name == "main": │ │ │ │ /home/ToolBench-master/toolbench/inference/Downstream_tasks/rapidapi.py:1 │ │ 0 in │ │ │ │ 7 from termcolor import colored │ │ 8 import random │ │ 9 from toolbench.inference.LLM.chatgpt_function_model import ChatGPTFunc │ │ ❱ 10 from toolbench.inference.LLM.davinci_model import Davinci │ │ 11 from toolbench.inference.LLM.tool_llama_lora_model import ToolLLaMALoR │ │ 12 from toolbench.inference.LLM.tool_llama_model import ToolLLaMA │ │ 13 from toolbench.inference.LLM.retriever import ToolRetriever │ │ │ │ /home/ToolBench-master/toolbench/inference/LLM/davinci_model.py:9 in │ │ │ │ │ │ 6 import random │ │ 7 import openai │ │ 8 from typing import Optional │ │ ❱ 9 from toolbench.model.model_adapter import get_conversation_template │ │ 10 from toolbench.inference.utils import SimpleChatIO, react_parser │ │ 11 from toolbench.inference.Prompts.ReAct_prompts import FORMAT_INSTRUCTI │ │ 12 │ │ │ │ /home/ToolBench-master/toolbench/model/init.py:1 in │ │ │ │ ❱ 1 from toolbench.model.model_adapter import ( │ │ 2 │ load_model, │ │ 3 │ get_conversation_template, │ │ 4 │ add_model_args, │ │ │ │ /home/ToolBench-master/toolbench/model/model_adapter.py:21 in │ │ │ │ 18 │ AutoTokenizer, │ │ 19 │ LlamaForCausalLM, │ │ 20 ) │ │ ❱ 21 from peft import PeftModel │ │ 22 │ │ 23 from toolbench.tool_conversation import Conversation, get_conv_templat │ │ 24 from toolbench.model.compression import load_compress_model │ │ │ │ /home/anaconda3/envs/toolllm/lib/python3.11/site-packages/peft/init.py:2 │ │ 2 in │ │ │ │ 19 │ │ 20 version = "0.3.0" │ │ 21 │ │ ❱ 22 from .mapping import MODEL_TYPE_TO_PEFT_MODEL_MAPPING, PEFT_TYPE_TO_CON │ │ 23 from .peft_model import ( │ │ 24 │ PeftModel, │ │ 25 │ PeftModelForCausalLM, │ │ │ │ /home/anaconda3/envs/toolllm/lib/python3.11/site-packages/peft/mapping.py:16 │ │ in │ │ │ │ 13 # See the License for the specific language governing permissions and │ │ 14 # limitations under the License. │ │ 15 │ │ ❱ 16 from .peft_model import ( │ │ 17 │ PeftModel, │ │ 18 │ PeftModelForCausalLM, │ │ 19 │ PeftModelForSeq2SeqLM, │ │ │ │ /home/anaconda3/envs/toolllm/lib/python3.11/site-packages/peft/peft_model.py │ │ :31 in │ │ │ │ 28 from transformers.modeling_outputs import SequenceClassifierOutput, T │ │ 29 from transformers.utils import PushToHubMixin │ │ 30 │ │ ❱ 31 from .tuners import ( │ │ 32 │ AdaLoraModel, │ │ 33 │ AdaptionPromptModel, │ │ 34 │ LoraModel, │ │ │ │ /home/anaconda3/envs/toolllm/lib/python3.11/site-packages/peft/tuners/init │ │ .py:21 in │ │ │ │ 18 # limitations under the License. │ │ 19 │ │ 20 from .adaption_prompt import AdaptionPromptConfig, AdaptionPromptModel │ │ ❱ 21 from .lora import LoraConfig, LoraModel │ │ 22 from .adalora import AdaLoraConfig, AdaLoraModel │ │ 23 from .p_tuning import PromptEncoder, PromptEncoderConfig, PromptEncoder │ │ 24 from .prefix_tuning import PrefixEncoder, PrefixTuningConfig │ │ │ │ /home/anaconda3/envs/toolllm/lib/python3.11/site-packages/peft/tuners/lora.p │ │ y:40 in │ │ │ │ 37 │ │ 38 │ │ 39 if is_bnb_available(): │ │ ❱ 40 │ import bitsandbytes as bnb │ │ 41 │ │ 42 │ │ 43 @dataclass │ │ │ │ /home/anaconda3/envs/toolllm/lib/python3.11/site-packages/bitsandbytes-0.38. │ │ 0.post2-py3.11.egg/bitsandbytes/init.py:7 in │ │ │ │ 4 # LICENSE file in the root directory of this source tree. │ │ 5 │ │ 6 from . import cuda_setup, utils │ │ ❱ 7 from .autograd._functions import ( │ │ 8 │ MatmulLtState, │ │ 9 │ bmm_cublas, │ │ 10 │ matmul, │ │ │ │ /home/anaconda3/envs/toolllm/lib/python3.11/site-packages/bitsandbytes-0.38. │ │ 0.post2-py3.11.egg/bitsandbytes/autograd/init.py:1 in │ │ │ │ ❱ 1 from ._functions import undo_layout, get_inverse_transform_indices │ │ 2 │ │ │ │ /home/anaconda3/envs/toolllm/lib/python3.11/site-packages/bitsandbytes-0.38. │ │ 0.post2-py3.11.egg/bitsandbytes/autograd/_functions.py:9 in │ │ │ │ 6 │ │ 7 import torch │ │ 8 │ │ ❱ 9 import bitsandbytes.functional as F │ │ 10 │ │ 11 │ │ 12 # math.prod not compatible with python < 3.8 │ │ │ │ /home/anaconda3/envs/toolllm/lib/python3.11/site-packages/bitsandbytes-0.38. │ │ 0.post2-py3.11.egg/bitsandbytes/functional.py:17 in │ │ │ │ 14 from typing import Tuple │ │ 15 from torch import Tensor │ │ 16 │ │ ❱ 17 from .cextension import COMPILED_WITH_CUDA, lib │ │ 18 │ │ 19 │ │ 20 # math.prod not compatible with python < 3.8 │ │ │ │ /home/anaconda3/envs/toolllm/lib/python3.11/site-packages/bitsandbytes-0.38. │ │ 0.post2-py3.11.egg/bitsandbytes/cextension.py:20 in │ │ │ │ 17 │ if lib is None and torch.cuda.is_available(): │ │ 18 │ │ CUDASetup.get_instance().generate_instructions() │ │ 19 │ │ CUDASetup.get_instance().print_log_stack() │ │ ❱ 20 │ │ raise RuntimeError(''' │ │ 21 │ │ CUDA Setup failed despite GPU being available. Please run the f │ │ 22 │ │ │ │ 23 │ │ python -m bitsandbytes │ ╰──────────────────────────────────────────────────────────────────────────────╯ RuntimeError: CUDA Setup failed despite GPU being available. Please run the following command to get more information:

    python -m bitsandbytes

    Inspect the output of the command and see if you can locate CUDA

libraries. You might need to add them to your LD_LIBRARY_PATH. If you suspect a bug, please take the information from python -m bitsandbytes and open an issue at: https://github.com/TimDettmers/bitsandbytes/issues ./scripts/inference_toolllama_helloworld_demo.sh:行3: --tool_root_dir: 未找到命令 ./scripts/inference_toolllama_helloworld_demo.sh:行4: --backbone_model: 未找到命令 ./scripts/inference_toolllama_helloworld_demo.sh:行5: --model_path: 未找到命令 ./scripts/inference_toolllama_helloworld_demo.sh:行6: --max_observation_length: 未找到命令 ./scripts/inference_toolllama_helloworld_demo.sh:行7: --observ_compress_method: 未找到命令 ./scripts/inference_toolllama_helloworld_demo.sh:行8: --method: 未找到命令 ./scripts/inference_toolllama_helloworld_demo.sh:行9: --input_query_file: 未找到命令 ./scripts/inference_toolllama_helloworld_demo.sh:行10: --output_answer_file: 未找到命令 ./scripts/inference_toolllama_helloworld_demo.sh:行11: --api_customization: 未找到命令

终端输入nvcc -V 能看出来cuda是正确安装的，

找不到解决办法了55555555，特来问问大家，有知道怎么解决的嘛

[Cgf601]

你好 请问你解决了吗