안녕하세요. LLaVA를 이용해서 석사 논문 준비중인 대학원생입니다. 파인튜닝할 수 있는 멀티모달 LLM을 찾던 도중 이 모델을 알게되어 사용하고 있었습니다.

경로 설정해놓고 아나콘다 가상환경도 만든 다음에 세팅하라는대로 다 했고요, 적혀있던 inference 명령어 그대로 실행했는데, 이런 오류가 떴습니다. CUDA 관련 오류인걸로 보이네요.

/home/work/anaconda3/envs/kollava/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:106: UserWarning:

_================================================================================ WARNING: Manual override via BNB_CUDA_VERSION env variable detected! BNB_CUDA_VERSION=XXX can be used to load a bitsandbytes version that is different from the PyTorch CUDA version. If this was unintended set the BNB_CUDA_VERSION variable to an empty string: export BNB_CUDA_VERSION= If you use the manual override make sure the right libcudart.so is in your LD_LIBRARY_PATH For example by adding the following to your .bashrc: export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:<path_to_cuda_dir/lib64 Loading CUDA version: BNB_CUDAVERSION=123 ================================================================================

warn((f'\n\n{"="*80}\n' False

===================================BUG REPORT=================================== /home/work/anaconda3/envs/kollava/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:166: UserWarning: Welcome to bitsandbytes. For bug reports, please run

python -m bitsandbytes

warn(msg)

/home/work/anaconda3/envs/kollava/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:166: UserWarning: /home/work/anaconda3/envs/kollava did not contain ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] as expected! Searching further paths... warn(msg) The following directories listed in your path were found to be non-existent: {PosixPath('/usr/local/cuda/extras/CUPTI/lib64'), PosixPath('/usr/local/nvidia/lib64')} /home/work/anaconda3/envs/kollava/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:166: UserWarning: /usr/local/cuda/compat/lib.real:/usr/local/lib/python3.10/dist-packages/torch/lib:/usr/local/lib/python3.10/dist-packages/torch_tensorrt/lib:/usr/local/cuda/compat/lib:/usr/local/nvidia/lib:/usr/local/nvidia/lib64:/usr/local/cuda/extras/CUPTI/lib64:/usr/local/cuda-12.3:/usr/local/cuda-12.3/include:/usr/include/x86_64-linux-gnu did not contain ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] as expected! Searching further paths... warn(msg) The following directories listed in your path were found to be non-existent: {PosixPath('7080/bai/ngc-pytorch'), PosixPath('bai-repo'), PosixPath('24.01-pytorch2.2-py310-cuda12.3')} CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching in backup paths... DEBUG: Possible options found for libcudart.so: {PosixPath('/usr/local/cuda/lib64/libcudart.so')} CUDA SETUP: PyTorch settings found: CUDA_VERSION=117, Highest Compute Capability: 7.5. CUDA SETUP: To manually override the PyTorch CUDA version please see:https://github.com/TimDettmers/bitsandbytes/blob/main/how_to_use_nonpytorch_cuda.md CUDA SETUP: Required library version not found: libbitsandbytes_cuda117.so. Maybe you need to compile it from source? CUDA SETUP: Defaulting to libbitsandbytes_cpu.so...

================================================ERROR===================================== CUDA SETUP: CUDA detection failed! Possible reasons:

You need to manually override the PyTorch CUDA version. Please see: "https://github.com/TimDettmers/bitsandbytes/blob/main/how_to_use_nonpytorch_cuda.md
CUDA driver not installed
CUDA not installed
You have multiple conflicting CUDA libraries
Required library not pre-compiled for this bitsandbytes release! CUDA SETUP: If you compiled from source, try again with make CUDA_VERSION=DETECTED_CUDA_VERSION for example, make CUDA_VERSION=113. CUDA SETUP: The CUDA version for the compile might depend on your conda install. Inspect CUDA version via conda list | grep cuda.

CUDA SETUP: Something unexpected happened. Please compile from source: git clone https://github.com/TimDettmers/bitsandbytes.git cd bitsandbytes CUDA_VERSION=117 make cuda11x python setup.py install CUDA SETUP: Setup Failed! Traceback (most recent call last): File "/home/work/anaconda3/envs/kollava/lib/python3.10/site-packages/transformers/utils/import_utils.py", line 1099, in _get_module return importlib.import_module("." + module_name, self.name) File "/home/work/anaconda3/envs/kollava/lib/python3.10/importlib/init.py", line 126, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "", line 1050, in _gcd_import File "", line 1027, in _find_and_load File "", line 1006, in _find_and_load_unlocked File "", line 688, in _load_unlocked File "", line 883, in exec_module File "", line 241, in _call_with_frames_removed File "/home/work/anaconda3/envs/kollava/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py", line 32, in from ...modeling_utils import PreTrainedModel File "/home/work/anaconda3/envs/kollava/lib/python3.10/site-packages/transformers/modeling_utils.py", line 38, in from .deepspeed import deepspeed_config, is_deepspeed_zero3_enabled File "/home/work/anaconda3/envs/kollava/lib/python3.10/site-packages/transformers/deepspeed.py", line 37, in from accelerate.utils.deepspeed import HfDeepSpeedConfig as DeepSpeedConfig File "/home/work/anaconda3/envs/kollava/lib/python3.10/site-packages/accelerate/init.py", line 3, in from .accelerator import Accelerator File "/home/work/anaconda3/envs/kollava/lib/python3.10/site-packages/accelerate/accelerator.py", line 35, in from .checkpointing import load_accelerator_state, load_custom_state, save_accelerator_state, save_custom_state File "/home/work/anaconda3/envs/kollava/lib/python3.10/site-packages/accelerate/checkpointing.py", line 24, in from .utils import ( File "/home/work/anaconda3/envs/kollava/lib/python3.10/site-packages/accelerate/utils/init.py", line 131, in from .bnb import has_4bit_bnb_layers, load_and_quantize_model File "/home/work/anaconda3/envs/kollava/lib/python3.10/site-packages/accelerate/utils/bnb.py", line 42, in import bitsandbytes as bnb File "/home/work/anaconda3/envs/kollava/lib/python3.10/site-packages/bitsandbytes/init.py", line 6, in from . import cuda_setup, utils, research File "/home/work/anaconda3/envs/kollava/lib/python3.10/site-packages/bitsandbytes/research/init.py", line 1, in from . import nn File "/home/work/anaconda3/envs/kollava/lib/python3.10/site-packages/bitsandbytes/research/nn/init.py", line 1, in from .modules import LinearFP8Mixed, LinearFP8Global File "/home/work/anaconda3/envs/kollava/lib/python3.10/site-packages/bitsandbytes/research/nn/modules.py", line 8, in from bitsandbytes.optim import GlobalOptimManager File "/home/work/anaconda3/envs/kollava/lib/python3.10/site-packages/bitsandbytes/optim/init.py", line 6, in from bitsandbytes.cextension import COMPILED_WITH_CUDA File "/home/work/anaconda3/envs/kollava/lib/python3.10/site-packages/bitsandbytes/cextension.py", line 20, in raise RuntimeError(''' RuntimeError: CUDA Setup failed despite GPU being available. Please run the following command to get more information:

    _python -m bitsandbytes

    Inspect the output of the command and see if you can locate CUDA libraries. You might need to add them
    to your LD_LIBRARY_PATH. If you suspect a bug, please take the information from python -m bitsandbytes
    and open an issue at: https://github.com/TimDettmers/bitsandbytes/issues_

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/home/work/anaconda3/envs/kollava/lib/python3.10/runpy.py", line 187, in _run_module_as_main mod_name, mod_spec, code = _get_module_details(mod_name, _Error) File "/home/work/anaconda3/envs/kollava/lib/python3.10/runpy.py", line 110, in _get_module_details import(pkg_name) File "/home/work/testdataset1/KoLLaVA/llava/init.py", line 1, in from .model import LlavaLlamaForCausalLM File "/home/work/testdataset1/KoLLaVA/llava/model/init.py", line 1, in from .language_model.llava_llama import LlavaLlamaForCausalLM, LlavaConfig File "/home/work/testdataset1/KoLLaVA/llava/model/language_model/llava_llama.py", line 21, in from transformers import AutoConfig, AutoModelForCausalLM, \ File "", line 1075, in _handle_fromlist File "/home/work/anaconda3/envs/kollava/lib/python3.10/site-packages/transformers/utils/import_utils.py", line 1090, in getattr value = getattr(module, name) File "/home/work/anaconda3/envs/kollava/lib/python3.10/site-packages/transformers/utils/import_utils.py", line 1089, in getattr module = self._get_module(self._class_to_module[name]) File "/home/work/anaconda3/envs/kollava/lib/python3.10/site-packages/transformers/utils/import_utils.py", line 1101, in _get_module raise RuntimeError( RuntimeError: Failed to import transformers.models.llama.modeling_llama because of the following error (look up to see its traceback):

    CUDA Setup failed despite GPU being available. Please run the following command to get more information:

    _python -m bitsandbytes

    Inspect the output of the command and see if you can locate CUDA libraries. You might need to add them
    to your LD_LIBRARY_PATH. If you suspect a bug, please take the information from python -m bitsandbytes
    and open an issue at: https://github.com/TimDettmers/bitsandbytes/issues_

전에도 오류가 계속 떴지만 이거랑은 다른 거였습니다. 그래서 아예 다시 git clone 하고, 가상환경도 새로 만들고 다 해봤는데, 현재는 이 오류만 계속 뜹니다. 다시 git clone하고, 가상환경도 다시 만드는데도 해결되지 않네요. 이런 에러는 주로 어떨 때 나오나요?

참고로 아나콘다는 제 리눅스 서버 환경에 Anaconda3-2020.11-Linux-x86_64.sh로 설치했습니다.

감사합니다.

tabtoyou / KoLLaVA

Inference를 돌리다가 이런 오류가 떴네요. #22

warn(msg)