[Bug] 推理加载模型时报错ImportError: libtorch_cuda_cpp.so: cannot open shared object file: No such file or directory

jamesbondzhou commented 1 month ago

Checklist

[X] 1. I have searched related issues but cannot get the expected help.
[X] 2. The bug has not been fixed in the latest version.
[X] 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.

Describe the bug

按照提示装好环境后，执行 model=AutoModel.from_pretrained(path,torch_dtype=torch.float16,low_cpu_mem_usage=True,trust_remote_code=True,device_map='auto').eval().cuda()报错，如下：

File /data/anaconda3/envs/videochat_new/lib/python3.9/site-packages/transformers/models/llama/modeling_llama.py:55 54 if is_flash_attn_2_available(): ---> 55 from flash_attn import flash_attn_func, flash_attn_varlen_func 56 from flash_attn.bert_padding import index_first_axis, pad_input, unpad_input # noqa

File /data/anaconda3/envs/videochat_new/lib/python3.9/site-packages/flash_attn/init.py:3 1 version = "2.3.6" ----> 3 from flash_attn.flash_attn_interface import ( 4 flash_attn_func, 5 flash_attn_kvpacked_func, 6 flash_attn_qkvpacked_func, 7 flash_attn_varlen_func, 8 flash_attn_varlen_kvpacked_func, 9 flash_attn_varlen_qkvpacked_func, 10 flash_attn_with_kvcache, 11 )

File /data/anaconda3/envs/videochat_new/lib/python3.9/site-packages/flash_attn/flash_attn_interface.py:10 8 # isort: off 9 # We need to import the CUDA kernels after importing torch ---> 10 import flash_attn_2_cuda as flash_attn_cuda 12 # isort: on

ImportError: libtorch_cuda_cpp.so: cannot open shared object file: No such file or directory

Reproduction

model=AutoModel.from_pretrained(path,torch_dtype=torch.float16,low_cpu_mem_usage=True,trust_remote_code=True,device_map='auto').eval().cuda()

Environment

使用官方环境，
torch=2.0.0
transformers=4.37.2
flash-attn=2.3.6

Error traceback

No response

jamesbondzhou commented 1 month ago

推理使用的模型是'OpenGVLab/Mini-InternVL-Chat-4B-V1-5'

czczup commented 1 month ago

Hello, the installation of Flash Attention might not have been successful. You can try uninstalling Flash Attention first and then run it again, or follow the official instructions for Flash Attention to reinstall it: https://github.com/Dao-AILab/flash-attention.

jamesbondzhou commented 1 month ago

Hello, the installation of Flash Attention might not have been successful. You can try uninstalling Flash Attention first and then run it again, or follow the official instructions for Flash Attention to reinstall it: https://github.com/Dao-AILab/flash-attention.

YES! Uninstalling Flash Attention ,it works! 3KS Bro

OpenGVLab / InternVL