search

vllm-project / vllm

A high-throughput and memory-efficient inference and serving engine for LLMs
https://docs.vllm.ai
Apache License 2.0
27.25k stars 4k forks source link

ImportError: libcudart.so.12 #1718

Closed tranhoangnguyen03 closed 2 months ago

tranhoangnguyen03 commented 10 months ago

Was working yesterday. Today it gives the following error:

from vllm import cuda_utils
ImportError: libcudart.so.12: cannot open shared object file: No such file or directory

I've tried turning it off and on again multiple times.

WoosukKwon commented 10 months ago

Hi @tranhoangnguyen03 , we just released a new version (v0.2.2) which depends on PyTorch 2.1 and CUDA 12.1. This might cause error if you environment does not have CUDA 12.1.

  1. Please upgrade PyTorch using pip install --upgrade torch and see if the error persists.
  2. If it does not work, please install vLLM + CUDA 11.8 following the instructions here.
tranhoangnguyen03 commented 10 months ago

In case anyone else runs into the same trouble as I did. Here's how to make it work on Google Colab:

!pip -q install --upgrade fschat accelerate autoawq vllm
!pip install torch==2.1.0+cu121 torchvision==0.16.0+cu121 torchaudio==2.1.0 torchtext==0.16.0+cpu torchdata==0.7.0 --index-url https://download.pytorch.org/whl/cu121
Tanio253 commented 10 months ago

In case anyone else runs into the same trouble as I did. Here's how to make it work on Google Colab:

!pip -q install --upgrade fschat accelerate autoawq vllm
!pip install torch==2.1.0+cu121 torchvision==0.16.0+cu121 torchaudio==2.1.0 torchtext==0.16.0+cpu torchdata==0.7.0 --index-url https://download.pytorch.org/whl/cu121

Thank you! Super useful.

Hukongtao commented 10 months ago

Hi @tranhoangnguyen03 , we just released a new version (v0.2.2) which depends on PyTorch 2.1 and CUDA 12.1. This might cause error if you environment does not have CUDA 12.1.

  1. Please upgrade PyTorch using pip install --upgrade torch and see if the error persists.
  2. If it does not work, please install vLLM + CUDA 11.8 following the instructions here.

Is there a cuda11.7 version?

PeterXiaTian commented 9 months ago

Hi @tranhoangnguyen03 , we just released a new version (v0.2.2) which depends on PyTorch 2.1 and CUDA 12.1. This might cause error if you environment does not have CUDA 12.1.

  1. Please upgrade PyTorch using pip install --upgrade torch and see if the error persists.
  2. If it does not work, please install vLLM + CUDA 11.8 following the instructions here.

Is there a cuda11.7 version?

Is there a cuda11.1 version?

datalee commented 7 months ago

CUDA 12.1.

Is there a cuda11.7 version?

AWangji commented 4 months ago

Hi @tranhoangnguyen03 , we just released a new version (v0.2.2) which depends on PyTorch 2.1 and CUDA 12.1. This might cause error if you environment does not have CUDA 12.1.

  1. Please upgrade PyTorch using pip install --upgrade torch and see if the error persists.
  2. If it does not work, please install vLLM + CUDA 11.8 following the instructions here.

Is there a cuda11.7 version?

same question, is there anyone solve it?

DarkLight1337 commented 3 months ago

For other CUDA versions, please build vLLM from source.