Open xiaohaiqing opened 3 months ago
It seems you have a mixed installation of both cuda 11 and cuda 12. Please try a fresh new install with either cuda 11.8 or cuda 12.1 .
目前cuda=12.1
nvidia-cuda-nvrtc-cu11 11.7.99 nvidia-cuda-nvrtc-cu12 12.1.105
You see here you have two cuda versions. You can only have one to make vllm work.
torch-2.2.1 and vllm-0.3.3, the error is _C.cpython-311-x86_64-linux-gnu.so: undefined symbol: _ZN2at4_ops15to_dtype_layout4callERKNS_6TensorEN3c108optionalINS5_10ScalarTypeEEENS6_INS5_6LayoutEEENS6_INS5_6DeviceEEENS6_IbEEbbNS6_INS5_12MemoryFormatEEE torch-2.1.2 huggingface has error:RuntimeError: cutlassF: no kernel found to launch!
accelerate 0.28.0 aiofiles 23.2.1 aiohttp 3.9.3 aiosignal 1.3.1 altair 5.2.0 annotated-types 0.6.0 anyio 4.3.0 archspec 0.2.3 attrs 23.2.0 boltons 23.0.0 Brotli 1.0.9 certifi 2024.2.2 cffi 1.16.0 charset-normalizer 2.0.4 click 8.1.7 cloudpickle 3.0.0 conda 24.3.0 conda-content-trust 0.2.0 conda-libmamba-solver 23.12.0 conda-package-handling 2.2.0 conda_package_streaming 0.9.0 contourpy 1.2.0 cryptography 41.0.7 cupy-cuda12x 12.1.0 cycler 0.12.1 datasets 2.18.0 dill 0.3.8 diskcache 5.6.3 distro 1.8.0 docstring_parser 0.16 einops 0.7.0 fastapi 0.110.0 fastrlock 0.8.2 ffmpy 0.3.2 filelock 3.13.1 fire 0.6.0 fonttools 4.50.0 frozenlist 1.4.1 fsspec 2024.2.0 gmpy2 2.1.2 gradio 3.50.2 gradio_client 0.6.1 h11 0.14.0 httpcore 1.0.4 httptools 0.6.1 httpx 0.27.0 huggingface-hub 0.21.4 idna 3.4 importlib_resources 6.4.0 interegular 0.3.3 Jinja2 3.1.3 joblib 1.3.2 jsonpatch 1.32 jsonpointer 2.1 jsonschema 4.21.1 jsonschema-specifications 2023.12.1 kiwisolver 1.4.5 lark 1.1.9 libmambapy 1.5.3 llvmlite 0.42.0 markdown-it-py 3.0.0 MarkupSafe 2.1.3 matplotlib 3.8.3 mdurl 0.1.2 menuinst 2.0.1 mkl-fft 1.3.8 mkl-random 1.2.4 mkl-service 2.4.0 mpmath 1.3.0 msgpack 1.0.8 multidict 6.0.5 multiprocess 0.70.16 nest-asyncio 1.6.0 networkx 3.1 ninja 1.11.1.1 numba 0.59.1 numpy 1.26.4 nvidia-cublas-cu12 12.1.3.1 nvidia-cuda-cupti-cu12 12.1.105 nvidia-cuda-nvrtc-cu12 12.1.105 nvidia-cuda-runtime-cu12 12.1.105 nvidia-cudnn-cu12 8.9.2.26 nvidia-cufft-cu12 11.0.2.54 nvidia-curand-cu12 10.3.2.106 nvidia-cusolver-cu12 11.4.5.107 nvidia-cusparse-cu12 12.1.0.106 nvidia-nccl-cu12 2.18.1 nvidia-nvjitlink-cu12 12.4.99 nvidia-nvtx-cu12 12.1.105 orjson 3.9.15 outlines 0.0.36 packaging 23.1 pandas 2.2.1 peft 0.10.0 pillow 10.2.0 pip 23.3.1 platformdirs 3.10.0 pluggy 1.0.0 prometheus_client 0.20.0 protobuf 5.26.0 psutil 5.9.8 pyarrow 15.0.2 pyarrow-hotfix 0.6 pycosat 0.6.6 pycparser 2.21 pydantic 2.6.4 pydantic_core 2.16.3 pydub 0.25.1 Pygments 2.17.2 pynvml 11.5.0 pyOpenSSL 23.2.0 pyparsing 3.1.2 PySocks 1.7.1 python-dateutil 2.9.0.post0 python-dotenv 1.0.1 python-multipart 0.0.9 pytz 2024.1 PyYAML 6.0.1 ray 2.10.0 referencing 0.34.0 regex 2023.12.25 requests 2.31.0 rich 13.7.1 rpds-py 0.18.0 ruamel.yaml 0.17.21 safetensors 0.4.2 scipy 1.12.0 semantic-version 2.10.0 sentencepiece 0.2.0 setuptools 68.2.2 shtab 1.7.1 six 1.16.0 sniffio 1.3.1 sse-starlette 2.0.0 starlette 0.36.3 sympy 1.12 termcolor 2.4.0 tokenizers 0.15.2 toolz 0.12.1 torch 2.1.2 torchaudio 2.2.1 torchvision 0.17.1 tqdm 4.65.0 transformers 4.39.0 triton 2.1.0 trl 0.8.1 truststore 0.8.0 typing_extensions 4.9.0 tyro 0.7.3 tzdata 2024.1 urllib3 1.26.18 uvicorn 0.29.0 uvloop 0.19.0 vllm 0.3.3 watchfiles 0.21.0 websockets 11.0.3 wheel 0.41.2 xformers 0.0.23.post1 xxhash 3.4.1 yarl 1.9.4 zstandard 0.19.0
@xiaohaiqing In my case, this is due to the older torch not being compatible with the newest torch.
torch==2.1.1
xformers
, downgrade xformers==0.0.23
.Hoping this will help you.
Your current environment
How you are installing vllm
请问这是什么原因呢?