search

vllm-project / vllm

A high-throughput and memory-efficient inference and serving engine for LLMs
https://docs.vllm.ai
Apache License 2.0
26.09k stars 3.82k forks source link

[Misc]: Source code compilation vllm failed Installation Environment: A40-48G python=3.11 cuda-12.1 #6683

Open Micla-SHL opened 1 month ago

Micla-SHL commented 1 month ago

Anything you want to discuss about vllm.

(vllm) fm2024@fm2024:~/Micla/Project/vllm$ pip install -e .
Obtaining file:///home/fm2024/Micla/Project/vllm
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build editable ... done
  Preparing editable metadata (pyproject.toml) ... done
Requirement already satisfied: cmake>=3.21 in /home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages (from vllm==0.5.3) (3.30.1)
Requirement already satisfied: ninja in /home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages (from vllm==0.5.3) (1.11.1.1)
Requirement already satisfied: psutil in /home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages (from vllm==0.5.3) (6.0.0)
Requirement already satisfied: sentencepiece in /home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages (from vllm==0.5.3) (0.2.0)
Requirement already satisfied: numpy<2.0.0 in /home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages (from vllm==0.5.3) (1.26.4)
Requirement already satisfied: requests in /home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages (from vllm==0.5.3) (2.32.3)
Requirement already satisfied: tqdm in /home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages (from vllm==0.5.3) (4.66.4)
Requirement already satisfied: py-cpuinfo in /home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages (from vllm==0.5.3) (9.0.0)
Requirement already satisfied: transformers>=4.42.4 in /home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages (from vllm==0.5.3) (4.42.4)
Requirement already satisfied: tokenizers>=0.19.1 in /home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages (from vllm==0.5.3) (0.19.1)
Requirement already satisfied: fastapi in /home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages (from vllm==0.5.3) (0.111.1)
Requirement already satisfied: aiohttp in /home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages (from vllm==0.5.3) (3.9.5)
Requirement already satisfied: openai in /home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages (from vllm==0.5.3) (1.37.0)
Requirement already satisfied: uvicorn[standard] in /home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages (from vllm==0.5.3) (0.30.3)
Requirement already satisfied: pydantic>=2.0 in /home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages (from vllm==0.5.3) (2.8.2)
Requirement already satisfied: pillow in /home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages (from vllm==0.5.3) (10.4.0)
Requirement already satisfied: prometheus-client>=0.18.0 in /home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages (from vllm==0.5.3) (0.20.0)
Requirement already satisfied: prometheus-fastapi-instrumentator>=7.0.0 in /home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages (from vllm==0.5.3) (7.0.0)
Requirement already satisfied: tiktoken>=0.6.0 in /home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages (from vllm==0.5.3) (0.7.0)
Requirement already satisfied: lm-format-enforcer==0.10.3 in /home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages (from vllm==0.5.3) (0.10.3)
Requirement already satisfied: outlines<0.1,>=0.0.43 in /home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages (from vllm==0.5.3) (0.0.46)
Requirement already satisfied: typing-extensions in /home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages (from vllm==0.5.3) (4.11.0)
Requirement already satisfied: filelock>=3.10.4 in /home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages (from vllm==0.5.3) (3.15.4)
Requirement already satisfied: pyzmq in /home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages (from vllm==0.5.3) (26.0.3)
Requirement already satisfied: ray>=2.9 in /home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages (from vllm==0.5.3) (2.32.0)
Requirement already satisfied: nvidia-ml-py in /home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages (from vllm==0.5.3) (12.555.43)
Requirement already satisfied: torch==2.3.1 in /home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages (from vllm==0.5.3) (2.3.1)
Requirement already satisfied: torchvision==0.18.1 in /home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages (from vllm==0.5.3) (0.18.1)
Requirement already satisfied: xformers==0.0.27 in /home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages (from vllm==0.5.3) (0.0.27)
Requirement already satisfied: vllm-flash-attn==2.5.9.post1 in /home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages (from vllm==0.5.3) (2.5.9.post1)
Requirement already satisfied: interegular>=0.3.2 in /home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages (from lm-format-enforcer==0.10.3->vllm==0.5.3) (0.3.3)
Requirement already satisfied: packaging in /home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages (from lm-format-enforcer==0.10.3->vllm==0.5.3) (24.1)
Requirement already satisfied: pyyaml in /home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages (from lm-format-enforcer==0.10.3->vllm==0.5.3) (6.0.1)
Requirement already satisfied: sympy in /home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages (from torch==2.3.1->vllm==0.5.3) (1.12)
Requirement already satisfied: networkx in /home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages (from torch==2.3.1->vllm==0.5.3) (3.3)
Requirement already satisfied: jinja2 in /home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages (from torch==2.3.1->vllm==0.5.3) (3.1.4)
Requirement already satisfied: fsspec in /home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages (from torch==2.3.1->vllm==0.5.3) (2024.5.0)
Requirement already satisfied: nvidia-cuda-nvrtc-cu12==12.1.105 in /home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages (from torch==2.3.1->vllm==0.5.3) (12.1.105)
Requirement already satisfied: nvidia-cuda-runtime-cu12==12.1.105 in /home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages (from torch==2.3.1->vllm==0.5.3) (12.1.105)
Requirement already satisfied: nvidia-cuda-cupti-cu12==12.1.105 in /home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages (from torch==2.3.1->vllm==0.5.3) (12.1.105)
Requirement already satisfied: nvidia-cudnn-cu12==8.9.2.26 in /home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages (from torch==2.3.1->vllm==0.5.3) (8.9.2.26)
Requirement already satisfied: nvidia-cublas-cu12==12.1.3.1 in /home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages (from torch==2.3.1->vllm==0.5.3) (12.1.3.1)
Requirement already satisfied: nvidia-cufft-cu12==11.0.2.54 in /home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages (from torch==2.3.1->vllm==0.5.3) (11.0.2.54)
Requirement already satisfied: nvidia-curand-cu12==10.3.2.106 in /home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages (from torch==2.3.1->vllm==0.5.3) (10.3.2.106)
Requirement already satisfied: nvidia-cusolver-cu12==11.4.5.107 in /home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages (from torch==2.3.1->vllm==0.5.3) (11.4.5.107)
Requirement already satisfied: nvidia-cusparse-cu12==12.1.0.106 in /home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages (from torch==2.3.1->vllm==0.5.3) (12.1.0.106)
Requirement already satisfied: nvidia-nccl-cu12==2.20.5 in /home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages (from torch==2.3.1->vllm==0.5.3) (2.20.5)
Requirement already satisfied: nvidia-nvtx-cu12==12.1.105 in /home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages (from torch==2.3.1->vllm==0.5.3) (12.1.105)
Requirement already satisfied: triton==2.3.1 in /home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages (from torch==2.3.1->vllm==0.5.3) (2.3.1)
Requirement already satisfied: nvidia-nvjitlink-cu12 in /home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages (from nvidia-cusolver-cu12==11.4.5.107->torch==2.3.1->vllm==0.5.3) (12.5.82)
Requirement already satisfied: lark in /home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages (from outlines<0.1,>=0.0.43->vllm==0.5.3) (1.1.9)
Requirement already satisfied: nest-asyncio in /home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages (from outlines<0.1,>=0.0.43->vllm==0.5.3) (1.6.0)
Requirement already satisfied: cloudpickle in /home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages (from outlines<0.1,>=0.0.43->vllm==0.5.3) (3.0.0)
Requirement already satisfied: diskcache in /home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages (from outlines<0.1,>=0.0.43->vllm==0.5.3) (5.6.3)
Requirement already satisfied: numba in /home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages (from outlines<0.1,>=0.0.43->vllm==0.5.3) (0.60.0)
Requirement already satisfied: referencing in /home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages (from outlines<0.1,>=0.0.43->vllm==0.5.3) (0.35.1)
Requirement already satisfied: jsonschema in /home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages (from outlines<0.1,>=0.0.43->vllm==0.5.3) (4.23.0)
Requirement already satisfied: datasets in /home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages (from outlines<0.1,>=0.0.43->vllm==0.5.3) (2.20.0)
Requirement already satisfied: pycountry in /home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages (from outlines<0.1,>=0.0.43->vllm==0.5.3) (24.6.1)
Requirement already satisfied: pyairports in /home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages (from outlines<0.1,>=0.0.43->vllm==0.5.3) (2.1.1)
Requirement already satisfied: starlette<1.0.0,>=0.30.0 in /home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages (from prometheus-fastapi-instrumentator>=7.0.0->vllm==0.5.3) (0.37.2)
Requirement already satisfied: annotated-types>=0.4.0 in /home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages (from pydantic>=2.0->vllm==0.5.3) (0.7.0)
Requirement already satisfied: pydantic-core==2.20.1 in /home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages (from pydantic>=2.0->vllm==0.5.3) (2.20.1)
Requirement already satisfied: click>=7.0 in /home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages (from ray>=2.9->vllm==0.5.3) (8.1.7)
Requirement already satisfied: msgpack<2.0.0,>=1.0.0 in /home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages (from ray>=2.9->vllm==0.5.3) (1.0.8)
Requirement already satisfied: protobuf!=3.19.5,>=3.15.3 in /home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages (from ray>=2.9->vllm==0.5.3) (5.27.2)
Requirement already satisfied: aiosignal in /home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages (from ray>=2.9->vllm==0.5.3) (1.3.1)
Requirement already satisfied: frozenlist in /home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages (from ray>=2.9->vllm==0.5.3) (1.4.1)
Requirement already satisfied: regex>=2022.1.18 in /home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages (from tiktoken>=0.6.0->vllm==0.5.3) (2024.5.15)
Requirement already satisfied: charset-normalizer<4,>=2 in /home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages (from requests->vllm==0.5.3) (2.0.4)
Requirement already satisfied: idna<4,>=2.5 in /home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages (from requests->vllm==0.5.3) (3.7)
Requirement already satisfied: urllib3<3,>=1.21.1 in /home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages (from requests->vllm==0.5.3) (2.2.2)
Requirement already satisfied: certifi>=2017.4.17 in /home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages (from requests->vllm==0.5.3) (2024.7.4)
Requirement already satisfied: huggingface-hub<1.0,>=0.16.4 in /home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages (from tokenizers>=0.19.1->vllm==0.5.3) (0.24.0)
Requirement already satisfied: safetensors>=0.4.1 in /home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages (from transformers>=4.42.4->vllm==0.5.3) (0.4.3)
Requirement already satisfied: attrs>=17.3.0 in /home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages (from aiohttp->vllm==0.5.3) (23.2.0)
Requirement already satisfied: multidict<7.0,>=4.5 in /home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages (from aiohttp->vllm==0.5.3) (6.0.5)
Requirement already satisfied: yarl<2.0,>=1.0 in /home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages (from aiohttp->vllm==0.5.3) (1.9.4)
Requirement already satisfied: fastapi-cli>=0.0.2 in /home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages (from fastapi->vllm==0.5.3) (0.0.4)
Requirement already satisfied: httpx>=0.23.0 in /home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages (from fastapi->vllm==0.5.3) (0.27.0)
Requirement already satisfied: python-multipart>=0.0.7 in /home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages (from fastapi->vllm==0.5.3) (0.0.9)
Requirement already satisfied: email_validator>=2.0.0 in /home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages (from fastapi->vllm==0.5.3) (2.2.0)
Requirement already satisfied: h11>=0.8 in /home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages (from uvicorn[standard]->vllm==0.5.3) (0.14.0)
Requirement already satisfied: httptools>=0.5.0 in /home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages (from uvicorn[standard]->vllm==0.5.3) (0.6.1)
Requirement already satisfied: python-dotenv>=0.13 in /home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages (from uvicorn[standard]->vllm==0.5.3) (1.0.1)
Requirement already satisfied: uvloop!=0.15.0,!=0.15.1,>=0.14.0 in /home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages (from uvicorn[standard]->vllm==0.5.3) (0.19.0)
Requirement already satisfied: watchfiles>=0.13 in /home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages (from uvicorn[standard]->vllm==0.5.3) (0.22.0)
Requirement already satisfied: websockets>=10.4 in /home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages (from uvicorn[standard]->vllm==0.5.3) (12.0)
Requirement already satisfied: anyio<5,>=3.5.0 in /home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages (from openai->vllm==0.5.3) (4.4.0)
Requirement already satisfied: distro<2,>=1.7.0 in /home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages (from openai->vllm==0.5.3) (1.9.0)
Requirement already satisfied: sniffio in /home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages (from openai->vllm==0.5.3) (1.3.1)
Requirement already satisfied: dnspython>=2.0.0 in /home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages (from email_validator>=2.0.0->fastapi->vllm==0.5.3) (2.6.1)
Requirement already satisfied: typer>=0.12.3 in /home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages (from fastapi-cli>=0.0.2->fastapi->vllm==0.5.3) (0.12.3)
Requirement already satisfied: httpcore==1.* in /home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages (from httpx>=0.23.0->fastapi->vllm==0.5.3) (1.0.5)
Requirement already satisfied: MarkupSafe>=2.0 in /home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages (from jinja2->torch==2.3.1->vllm==0.5.3) (2.1.3)
Requirement already satisfied: pyarrow>=15.0.0 in /home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages (from datasets->outlines<0.1,>=0.0.43->vllm==0.5.3) (17.0.0)
Requirement already satisfied: pyarrow-hotfix in /home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages (from datasets->outlines<0.1,>=0.0.43->vllm==0.5.3) (0.6)
Requirement already satisfied: dill<0.3.9,>=0.3.0 in /home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages (from datasets->outlines<0.1,>=0.0.43->vllm==0.5.3) (0.3.8)
Requirement already satisfied: pandas in /home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages (from datasets->outlines<0.1,>=0.0.43->vllm==0.5.3) (2.2.2)
Requirement already satisfied: xxhash in /home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages (from datasets->outlines<0.1,>=0.0.43->vllm==0.5.3) (3.4.1)
Requirement already satisfied: multiprocess in /home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages (from datasets->outlines<0.1,>=0.0.43->vllm==0.5.3) (0.70.16)
Requirement already satisfied: jsonschema-specifications>=2023.03.6 in /home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages (from jsonschema->outlines<0.1,>=0.0.43->vllm==0.5.3) (2023.12.1)
Requirement already satisfied: rpds-py>=0.7.1 in /home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages (from jsonschema->outlines<0.1,>=0.0.43->vllm==0.5.3) (0.19.0)
Requirement already satisfied: llvmlite<0.44,>=0.43.0dev0 in /home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages (from numba->outlines<0.1,>=0.0.43->vllm==0.5.3) (0.43.0)
Requirement already satisfied: mpmath>=0.19 in /home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages (from sympy->torch==2.3.1->vllm==0.5.3) (1.3.0)
Requirement already satisfied: shellingham>=1.3.0 in /home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages (from typer>=0.12.3->fastapi-cli>=0.0.2->fastapi->vllm==0.5.3) (1.5.4)
Requirement already satisfied: rich>=10.11.0 in /home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages (from typer>=0.12.3->fastapi-cli>=0.0.2->fastapi->vllm==0.5.3) (13.7.1)
Requirement already satisfied: python-dateutil>=2.8.2 in /home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages (from pandas->datasets->outlines<0.1,>=0.0.43->vllm==0.5.3) (2.9.0.post0)
Requirement already satisfied: pytz>=2020.1 in /home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages (from pandas->datasets->outlines<0.1,>=0.0.43->vllm==0.5.3) (2024.1)
Requirement already satisfied: tzdata>=2022.7 in /home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages (from pandas->datasets->outlines<0.1,>=0.0.43->vllm==0.5.3) (2024.1)
Requirement already satisfied: six>=1.5 in /home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages (from python-dateutil>=2.8.2->pandas->datasets->outlines<0.1,>=0.0.43->vllm==0.5.3) (1.16.0)
Requirement already satisfied: markdown-it-py>=2.2.0 in /home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages (from rich>=10.11.0->typer>=0.12.3->fastapi-cli>=0.0.2->fastapi->vllm==0.5.3) (3.0.0)
Requirement already satisfied: pygments<3.0.0,>=2.13.0 in /home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages (from rich>=10.11.0->typer>=0.12.3->fastapi-cli>=0.0.2->fastapi->vllm==0.5.3) (2.18.0)
Requirement already satisfied: mdurl~=0.1 in /home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages (from markdown-it-py>=2.2.0->rich>=10.11.0->typer>=0.12.3->fastapi-cli>=0.0.2->fastapi->vllm==0.5.3) (0.1.2)
Building wheels for collected packages: vllm
  Building editable for vllm (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Building editable for vllm (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [215 lines of output]
      /tmp/pip-build-env-h2w2qiuk/overlay/lib/python3.11/site-packages/torch/cuda/__init__.py:118: UserWarning: CUDA initialization: CUDA unknown error - this may be due to an incorrectly set up environment, e.g. changing env variable CUDA_VISIBLE_DEVICES after program start. Setting the available devices to be zero. (Triggered internally at ../c10/cuda/CUDAFunctions.cpp:108.)
        return torch._C._cuda_getDeviceCount() > 0
      No CUDA runtime is found, using CUDA_HOME='/usr/local/cuda-12.1'
      running editable_wheel
      creating /tmp/pip-wheel-_jyhrlo9/.tmp-cexd5u49/vllm.egg-info
      writing /tmp/pip-wheel-_jyhrlo9/.tmp-cexd5u49/vllm.egg-info/PKG-INFO
      writing dependency_links to /tmp/pip-wheel-_jyhrlo9/.tmp-cexd5u49/vllm.egg-info/dependency_links.txt
      writing entry points to /tmp/pip-wheel-_jyhrlo9/.tmp-cexd5u49/vllm.egg-info/entry_points.txt
      writing requirements to /tmp/pip-wheel-_jyhrlo9/.tmp-cexd5u49/vllm.egg-info/requires.txt
      writing top-level names to /tmp/pip-wheel-_jyhrlo9/.tmp-cexd5u49/vllm.egg-info/top_level.txt
      writing manifest file '/tmp/pip-wheel-_jyhrlo9/.tmp-cexd5u49/vllm.egg-info/SOURCES.txt'
      reading manifest file '/tmp/pip-wheel-_jyhrlo9/.tmp-cexd5u49/vllm.egg-info/SOURCES.txt'
      reading manifest template 'MANIFEST.in'
      adding license file 'LICENSE'
      writing manifest file '/tmp/pip-wheel-_jyhrlo9/.tmp-cexd5u49/vllm.egg-info/SOURCES.txt'
      creating '/tmp/pip-wheel-_jyhrlo9/.tmp-cexd5u49/vllm-0.5.3.dist-info'
      creating /tmp/pip-wheel-_jyhrlo9/.tmp-cexd5u49/vllm-0.5.3.dist-info/WHEEL
      running build_py
      running build_ext
      -- The CXX compiler identification is GNU 10.5.0
      -- Detecting CXX compiler ABI info
      -- Detecting CXX compiler ABI info - done
      -- Check for working CXX compiler: /usr/bin/c++ - skipped
      -- Detecting CXX compile features
      -- Detecting CXX compile features - done
      -- Build type: RelWithDebInfo
      -- Target device: cuda
      -- Found Python: /home/fm2024/miniconda3/envs/vllm/bin/python (found version "3.11.9") found components: Interpreter Development.Module Development.SABIModule
      -- Found python matching: /home/fm2024/miniconda3/envs/vllm/bin/python.
      -- Found CUDA: /usr/local/cuda-12.1 (found version "12.1")
      -- The CUDA compiler identification is NVIDIA 12.1.66
      -- Detecting CUDA compiler ABI info
      -- Detecting CUDA compiler ABI info - done
      -- Check for working CUDA compiler: /usr/local/cuda-12.1/bin/nvcc - skipped
      -- Detecting CUDA compile features
      -- Detecting CUDA compile features - done
      -- Found CUDAToolkit: /usr/local/cuda-12.1/include (found version "12.1.66")
      -- Performing Test CMAKE_HAVE_LIBC_PTHREAD
      -- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
      -- Looking for pthread_create in pthreads
      -- Looking for pthread_create in pthreads - not found
      -- Looking for pthread_create in pthread
      -- Looking for pthread_create in pthread - found
      -- Found Threads: TRUE
      -- Caffe2: CUDA detected: 12.1
      -- Caffe2: CUDA nvcc is: /usr/local/cuda-12.1/bin/nvcc
      -- Caffe2: CUDA toolkit directory: /usr/local/cuda-12.1
      -- Caffe2: Header version is: 12.1
      -- /usr/local/cuda-12.1/lib64/libnvrtc.so shorthash is d540eb83
      -- USE_CUDNN is set to 0. Compiling without cuDNN support
      -- USE_CUSPARSELT is set to 0. Compiling without cuSPARSELt support
      -- Automatic GPU detection failed. Building for common architectures.
      -- Autodetected CUDA architecture(s): 5.0;8.0;8.6;8.9;9.0;9.0a
      CMake Error at /tmp/pip-build-env-h2w2qiuk/overlay/lib/python3.11/site-packages/torch/share/cmake/Caffe2/Modules_CUDA_fix/upstream/FindCUDA/select_compute_arch.cmake:227 (message):
        Unknown CUDA Architecture Name 9.0a in CUDA_SELECT_NVCC_ARCH_FLAGS
      Call Stack (most recent call first):
        /tmp/pip-build-env-h2w2qiuk/overlay/lib/python3.11/site-packages/torch/share/cmake/Caffe2/public/utils.cmake:401 (cuda_select_nvcc_arch_flags)
        /tmp/pip-build-env-h2w2qiuk/overlay/lib/python3.11/site-packages/torch/share/cmake/Caffe2/public/cuda.cmake:342 (torch_cuda_get_nvcc_gencode_flag)
        /tmp/pip-build-env-h2w2qiuk/overlay/lib/python3.11/site-packages/torch/share/cmake/Caffe2/Caffe2Config.cmake:87 (include)
        /tmp/pip-build-env-h2w2qiuk/overlay/lib/python3.11/site-packages/torch/share/cmake/Torch/TorchConfig.cmake:68 (find_package)
        CMakeLists.txt:67 (find_package)

      CMake Error at /tmp/pip-build-env-h2w2qiuk/overlay/lib/python3.11/site-packages/torch/share/cmake/Caffe2/Modules_CUDA_fix/upstream/FindCUDA/select_compute_arch.cmake:231 (message):
        arch_bin wasn't set for some reason
      Call Stack (most recent call first):
        /tmp/pip-build-env-h2w2qiuk/overlay/lib/python3.11/site-packages/torch/share/cmake/Caffe2/public/utils.cmake:401 (cuda_select_nvcc_arch_flags)
        /tmp/pip-build-env-h2w2qiuk/overlay/lib/python3.11/site-packages/torch/share/cmake/Caffe2/public/cuda.cmake:342 (torch_cuda_get_nvcc_gencode_flag)
        /tmp/pip-build-env-h2w2qiuk/overlay/lib/python3.11/site-packages/torch/share/cmake/Caffe2/Caffe2Config.cmake:87 (include)
        /tmp/pip-build-env-h2w2qiuk/overlay/lib/python3.11/site-packages/torch/share/cmake/Torch/TorchConfig.cmake:68 (find_package)
        CMakeLists.txt:67 (find_package)

      -- Added CUDA NVCC flags for: -gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_89,code=sm_89;-gencode;arch=compute_90,code=sm_90
      CMake Warning at /tmp/pip-build-env-h2w2qiuk/overlay/lib/python3.11/site-packages/torch/share/cmake/Torch/TorchConfig.cmake:22 (message):
        static library kineto_LIBRARY-NOTFOUND not found.
      Call Stack (most recent call first):
        /tmp/pip-build-env-h2w2qiuk/overlay/lib/python3.11/site-packages/torch/share/cmake/Torch/TorchConfig.cmake:127 (append_torchlib_if_found)
        CMakeLists.txt:67 (find_package)

      -- Found Torch: /tmp/pip-build-env-h2w2qiuk/overlay/lib/python3.11/site-packages/torch/lib/libtorch.so
      -- CUDA supported arches: 7.0;7.5;8.0;8.6;8.9;9.0
      -- discarding unsupported CUDA arch 3.11.
      -- CUDA target arches: 80-real;86-real;89-real;90-real
      -- CMake Version: 3.30.1
      -- CUTLASS 3.5.0
      -- CUDART: /usr/local/cuda-12.1/lib64/libcudart.so
      -- CUDA Driver: /usr/local/cuda-12.1/lib64/stubs/libcuda.so
      -- NVRTC: /usr/local/cuda-12.1/lib64/libnvrtc.so
      -- Default Install Location: install
      -- Found Python3: /home/fm2024/miniconda3/envs/vllm/bin/python3.11 (found suitable version "3.11.9", minimum required is "3.5") found components: Interpreter
      -- CUDA Compilation Architectures: 70;72;75;80;86;87;89;90;90a
      -- Enable caching of reference results in conv unit tests
      -- Enable rigorous conv problem sizes in conv unit tests
      -- Using NVCC flags: --expt-relaxed-constexpr;-DCUTLASS_TEST_LEVEL=0;-DCUTLASS_TEST_ENABLE_CACHED_RESULTS=1;-DCUTLASS_CONV_UNIT_TEST_RIGOROUS_SIZE_ENABLED=1;-DCUTLASS_DEBUG_TRACE_LEVEL=0;-Xcompiler=-Wconversion;-Xcompiler=-fno-strict-aliasing;-lineinfo
      fatal: not a git repository (or any of the parent directories): .git
      -- CUTLASS Revision: Unable to detect, Git returned code 128.
      -- Configuring cublas ...
      -- cuBLAS Disabled.
      -- Configuring cuBLAS ... done.
      -- Completed generation of library instances. See /tmp/tmpv9d9ogrg.build-temp/_deps/cutlass-build/tools/library/library_instance_generation.log for more information.
      -- Punica target arches: 80-real;86-real;89-real;90-real
      -- Enabling C extension.
      -- Enabling moe extension.
      -- Configuring incomplete, errors occurred!
      Traceback (most recent call last):
        File "/tmp/pip-build-env-h2w2qiuk/overlay/lib/python3.11/site-packages/setuptools/command/editable_wheel.py", line 153, in run
          self._create_wheel_file(bdist_wheel)
        File "/tmp/pip-build-env-h2w2qiuk/overlay/lib/python3.11/site-packages/setuptools/command/editable_wheel.py", line 355, in _create_wheel_file
          files, mapping = self._run_build_commands(dist_name, unpacked, lib, tmp)
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-h2w2qiuk/overlay/lib/python3.11/site-packages/setuptools/command/editable_wheel.py", line 278, in _run_build_commands
          self._run_build_subcommands()
        File "/tmp/pip-build-env-h2w2qiuk/overlay/lib/python3.11/site-packages/setuptools/command/editable_wheel.py", line 305, in _run_build_subcommands
          self.run_command(name)
        File "/tmp/pip-build-env-h2w2qiuk/overlay/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 316, in run_command
          self.distribution.run_command(command)
        File "/tmp/pip-build-env-h2w2qiuk/overlay/lib/python3.11/site-packages/setuptools/dist.py", line 956, in run_command
          super().run_command(command)
        File "/tmp/pip-build-env-h2w2qiuk/overlay/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 989, in run_command
          cmd_obj.run()
        File "/tmp/pip-build-env-h2w2qiuk/overlay/lib/python3.11/site-packages/setuptools/command/build_ext.py", line 93, in run
          _build_ext.run(self)
        File "/tmp/pip-build-env-h2w2qiuk/overlay/lib/python3.11/site-packages/setuptools/_distutils/command/build_ext.py", line 359, in run
          self.build_extensions()
        File "<string>", line 222, in build_extensions
        File "<string>", line 204, in configure
        File "/home/fm2024/miniconda3/envs/vllm/lib/python3.11/subprocess.py", line 413, in check_call
          raise CalledProcessError(retcode, cmd)
      subprocess.CalledProcessError: Command '['cmake', '/home/fm2024/Micla/Project/vllm', '-G', 'Ninja', '-DCMAKE_BUILD_TYPE=RelWithDebInfo', '-DCMAKE_LIBRARY_OUTPUT_DIRECTORY=/tmp/tmp00yzuxtu.build-lib/vllm', '-DCMAKE_ARCHIVE_OUTPUT_DIRECTORY=/tmp/tmpv9d9ogrg.build-temp', '-DVLLM_TARGET_DEVICE=cuda', '-DCMAKE_CXX_COMPILER_LAUNCHER=ccache', '-DCMAKE_CUDA_COMPILER_LAUNCHER=ccache', '-DVLLM_PYTHON_EXECUTABLE=/home/fm2024/miniconda3/envs/vllm/bin/python', '-DNVCC_THREADS=1', '-DCMAKE_JOB_POOL_COMPILE:STRING=compile', '-DCMAKE_JOB_POOLS:STRING=compile=48']' returned non-zero exit status 1.
      /tmp/pip-build-env-h2w2qiuk/overlay/lib/python3.11/site-packages/setuptools/_distutils/dist.py:989: _DebuggingTips: Problem in editable installation.
      !!

              ********************************************************************************
              An error happened while installing `vllm` in editable mode.

              The following steps are recommended to help debug this problem:

              - Try to install the project normally, without using the editable mode.
                Does the error still persist?
                (If it does, try fixing the problem before attempting the editable mode).
              - If you are using binary extensions, make sure you have all OS-level
                dependencies installed (e.g. compilers, toolchains, binary libraries, ...).
              - Try the latest version of setuptools (maybe the error was already fixed).
              - If you (or your project dependencies) are using any setuptools extension
                or customization, make sure they support the editable mode.

              After following the steps above, if the problem still persists and
              you think this is related to how setuptools handles editable installations,
              please submit a reproducible example
              (see https://stackoverflow.com/help/minimal-reproducible-example) to:

                  https://github.com/pypa/setuptools/issues

              See https://setuptools.pypa.io/en/latest/userguide/development_mode.html for details.
              ********************************************************************************

      !!
        cmd_obj.run()
      Traceback (most recent call last):
        File "/home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
        File "/home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/home/fm2024/miniconda3/envs/vllm/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 273, in build_editable
          return hook(wheel_directory, config_settings, metadata_directory)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-h2w2qiuk/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 453, in build_editable
          return self._build_with_temp_dir(
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-h2w2qiuk/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 397, in _build_with_temp_dir
          self.run_setup()
        File "/tmp/pip-build-env-h2w2qiuk/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 313, in run_setup
          exec(code, locals())
        File "<string>", line 459, in <module>
        File "/tmp/pip-build-env-h2w2qiuk/overlay/lib/python3.11/site-packages/setuptools/__init__.py", line 108, in setup
          return distutils.core.setup(**attrs)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-h2w2qiuk/overlay/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 184, in setup
          return run_commands(dist)
                 ^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-h2w2qiuk/overlay/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 200, in run_commands
          dist.run_commands()
        File "/tmp/pip-build-env-h2w2qiuk/overlay/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 970, in run_commands
          self.run_command(cmd)
        File "/tmp/pip-build-env-h2w2qiuk/overlay/lib/python3.11/site-packages/setuptools/dist.py", line 956, in run_command
          super().run_command(command)
        File "/tmp/pip-build-env-h2w2qiuk/overlay/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 989, in run_command
          cmd_obj.run()
        File "/tmp/pip-build-env-h2w2qiuk/overlay/lib/python3.11/site-packages/setuptools/command/editable_wheel.py", line 153, in run
          self._create_wheel_file(bdist_wheel)
        File "/tmp/pip-build-env-h2w2qiuk/overlay/lib/python3.11/site-packages/setuptools/command/editable_wheel.py", line 355, in _create_wheel_file
          files, mapping = self._run_build_commands(dist_name, unpacked, lib, tmp)
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-h2w2qiuk/overlay/lib/python3.11/site-packages/setuptools/command/editable_wheel.py", line 278, in _run_build_commands
          self._run_build_subcommands()
        File "/tmp/pip-build-env-h2w2qiuk/overlay/lib/python3.11/site-packages/setuptools/command/editable_wheel.py", line 305, in _run_build_subcommands
          self.run_command(name)
        File "/tmp/pip-build-env-h2w2qiuk/overlay/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 316, in run_command
          self.distribution.run_command(command)
        File "/tmp/pip-build-env-h2w2qiuk/overlay/lib/python3.11/site-packages/setuptools/dist.py", line 956, in run_command
          super().run_command(command)
        File "/tmp/pip-build-env-h2w2qiuk/overlay/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 989, in run_command
          cmd_obj.run()
        File "/tmp/pip-build-env-h2w2qiuk/overlay/lib/python3.11/site-packages/setuptools/command/build_ext.py", line 93, in run
          _build_ext.run(self)
        File "/tmp/pip-build-env-h2w2qiuk/overlay/lib/python3.11/site-packages/setuptools/_distutils/command/build_ext.py", line 359, in run
          self.build_extensions()
        File "<string>", line 222, in build_extensions
        File "<string>", line 204, in configure
        File "/home/fm2024/miniconda3/envs/vllm/lib/python3.11/subprocess.py", line 413, in check_call
          raise CalledProcessError(retcode, cmd)
      subprocess.CalledProcessError: Command '['cmake', '/home/fm2024/Micla/Project/vllm', '-G', 'Ninja', '-DCMAKE_BUILD_TYPE=RelWithDebInfo', '-DCMAKE_LIBRARY_OUTPUT_DIRECTORY=/tmp/tmp00yzuxtu.build-lib/vllm', '-DCMAKE_ARCHIVE_OUTPUT_DIRECTORY=/tmp/tmpv9d9ogrg.build-temp', '-DVLLM_TARGET_DEVICE=cuda', '-DCMAKE_CXX_COMPILER_LAUNCHER=ccache', '-DCMAKE_CUDA_COMPILER_LAUNCHER=ccache', '-DVLLM_PYTHON_EXECUTABLE=/home/fm2024/miniconda3/envs/vllm/bin/python', '-DNVCC_THREADS=1', '-DCMAKE_JOB_POOL_COMPILE:STRING=compile', '-DCMAKE_JOB_POOLS:STRING=compile=48']' returned non-zero exit status 1.
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building editable for vllm
Failed to build vllm
ERROR: ERROR: Failed to build installable wheels for some pyproject.toml based projects (vllm)

I successfully installed vllm and reasoning deployment on a 4090, and now I have switched to A40, and the compilation and installation has been failing, saying that there is a problem with my cuda, but I think there is no problem, and I have also done some verification. Now I have no idea, so I am here to ask for help

youkaichao commented 1 month ago

might be a bug of pytorch, which i fixed recently: https://github.com/pytorch/pytorch/pull/123243

the solution is to explicitly define TORCH_CUDA_ARCH_LIST environment variable, e.g. export TORCH_CUDA_ARCH_LIST=9.0a

Micla-SHL commented 1 month ago

might be a bug of pytorch, which i fixed recently: pytorch/pytorch#123243

the solution is to explicitly define TORCH_CUDA_ARCH_LIST environment variable, e.g. export TORCH_CUDA_ARCH_LIST=9.0a

Thanks, I now think it has nothing to do with vllm. My machine can run nvidia-smi, but torch.cuda.is_available()= False, so I reinstalled the graphics driver and it returned to normal.

Micla-SHL commented 1 month ago

I found that vllm should consume as many resources as possible: INFO 07-24 15:04:47 gpu_executor.py:84] # GPU blocks: 11937, # CPU blocks: 2048 This is inconsistent with the memory consumed when I run the same code on 4090. How can I set a limit when initializing it?