search

vllm-project / vllm

A high-throughput and memory-efficient inference and serving engine for LLMs
https://docs.vllm.ai
Apache License 2.0
30.8k stars 4.67k forks source link

Feature 'f16 arithemetic and compare instructions' requires .target sm_53 or higher #8708

Open shahizat opened 2 months ago

shahizat commented 2 months ago

Your current environment

Collecting environment information... PyTorch version: 2.3.0 Is debug build: False CUDA used to build PyTorch: 12.2 ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.4 LTS (aarch64) GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 Clang version: Could not collect CMake version: version 3.30.3 Libc version: glibc-2.35

Python version: 3.10.12 (main, Jul 29 2024, 16:56:48) [GCC 11.4.0] (64-bit runtime) Python platform: Linux-5.15.136-tegra-aarch64-with-glibc2.35 Is CUDA available: True CUDA runtime version: 12.2.140 CUDA_MODULE_LOADING set to: LAZY GPU models and configuration: GPU 0: Orin (nvgpu) Nvidia driver version: N/A cuDNN version: Probably one of the following: /usr/lib/aarch64-linux-gnu/libcudnn.so.8.9.4 /usr/lib/aarch64-linux-gnu/libcudnn_adv_infer.so.8.9.4 /usr/lib/aarch64-linux-gnu/libcudnn_adv_train.so.8.9.4 /usr/lib/aarch64-linux-gnu/libcudnn_cnn_infer.so.8.9.4 /usr/lib/aarch64-linux-gnu/libcudnn_cnn_train.so.8.9.4 /usr/lib/aarch64-linux-gnu/libcudnn_ops_infer.so.8.9.4 /usr/lib/aarch64-linux-gnu/libcudnn_ops_train.so.8.9.4 HIP runtime version: N/A MIOpen runtime version: N/A Is XNNPACK available: True

CPU: Architecture: aarch64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 12 On-line CPU(s) list: 0-11 Vendor ID: ARM Model name: Cortex-A78AE Model: 1 Thread(s) per core: 1 Core(s) per cluster: 4 Socket(s): - Cluster(s): 3 Stepping: r0p1 CPU max MHz: 2201.6001 CPU min MHz: 115.2000 BogoMIPS: 62.50 Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm lrcpc dcpop asimddp uscat ilrcpc flagm paca pacg L1d cache: 768 KiB (12 instances) L1i cache: 768 KiB (12 instances) L2 cache: 3 MiB (12 instances) L3 cache: 6 MiB (3 instances) NUMA node(s): 1 NUMA node0 CPU(s): 0-11 Vulnerability Gather data sampling: Not affected Vulnerability Itlb multihit: Not affected Vulnerability L1tf: Not affected Vulnerability Mds: Not affected Vulnerability Meltdown: Not affected Vulnerability Mmio stale data: Not affected Vulnerability Retbleed: Not affected Vulnerability Spec rstack overflow: Not affected Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl Vulnerability Spectre v1: Mitigation; __user pointer sanitization Vulnerability Spectre v2: Mitigation; CSV2, but not BHB Vulnerability Srbds: Not affected Vulnerability Tsx async abort: Not affected

Versions of relevant libraries: [pip3] mypy==1.11.2 [pip3] mypy-extensions==1.0.0 [pip3] numpy==1.26.4 [pip3] onnx==1.16.1 [pip3] onnx-graphsurgeon==0.3.12 [pip3] onnxruntime==1.19.2 [pip3] pyzmq==26.0.3 [pip3] sentence-transformers==3.0.1 [pip3] torch==2.3.0 [pip3] torchvision==0.18.0a0+6043bc2 [pip3] transformers==4.41.1 [pip3] tritonclient==2.48.0 [conda] Could not collect ROCM Version: Could not collect Neuron SDK Version: N/A vLLM Version: N/A vLLM Build Flags: CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled GPU Topology: Could not collect

How would you like to use vllm

Greeting to all,

I want to run inference on the Nvidia AGX Orin dev kit. I don't know how to integrate it with vllm. I was building it using below command:

DOCKER_BUILDKIT=1 sudo docker build . --target vllm-openai --tag vllm/vllm-openai --build-arg max_jobs=10 --build-arg nvcc_threads=8 --build-arg torch_cuda_arch_list="8.7"

The error log: `883.5 ptxas /tmp/tmpxft_000003ed_00000000-6_attention_kernels.ptx, line 4985278; error : Feature 'f16 arithemetic and compare instructions' requires .target sm_53 or higher 883.5 ptxas fatal : Ptx assembly aborted due to errors 883.5 ninja: build stopped: subcommand failed.

Dockerfile:136

135 | ENV CCACHE_DIR=/root/.cache/ccache 136 | >>> RUN --mount=type=cache,target=/root/.cache/ccache \ 137 | >>> --mount=type=cache,target=/root/.cache/pip \ 138 | >>> if [ "$USE_SCCACHE" != "1" ]; then \ 139 | >>> python3 setup.py bdist_wheel --dist-dir=dist --py-limited-api=cp38; \ 140 | >>> fi 141 |

ERROR: failed to solve: process "/bin/sh -c if [ \"$USE_SCCACHE\" != \"1\" ]; then python3 setup.py bdist_wheel --dist-dir=dist --py-limited-api=cp38; fi" did not complete successfully: exit code: 1`

Thanks in advance for your support!

Before submitting a new issue...

youkaichao commented 2 months ago

you are building for aarch64 . please try https://github.com/vllm-project/vllm/issues/2021#issuecomment-2366858271

gongchengli commented 2 months ago

Your current environment

Collecting environment information... PyTorch version: 2.3.0 Is debug build: False CUDA used to build PyTorch: 12.2 ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.4 LTS (aarch64) GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 Clang version: Could not collect CMake version: version 3.30.3 Libc version: glibc-2.35

Python version: 3.10.12 (main, Jul 29 2024, 16:56:48) [GCC 11.4.0] (64-bit runtime) Python platform: Linux-5.15.136-tegra-aarch64-with-glibc2.35 Is CUDA available: True CUDA runtime version: 12.2.140 CUDA_MODULE_LOADING set to: LAZY GPU models and configuration: GPU 0: Orin (nvgpu) Nvidia driver version: N/A cuDNN version: Probably one of the following: /usr/lib/aarch64-linux-gnu/libcudnn.so.8.9.4 /usr/lib/aarch64-linux-gnu/libcudnn_adv_infer.so.8.9.4 /usr/lib/aarch64-linux-gnu/libcudnn_adv_train.so.8.9.4 /usr/lib/aarch64-linux-gnu/libcudnn_cnn_infer.so.8.9.4 /usr/lib/aarch64-linux-gnu/libcudnn_cnn_train.so.8.9.4 /usr/lib/aarch64-linux-gnu/libcudnn_ops_infer.so.8.9.4 /usr/lib/aarch64-linux-gnu/libcudnn_ops_train.so.8.9.4 HIP runtime version: N/A MIOpen runtime version: N/A Is XNNPACK available: True

CPU: Architecture: aarch64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 12 On-line CPU(s) list: 0-11 Vendor ID: ARM Model name: Cortex-A78AE Model: 1 Thread(s) per core: 1 Core(s) per cluster: 4 Socket(s): - Cluster(s): 3 Stepping: r0p1 CPU max MHz: 2201.6001 CPU min MHz: 115.2000 BogoMIPS: 62.50 Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm lrcpc dcpop asimddp uscat ilrcpc flagm paca pacg L1d cache: 768 KiB (12 instances) L1i cache: 768 KiB (12 instances) L2 cache: 3 MiB (12 instances) L3 cache: 6 MiB (3 instances) NUMA node(s): 1 NUMA node0 CPU(s): 0-11 Vulnerability Gather data sampling: Not affected Vulnerability Itlb multihit: Not affected Vulnerability L1tf: Not affected Vulnerability Mds: Not affected Vulnerability Meltdown: Not affected Vulnerability Mmio stale data: Not affected Vulnerability Retbleed: Not affected Vulnerability Spec rstack overflow: Not affected Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl Vulnerability Spectre v1: Mitigation; __user pointer sanitization Vulnerability Spectre v2: Mitigation; CSV2, but not BHB Vulnerability Srbds: Not affected Vulnerability Tsx async abort: Not affected

Versions of relevant libraries: [pip3] mypy==1.11.2 [pip3] mypy-extensions==1.0.0 [pip3] numpy==1.26.4 [pip3] onnx==1.16.1 [pip3] onnx-graphsurgeon==0.3.12 [pip3] onnxruntime==1.19.2 [pip3] pyzmq==26.0.3 [pip3] sentence-transformers==3.0.1 [pip3] torch==2.3.0 [pip3] torchvision==0.18.0a0+6043bc2 [pip3] transformers==4.41.1 [pip3] tritonclient==2.48.0 [conda] Could not collect ROCM Version: Could not collect Neuron SDK Version: N/A vLLM Version: N/A vLLM Build Flags: CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled GPU Topology: Could not collect

How would you like to use vllm

Greeting to all,

I want to run inference on the Nvidia AGX Orin dev kit. I don't know how to integrate it with vllm. I was building it using below command:

DOCKER_BUILDKIT=1 sudo docker build . --target vllm-openai --tag vllm/vllm-openai --build-arg max_jobs=10 --build-arg nvcc_threads=8 --build-arg torch_cuda_arch_list="8.7"

The error log:

`883.5 ptxas /tmp/tmpxft_000003ed_00000000-6_attention_kernels.ptx, line 4985278; error : Feature 'f16 arithemetic and compare instructions' requires .target sm_53 or higher 883.5 ptxas fatal : Ptx assembly aborted due to errors 883.5 ninja: build stopped: subcommand failed.

Dockerfile:136

135 | ENV CCACHE_DIR=/root/.cache/ccache

136 | >>> RUN --mount=type=cache,target=/root/.cache/ccache 137 | >>> --mount=type=cache,target=/root/.cache/pip 138 | >>> if [ "$USE_SCCACHE" != "1" ]; then 139 | >>> python3 setup.py bdist_wheel --dist-dir=dist --py-limited-api=cp38; 140 | >>> fi 141 | ERROR: failed to solve: process "/bin/sh -c if [ "$USE_SCCACHE" != "1" ]; then python3 setup.py bdist_wheel --dist-dir=dist --py-limited-api=cp38; fi" did not complete successfully: exit code: 1`

Thanks in advance for your support!

Before submitting a new issue...

  • [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

meet the same problem, maybe try --build-arg torch_cuda_arch_list="8.6" instead of "8.7" ? I am still working on building it ...

shahizat commented 1 month ago

Hello @youkaichao, @gongchengli, I have some good news to share @dusty-nv and @johnnynunez have done a great job by porting vLLM onto Nvidia Jetson Orin devices.

For more information, you can check out this link: https://github.com/dusty-nv/jetson-containers/tree/dev/packages/llm/vllm

If someone is interested, I am share the command I'm using to run it. I'm still experimenting with parameters.

sudo docker run --runtime nvidia \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HUGGING_FACE_HUB_TOKEN=YOUR_HF_TOKEN" \
    --env "HF_HOME=/root/.cache/huggingface" \
   --env "TRANSFORMERS_CACHE=/root/.cache/huggingface" \
    -p 8000:8000 \
    --ipc=host \
    dustynv/vllm:r36.4.0 \
    vllm serve /root/.cache/huggingface/models--meta-llama--Llama-3.1-8B-Instruct/snapshots/0e9e39f249a16976918f6564b8830bc894c89659 --gpu-memory-utilization 0.5  --enforce-eager --enable-prefix-caching --enable-chunked-prefill