Open noamwies opened 2 weeks ago
No response
VLLLm crash when running Qwen/Qwen2.5-Coder-32B-Instruct on two H100 GPUs
Command for reprduce:
sudo docker run --gpus all -e CUDA_VISIBLE_DEVICES=0,1 -e HUGGING_FACE_HUB_TOKEN=$HUGGING_FACE_HUB_TOKEN --ipc=host -p 8000:8000 -v /dev/shm:/dev/shm -v /nvme/.tritoncache/:/root/.triton/cache -v /nvme:/nvme -v $HF_HOME:/root/.cache/huggingface vllm/vllm-openai:v0.6.3.post1 --model Qwen/Qwen2.5-Coder-32B-Instruct --tensor-parallel-size 2 --disable-log-stats --dtype bfloat16 --max-model-len 8192 --swap-space 128 --disable-log-requests --enable-chunked-prefill=false
Error message:
(VllmWorkerProcess pid=347) ERROR 11-13 05:09:12 multiproc_worker_utils.py:229] Exception in worker VllmWorkerProcess while processing method start_worker_execution_loop. (VllmWorkerProcess pid=347) ERROR 11-13 05:09:12 multiproc_worker_utils.py:229] Traceback (most recent call last): (VllmWorkerProcess pid=347) ERROR 11-13 05:09:12 multiproc_worker_utils.py:229] File "/usr/local/lib/python3.12/dist-packages/vllm/executor/multiproc_worker_utils.py", line 223, in _run_worker_process (VllmWorkerProcess pid=347) ERROR 11-13 05:09:12 multiproc_worker_utils.py:229] output = executor(*args, **kwargs) (VllmWorkerProcess pid=347) ERROR 11-13 05:09:12 multiproc_worker_utils.py:229] ^^^^^^^^^^^^^^^^^^^^^^^^^ (VllmWorkerProcess pid=347) ERROR 11-13 05:09:12 multiproc_worker_utils.py:229] File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context (VllmWorkerProcess pid=347) ERROR 11-13 05:09:12 multiproc_worker_utils.py:229] return func(*args, **kwargs) (VllmWorkerProcess pid=347) ERROR 11-13 05:09:12 multiproc_worker_utils.py:229] ^^^^^^^^^^^^^^^^^^^^^ (VllmWorkerProcess pid=347) ERROR 11-13 05:09:12 multiproc_worker_utils.py:229] File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker_base.py", line 69, in start_worker_execution_loop (VllmWorkerProcess pid=347) ERROR 11-13 05:09:12 multiproc_worker_utils.py:229] output = self.execute_model(execute_model_req=None) (VllmWorkerProcess pid=347) ERROR 11-13 05:09:12 multiproc_worker_utils.py:229] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (VllmWorkerProcess pid=347) ERROR 11-13 05:09:12 multiproc_worker_utils.py:229] File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker_base.py", line 303, in execute_model (VllmWorkerProcess pid=347) ERROR 11-13 05:09:12 multiproc_worker_utils.py:229] inputs = self.prepare_input(execute_model_req) (VllmWorkerProcess pid=347) ERROR 11-13 05:09:12 multiproc_worker_utils.py:229] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (VllmWorkerProcess pid=347) ERROR 11-13 05:09:12 multiproc_worker_utils.py:229] File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker_base.py", line 293, in prepare_input (VllmWorkerProcess pid=347) ERROR 11-13 05:09:12 multiproc_worker_utils.py:229] return self._get_worker_input_from_broadcast() (VllmWorkerProcess pid=347) ERROR 11-13 05:09:12 multiproc_worker_utils.py:229] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (VllmWorkerProcess pid=347) ERROR 11-13 05:09:12 multiproc_worker_utils.py:229] File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker_base.py", line 237, in _get_worker_input_from_broadcast (VllmWorkerProcess pid=347) ERROR 11-13 05:09:12 multiproc_worker_utils.py:229] self.model_runner.make_model_input_from_broadcasted_tensor_dict( (VllmWorkerProcess pid=347) ERROR 11-13 05:09:12 multiproc_worker_utils.py:229] File "/usr/local/lib/python3.12/dist-packages/vllm/worker/model_runner.py", line 1561, in make_model_input_from_broadcasted_tensor_dict (VllmWorkerProcess pid=347) ERROR 11-13 05:09:12 multiproc_worker_utils.py:229] ModelInputForGPUWithSamplingMetadata.from_broadcasted_tensor_dict( (VllmWorkerProcess pid=347) ERROR 11-13 05:09:12 multiproc_worker_utils.py:229] File "/usr/local/lib/python3.12/dist-packages/vllm/worker/model_runner.py", line 175, in from_broadcasted_tensor_dict (VllmWorkerProcess pid=347) ERROR 11-13 05:09:12 multiproc_worker_utils.py:229] tensor_dict = _init_attn_metadata_from_tensor_dict( (VllmWorkerProcess pid=347) ERROR 11-13 05:09:12 multiproc_worker_utils.py:229] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (VllmWorkerProcess pid=347) ERROR 11-13 05:09:12 multiproc_worker_utils.py:229] File "/usr/local/lib/python3.12/dist-packages/vllm/worker/model_runner_base.py", line 53, in _init_attn_metadata_from_tensor_dict (VllmWorkerProcess pid=347) ERROR 11-13 05:09:12 multiproc_worker_utils.py:229] attn_metadata = attn_backend.make_metadata(**valid_attn_kwargs) (VllmWorkerProcess pid=347) ERROR 11-13 05:09:12 multiproc_worker_utils.py:229] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (VllmWorkerProcess pid=347) ERROR 11-13 05:09:12 multiproc_worker_utils.py:229] File "/usr/local/lib/python3.12/dist-packages/vllm/attention/backends/abstract.py", line 47, in make_metadata (VllmWorkerProcess pid=347) ERROR 11-13 05:09:12 multiproc_worker_utils.py:229] return cls.get_metadata_cls()(*args, **kwargs) (VllmWorkerProcess pid=347) ERROR 11-13 05:09:12 multiproc_worker_utils.py:229] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (VllmWorkerProcess pid=347) ERROR 11-13 05:09:12 multiproc_worker_utils.py:229] TypeError: FlashAttentionMetadata.__init__() missing 15 required positional arguments: 'num_prefills', 'num_prefill_tokens', 'num_decode_tokens', 'slot_mapping', 'seq_lens', 'seq_lens_tensor', 'max_query_len', 'max_decode_query_len', 'max_prefill_seq_len', 'max_decode_seq_len', 'query_start_loc', 'seq_start_loc', 'context_lens_tensor', 'block_tables', and 'use_cuda_graph'
did u solve it?
vanshaw2017
commented
4 days ago
Your current environment
The output of `python collect_env.py`
```text PyTorch version: 2.4.0+cu121 Is debug build: False CUDA used to build PyTorch: 12.1 ROCM used to build PyTorch: N/A OS: Ubuntu 22.04.4 LTS (x86_64) GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 Clang version: Could not collect CMake version: Could not collect Libc version: glibc-2.35 Python version: 3.12.7 (main, Oct 1 2024, 08:52:12) [GCC 11.4.0] (64-bit runtime) Python platform: Linux-6.5.0-1023-aws-x86_64-with-glibc2.35 Is CUDA available: True CUDA runtime version: Could not collect CUDA_MODULE_LOADING set to: LAZY GPU models and configuration: GPU 0: NVIDIA H100 80GB HBM3 GPU 1: NVIDIA H100 80GB HBM3 GPU 2: NVIDIA H100 80GB HBM3 GPU 3: NVIDIA H100 80GB HBM3 GPU 4: NVIDIA H100 80GB HBM3 GPU 5: NVIDIA H100 80GB HBM3 GPU 6: NVIDIA H100 80GB HBM3 GPU 7: NVIDIA H100 80GB HBM3 Nvidia driver version: 535.183.01 cuDNN version: Could not collect HIP runtime version: N/A MIOpen runtime version: N/A Is XNNPACK available: True CPU: Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Address sizes: 46 bits physical, 57 bits virtual Byte Order: Little Endian CPU(s): 192 On-line CPU(s) list: 0-191 Vendor ID: GenuineIntel Model name: Intel Xeon Processor (SapphireRapids) CPU family: 6 Model: 143 Thread(s) per core: 1 Core(s) per socket: 192 Socket(s): 1 Stepping: 4 BogoMIPS: 4200.00 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology cpuid tsc_known_freq pni pclmulqdq vmx ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch cpuid_fault invpcid_single ssbd ibrs ibpb stibp ibrs_enhanced tpr_shadow flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves avx_vnni avx512_bf16 wbnoinvd arat vnmi avx512vbmi umip pku ospke waitpkg avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq la57 rdpid bus_lock_detect cldemote movdiri movdir64b fsrm md_clear serialize tsxldtrk avx512_fp16 arch_capabilities Virtualization: VT-x Hypervisor vendor: KVM Virtualization type: full L1d cache: 6 MiB (192 instances) L1i cache: 6 MiB (192 instances) L2 cache: 768 MiB (192 instances) L3 cache: 16 MiB (1 instance) NUMA node(s): 1 NUMA node0 CPU(s): 0-191 Vulnerability Gather data sampling: Not affected Vulnerability Itlb multihit: Not affected Vulnerability L1tf: Not affected Vulnerability Mds: Not affected Vulnerability Meltdown: Not affected Vulnerability Mmio stale data: Unknown: No mitigations Vulnerability Retbleed: Not affected Vulnerability Spec rstack overflow: Not affected Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Vulnerability Spectre v2: Mitigation; Enhanced / Automatic IBRS; IBPB conditional; RSB filling; PBRSB-eIBRS SW sequence; BHI Syscall hardening, KVM SW loop Vulnerability Srbds: Not affected Vulnerability Tsx async abort: Mitigation; TSX disabled Versions of relevant libraries: [pip3] flashinfer==0.1.6+cu121torch2.4 [pip3] numpy==1.26.4 [pip3] nvidia-cublas-cu12==12.1.3.1 [pip3] nvidia-cuda-cupti-cu12==12.1.105 [pip3] nvidia-cuda-nvrtc-cu12==12.1.105 [pip3] nvidia-cuda-runtime-cu12==12.1.105 [pip3] nvidia-cudnn-cu12==9.1.0.70 [pip3] nvidia-cufft-cu12==11.0.2.54 [pip3] nvidia-curand-cu12==10.3.2.106 [pip3] nvidia-cusolver-cu12==11.4.5.107 [pip3] nvidia-cusparse-cu12==12.1.0.106 [pip3] nvidia-ml-py==12.560.30 [pip3] nvidia-nccl-cu12==2.20.5 [pip3] nvidia-nvjitlink-cu12==12.6.77 [pip3] nvidia-nvtx-cu12==12.1.105 [pip3] pyzmq==26.2.0 [pip3] torch==2.4.0 [pip3] torchvision==0.19.0 [pip3] transformers==4.45.2 [pip3] triton==3.0.0 [conda] Could not collect ROCM Version: Could not collect Neuron SDK Version: N/A vLLM Version: 0.6.3.post1 vLLM Build Flags: CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled GPU Topology: GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7 NIC0 NIC1 NIC2 NIC3 NIC4 NIC5 NIC6 NIC7 CPU Affinity NUMA Affinity GPU NUMA ID GPU0 X NV18 NV18 NV18 NV18 NV18 NV18 NV18 PHB PHB PHB PHB PHB PHB PHB PHB 0-191 0 N/A GPU1 NV18 X NV18 NV18 NV18 NV18 NV18 NV18 PHB PHB PHB PHB PHB PHB PHB PHB 0-191 0 N/A GPU2 NV18 NV18 X NV18 NV18 NV18 NV18 NV18 PHB PHB PHB PHB PHB PHB PHB PHB 0-191 0 N/A GPU3 NV18 NV18 NV18 X NV18 NV18 NV18 NV18 PHB PHB PHB PHB PHB PHB PHB PHB 0-191 0 N/A GPU4 NV18 NV18 NV18 NV18 X NV18 NV18 NV18 PHB PHB PHB PHB PHB PHB PHB PHB 0-191 0 N/A GPU5 NV18 NV18 NV18 NV18 NV18 X NV18 NV18 PHB PHB PHB PHB PHB PHB PHB PHB 0-191 0 N/A GPU6 NV18 NV18 NV18 NV18 NV18 NV18 X NV18 PHB PHB PHB PHB PHB PHB PHB PHB 0-191 0 N/A GPU7 NV18 NV18 NV18 NV18 NV18 NV18 NV18 X PHB PHB PHB PHB PHB PHB PHB PHB 0-191 0 N/A NIC0 PHB PHB PHB PHB PHB PHB PHB PHB X PHB PHB PHB PHB PHB PHB PHB NIC1 PHB PHB PHB PHB PHB PHB PHB PHB PHB X PHB PHB PHB PHB PHB PHB NIC2 PHB PHB PHB PHB PHB PHB PHB PHB PHB PHB X PHB PHB PHB PHB PHB NIC3 PHB PHB PHB PHB PHB PHB PHB PHB PHB PHB PHB X PHB PHB PHB PHB NIC4 PHB PHB PHB PHB PHB PHB PHB PHB PHB PHB PHB PHB X PHB PHB PHB NIC5 PHB PHB PHB PHB PHB PHB PHB PHB PHB PHB PHB PHB PHB X PHB PHB NIC6 PHB PHB PHB PHB PHB PHB PHB PHB PHB PHB PHB PHB PHB PHB X PHB NIC7 PHB PHB PHB PHB PHB PHB PHB PHB PHB PHB PHB PHB PHB PHB PHB X Legend: X = Self SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI) NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU) PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge) PIX = Connection traversing at most a single PCIe bridge NV# = Connection traversing a bonded set of # NVLinks NIC Legend: NIC0: mlx5_0 NIC1: mlx5_1 NIC2: mlx5_2 NIC3: mlx5_3 NIC4: mlx5_4 NIC5: mlx5_5 NIC6: mlx5_6 NIC7: mlx5_7 NVIDIA_VISIBLE_DEVICES=all NVIDIA_REQUIRE_CUDA=cuda>=12.4 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driv er>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,drive r<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526 brand=tesla,driver>=535,driver<536 brand= unknown,driver>=535,driver<536 brand=nvidia,driver>=535,driver<536 brand=nvidiartx,driver>=535,driver<536 brand=geforce,driver>=535,driver<536 brand=geforcertx,driver>=535,driver<536 brand=quadro,driver>=535,driver<536 brand=quadrortx,driver>=535,driver<536 brand=titan,dr iver>=535,driver<536 brand=titanrtx,driver>=535,driver<536 NVIDIA_DRIVER_CAPABILITIES=compute,utility VLLM_USAGE_SOURCE=production-docker-image CUDA_VERSION=12.4.1 CUDA_VISIBLE_DEVICES=2,3 CUDA_VISIBLE_DEVICES=2,3 LD_LIBRARY_PATH=/usr/local/lib/python3.12/dist-packages/cv2/../../lib64:/usr/local/nvidia/lib:/usr/local/nvidia/lib64 CUDA_MODULE_LOADING=LAZY ```Model Input Dumps
No response
🐛 Describe the bug
VLLLm crash when running Qwen/Qwen2.5-Coder-32B-Instruct on two H100 GPUs
Command for reprduce:
Error message:
Before submitting a new issue...