[Bug]: CUDA illegal memory access error when `enable_prefix_caching=True`

[mpoemsl]

Your current environment

The output of `python collect_env.py`
PyTorch version: 2.3.0+cu121
Is debug build: False
CUDA used to build PyTorch: 12.1
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.6 LTS (x86_64)
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0
Clang version: Could not collect
CMake version: version 3.29.3
Libc version: glibc-2.31

Python version: 3.11.9 (main, Apr  6 2024, 17:59:24) [GCC 9.4.0] (64-bit runtime)
Python platform: Linux-5.15.0-107-generic-x86_64-with-glibc2.31
Is CUDA available: True
CUDA runtime version: Could not collect
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration:
GPU 0: NVIDIA RTX A6000
GPU 1: NVIDIA RTX A6000
GPU 2: NVIDIA RTX A6000
GPU 3: NVIDIA RTX A6000
GPU 4: NVIDIA RTX A6000
GPU 5: NVIDIA RTX A6000
GPU 6: NVIDIA RTX A6000
GPU 7: NVIDIA RTX A6000

Nvidia driver version: 535.161.08
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:                       x86_64
CPU op-mode(s):                     32-bit, 64-bit
Byte Order:                         Little Endian
Address sizes:                      43 bits physical, 48 bits virtual
CPU(s):                             256
On-line CPU(s) list:                0-254
Off-line CPU(s) list:               255
Thread(s) per core:                 1
Core(s) per socket:                 64
Socket(s):                          2
NUMA node(s):                       2
Vendor ID:                          AuthenticAMD
CPU family:                         23
Model:                              49
Model name:                         AMD EPYC 7662 64-Core Processor
Stepping:                           0
Frequency boost:                    enabled
CPU MHz:                            1499.325
CPU max MHz:                        2154.2959
CPU min MHz:                        1500.0000
BogoMIPS:                           4000.28
Virtualization:                     AMD-V
L1d cache:                          2 MiB
L1i cache:                          2 MiB
L2 cache:                           32 MiB
L3 cache:                           256 MiB
NUMA node0 CPU(s):                  0-63,128-191
NUMA node1 CPU(s):                  64-127,192-254
Vulnerability Gather data sampling: Not affected
Vulnerability Itlb multihit:        Not affected
Vulnerability L1tf:                 Not affected
Vulnerability Mds:                  Not affected
Vulnerability Meltdown:             Not affected
Vulnerability Mmio stale data:      Not affected
Vulnerability Retbleed:             Mitigation; untrained return thunk; SMT enabled with STIBP protection
Vulnerability Spec rstack overflow: Mitigation; safe RET
Vulnerability Spec store bypass:    Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1:           Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:           Mitigation; Retpolines; IBPB conditional; STIBP always-on; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
Vulnerability Srbds:                Not affected
Vulnerability Tsx async abort:      Not affected
Flags:                              fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr rdpru wbnoinvd amd_ppin arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif v_spec_ctrl umip rdpid overflow_recov succor smca sme sev sev_es

Versions of relevant libraries:
[pip3] mypy-extensions==1.0.0
[pip3] numpy==1.26.4
[pip3] nvidia-nccl-cu12==2.20.5
[pip3] torch==2.3.0
[pip3] transformers==4.41.1
[pip3] triton==2.3.0
[conda] Could not collect
ROCM Version: Could not collect
Neuron SDK Version: N/A
vLLM Version: 0.5.0.post1
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled
GPU Topology:
GPU0    GPU1    GPU2    GPU3    GPU4    GPU5    GPU6    GPU7    CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0     X  NV4 NODE    NODE    SYS SYS SYS SYS 0-63,128-191    0       N/A
GPU1    NV4  X  NODE    NODE    SYS SYS SYS SYS 0-63,128-191    0       N/A
GPU2    NODE    NODE     X  NV4 SYS SYS SYS SYS 0-63,128-191    0       N/A
GPU3    NODE    NODE    NV4  X  SYS SYS SYS SYS 0-63,128-191    0       N/A
GPU4    SYS SYS SYS SYS  X  NV4 NODE    NODE    64-127,192-254  1       N/A
GPU5    SYS SYS SYS SYS NV4  X  NODE    NODE    64-127,192-254  1       N/A
GPU6    SYS SYS SYS SYS NODE    NODE     X  NV4 64-127,192-254  1       N/A
GPU7    SYS SYS SYS SYS NODE    NODE    NV4  X  64-127,192-254  1       N/A

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

🐛 Describe the bug

While testing the new version, I ran into this CUDA error (not immediately, after a few successful iterations).

llm-inference-mixtral-20240612-2-1  | ESC[36m(RayWorkerWrapper pid=14390)ESC[0m [rank1]:[E ProcessGroupNCCL.cpp:1414] [PG 2 Rank 1] Process group watchdog thread terminated with exception: CUDA error: an illegal me
mory access was encountered
llm-inference-mixtral-20240612-2-1  | ESC[36m(RayWorkerWrapper pid=14390)ESC[0m CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
llm-inference-mixtral-20240612-2-1  | ESC[36m(RayWorkerWrapper pid=14390)ESC[0m For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
llm-inference-mixtral-20240612-2-1  | ESC[36m(RayWorkerWrapper pid=14390)ESC[0m Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
llm-inference-mixtral-20240612-2-1  | ESC[36m(RayWorkerWrapper pid=14390)ESC[0m
llm-inference-mixtral-20240612-2-1  | ESC[36m(RayWorkerWrapper pid=14390)ESC[0m Exception raised from c10_cuda_check_implementation at ../c10/cuda/CUDAException.cpp:43 (most recent call first):
llm-inference-mixtral-20240612-2-1  | ESC[36m(RayWorkerWrapper pid=14390)ESC[0m frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7fec74098897 in /usr/local/lib/python3.11/dist-packages/torch/
lib/libc10.so)
llm-inference-mixtral-20240612-2-1  | ESC[36m(RayWorkerWrapper pid=14390)ESC[0m frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&) + 0x64 (0x7fec74048b25 in /usr/local
/lib/python3.11/dist-packages/torch/lib/libc10.so)
llm-inference-mixtral-20240612-2-1  | ESC[36m(RayWorkerWrapper pid=14390)ESC[0m frame #2: c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x118 (0x7fec8c150718 in /usr/local/lib
/python3.11/dist-packages/torch/lib/libc10_cuda.so)
llm-inference-mixtral-20240612-2-1  | ESC[36m(RayWorkerWrapper pid=14390)ESC[0m frame #3: c10d::ProcessGroupNCCL::WorkNCCL::finishedGPUExecutionInternal() const + 0x56 (0x7fbcd5978e36 in /usr/local/lib/python3.11/d
ist-packages/torch/lib/libtorch_cuda.so)
llm-inference-mixtral-20240612-2-1  | ESC[36m(RayWorkerWrapper pid=14390)ESC[0m frame #4: c10d::ProcessGroupNCCL::WorkNCCL::isCompleted() + 0x58 (0x7fbcd597cf38 in /usr/local/lib/python3.11/dist-packages/torch/lib/
libtorch_cuda.so)
llm-inference-mixtral-20240612-2-1  | ESC[36m(RayWorkerWrapper pid=14390)ESC[0m frame #5: c10d::ProcessGroupNCCL::watchdogHandler() + 0x77c (0x7fbcd59825ac in /usr/local/lib/python3.11/dist-packages/torch/lib/libto
rch_cuda.so)
llm-inference-mixtral-20240612-2-1  | ESC[36m(RayWorkerWrapper pid=14390)ESC[0m frame #6: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x10c (0x7fbcd598331c in /usr/local/lib/python3.11/dist-packages/torch/lib/libt
orch_cuda.so)
llm-inference-mixtral-20240612-2-1  | ESC[36m(RayWorkerWrapper pid=14390)ESC[0m frame #7: <unknown function> + 0xd6df4 (0x7fec90c95df4 in /usr/lib/x86_64-linux-gnu/libstdc++.so.6)
llm-inference-mixtral-20240612-2-1  | ESC[36m(RayWorkerWrapper pid=14390)ESC[0m frame #8: <unknown function> + 0x8609 (0x7fec92bb5609 in /usr/lib/x86_64-linux-gnu/libpthread.so.0)
llm-inference-mixtral-20240612-2-1  | ESC[36m(RayWorkerWrapper pid=14390)ESC[0m frame #9: clone + 0x43 (0x7fec92cef353 in /usr/lib/x86_64-linux-gnu/libc.so.6)
llm-inference-mixtral-20240612-2-1  | ESC[36m(RayWorkerWrapper pid=14390)ESC[0m
llm-inference-mixtral-20240612-2-1  | ESC[36m(RayWorkerWrapper pid=14390)ESC[0m [2024-06-14 10:35:20,297 E 14390 15027] logging.cc:101: Unhandled exception: N3c1016DistBackendErrorE. what(): [PG 2 Rank 1] Process g
roup watchdog thread terminated with exception: CUDA error: an illegal memory access was encountered
llm-inference-mixtral-20240612-2-1  | ESC[36m(RayWorkerWrapper pid=14390)ESC[0m CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
llm-inference-mixtral-20240612-2-1  | ESC[36m(RayWorkerWrapper pid=14390)ESC[0m For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
llm-inference-mixtral-20240612-2-1  | ESC[36m(RayWorkerWrapper pid=14390)ESC[0m Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
llm-inference-mixtral-20240612-2-1  | ESC[36m(RayWorkerWrapper pid=14390)ESC[0m
llm-inference-mixtral-20240612-2-1  | ESC[36m(RayWorkerWrapper pid=14390)ESC[0m Exception raised from c10_cuda_check_implementation at ../c10/cuda/CUDAException.cpp:43 (most recent call first):
llm-inference-mixtral-20240612-2-1  | ESC[36m(RayWorkerWrapper pid=14390)ESC[0m frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7fec74098897 in /usr/local/lib/python3.11/dist-packages/torch/
lib/libc10.so)
llm-inference-mixtral-20240612-2-1  | ESC[36m(RayWorkerWrapper pid=14390)ESC[0m frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&) + 0x64 (0x7fec74048b25 in /usr/local
/lib/python3.11/dist-packages/torch/lib/libc10.so)
llm-inference-mixtral-20240612-2-1  | ESC[36m(RayWorkerWrapper pid=14390)ESC[0m frame #2: c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x118 (0x7fec8c150718 in /usr/local/lib
/python3.11/dist-packages/torch/lib/libc10_cuda.so)
llm-inference-mixtral-20240612-2-1  | ESC[36m(RayWorkerWrapper pid=14390)ESC[0m frame #3: c10d::ProcessGroupNCCL::WorkNCCL::finishedGPUExecutionInternal() const + 0x56 (0x7fbcd5978e36 in /usr/local/lib/python3.11/d
ist-packages/torch/lib/libtorch_cuda.so)
llm-inference-mixtral-20240612-2-1  | ESC[36m(RayWorkerWrapper pid=14390)ESC[0m frame #4: c10d::ProcessGroupNCCL::WorkNCCL::isCompleted() + 0x58 (0x7fbcd597cf38 in /usr/local/lib/python3.11/dist-packages/torch/lib/
libtorch_cuda.so)
llm-inference-mixtral-20240612-2-1  | ESC[36m(RayWorkerWrapper pid=14390)ESC[0m frame #5: c10d::ProcessGroupNCCL::watchdogHandler() + 0x77c (0x7fbcd59825ac in /usr/local/lib/python3.11/dist-packages/torch/lib/libto
rch_cuda.so)
llm-inference-mixtral-20240612-2-1  | ESC[36m(RayWorkerWrapper pid=14390)ESC[0m frame #6: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x10c (0x7fbcd598331c in /usr/local/lib/python3.11/dist-packages/torch/lib/libt
orch_cuda.so)
llm-inference-mixtral-20240612-2-1  | ESC[36m(RayWorkerWrapper pid=14390)ESC[0m frame #7: <unknown function> + 0xd6df4 (0x7fec90c95df4 in /usr/lib/x86_64-linux-gnu/libstdc++.so.6)
llm-inference-mixtral-20240612-2-1  | ESC[36m(RayWorkerWrapper pid=14390)ESC[0m frame #8: <unknown function> + 0x8609 (0x7fec92bb5609 in /usr/lib/x86_64-linux-gnu/libpthread.so.0)
llm-inference-mixtral-20240612-2-1  | ESC[36m(RayWorkerWrapper pid=14390)ESC[0m frame #9: clone + 0x43 (0x7fec92cef353 in /usr/lib/x86_64-linux-gnu/libc.so.6)
llm-inference-mixtral-20240612-2-1  | ESC[36m(RayWorkerWrapper pid=14390)ESC[0m
llm-inference-mixtral-20240612-2-1  | ESC[36m(RayWorkerWrapper pid=14390)ESC[0m Exception raised from ncclCommWatchdog at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1418 (most recent call first):
llm-inference-mixtral-20240612-2-1  | ESC[36m(RayWorkerWrapper pid=14390)ESC[0m frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7fec74098897 in /usr/local/lib/python3.11/dist-packages/torch/
lib/libc10.so)
llm-inference-mixtral-20240612-2-1  | ESC[36m(RayWorkerWrapper pid=14390)ESC[0m frame #1: <unknown function> + 0xe32e33 (0x7fbcd5605e33 in /usr/local/lib/python3.11/dist-packages/torch/lib/libtorch_cuda.so)
llm-inference-mixtral-20240612-2-1  | ESC[36m(RayWorkerWrapper pid=14390)ESC[0m frame #2: <unknown function> + 0xd6df4 (0x7fec90c95df4 in /usr/lib/x86_64-linux-gnu/libstdc++.so.6)
llm-inference-mixtral-20240612-2-1  | ESC[36m(RayWorkerWrapper pid=14390)ESC[0m frame #3: <unknown function> + 0x8609 (0x7fec92bb5609 in /usr/lib/x86_64-linux-gnu/libpthread.so.0)
llm-inference-mixtral-20240612-2-1  | ESC[36m(RayWorkerWrapper pid=14390)ESC[0m frame #4: clone + 0x43 (0x7fec92cef353 in /usr/lib/x86_64-linux-gnu/libc.so.6)
llm-inference-mixtral-20240612-2-1  | ESC[36m(RayWorkerWrapper pid=14390)ESC[0m
llm-inference-mixtral-20240612-2-1  | ESC[36m(RayWorkerWrapper pid=14390)ESC[0m [2024-06-14 10:35:20,329 E 14390 15027] logging.cc:108: Stack trace:
llm-inference-mixtral-20240612-2-1  | ESC[36m(RayWorkerWrapper pid=14390)ESC[0m  /usr/local/lib/python3.11/dist-packages/ray/_raylet.so(+0x101867a) [0x7fec91dc367a] ray::operator<<()
llm-inference-mixtral-20240612-2-1  | ESC[36m(RayWorkerWrapper pid=14390)ESC[0m /usr/local/lib/python3.11/dist-packages/ray/_raylet.so(+0x101b138) [0x7fec91dc6138] ray::TerminateHandler()
llm-inference-mixtral-20240612-2-1  | ESC[36m(RayWorkerWrapper pid=14390)ESC[0m /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xaa37c) [0x7fec90c6937c]
llm-inference-mixtral-20240612-2-1  | ESC[36m(RayWorkerWrapper pid=14390)ESC[0m /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xaa3e7) [0x7fec90c693e7]
llm-inference-mixtral-20240612-2-1  | ESC[36m(RayWorkerWrapper pid=14390)ESC[0m /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xaa36f) [0x7fec90c6936f]
llm-inference-mixtral-20240612-2-1  | ESC[36m(RayWorkerWrapper pid=14390)ESC[0m /usr/local/lib/python3.11/dist-packages/torch/lib/libtorch_cuda.so(+0xe32ee4) [0x7fbcd5605ee4] c10d::ProcessGroupNCCL::ncclCommWatchdo
g()
llm-inference-mixtral-20240612-2-1  | ESC[36m(RayWorkerWrapper pid=14390)ESC[0m /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xd6df4) [0x7fec90c95df4]
llm-inference-mixtral-20240612-2-1  | ESC[36m(RayWorkerWrapper pid=14390)ESC[0m /usr/lib/x86_64-linux-gnu/libpthread.so.0(+0x8609) [0x7fec92bb5609] start_thread
llm-inference-mixtral-20240612-2-1  | ESC[36m(RayWorkerWrapper pid=14390)ESC[0m /usr/lib/x86_64-linux-gnu/libc.so.6(clone+0x43) [0x7fec92cef353] __clone
llm-inference-mixtral-20240612-2-1  | ESC[36m(RayWorkerWrapper pid=14390)ESC[0m
llm-inference-mixtral-20240612-2-1  | ESC[36m(RayWorkerWrapper pid=14390)ESC[0m *** SIGABRT received at time=1718361320 on cpu 156 ***
llm-inference-mixtral-20240612-2-1  | ESC[36m(RayWorkerWrapper pid=14390)ESC[0m PC: @     0x7fec92c1300b  (unknown)  raise
llm-inference-mixtral-20240612-2-1  | ESC[36m(RayWorkerWrapper pid=14390)ESC[0m     @     0x7fec92c13090       2192  (unknown)
llm-inference-mixtral-20240612-2-1  | ESC[36m(RayWorkerWrapper pid=14390)ESC[0m     @     0x7fec90c6937c  (unknown)  (unknown)
llm-inference-mixtral-20240612-2-1  | ESC[36m(RayWorkerWrapper pid=14390)ESC[0m     @     0x7fec90c69090  (unknown)  (unknown)
llm-inference-mixtral-20240612-2-1  | ESC[36m(RayWorkerWrapper pid=14390)ESC[0m [2024-06-14 10:35:20,331 E 14390 15027] logging.cc:365: *** SIGABRT received at time=1718361320 on cpu 156 ***
llm-inference-mixtral-20240612-2-1  | ESC[36m(RayWorkerWrapper pid=14390)ESC[0m [2024-06-14 10:35:20,331 E 14390 15027] logging.cc:365: PC: @     0x7fec92c1300b  (unknown)  raise
llm-inference-mixtral-20240612-2-1  | ESC[36m(RayWorkerWrapper pid=14390)ESC[0m [2024-06-14 10:35:20,331 E 14390 15027] logging.cc:365:     @     0x7fec92c13090       2192  (unknown)
llm-inference-mixtral-20240612-2-1  | ESC[36m(RayWorkerWrapper pid=14390)ESC[0m [2024-06-14 10:35:20,331 E 14390 15027] logging.cc:365:     @     0x7fec90c6937c  (unknown)  (unknown)
llm-inference-mixtral-20240612-2-1  | ESC[36m(RayWorkerWrapper pid=14390)ESC[0m [2024-06-14 10:35:20,332 E 14390 15027] logging.cc:365:     @     0x7fec90c69090  (unknown)  (unknown)
llm-inference-mixtral-20240612-2-1  | ESC[36m(RayWorkerWrapper pid=14390)ESC[0m Fatal Python error: Aborted
llm-inference-mixtral-20240612-2-1  | ESC[36m(RayWorkerWrapper pid=14390)ESC[0m
llm-inference-mixtral-20240612-2-1  | ESC[36m(RayWorkerWrapper pid=14390)ESC[0m
llm-inference-mixtral-20240612-2-1  | ESC[36m(RayWorkerWrapper pid=14390)ESC[0m Extension modules: msgpack._cmsgpack, google._upb._message, psutil._psutil_linux, psutil._psutil_posix, setproctitle, yaml._yaml, char
set_normalizer.md, uvloop.loop, ray._raylet, numpy.core._multiarray_umath, numpy.core._multiarray_tests, numpy.linalg._umath_linalg, numpy.fft._pocketfft_internal, numpy.random._common, numpy.random.bit_generator,
numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, torch._C, torch._C._fft, torch._C._linalg, torch.
_C._nested, torch._C._nn, torch._C._sparse, torch._C._special, sentencepiece._sentencepiece, pyarrow.lib, pyarrow._json, PIL._imaging, cuda_utils, __triton_launcher (total: 35)
llm-inference-mixtral-20240612-2-1  | ESC[33m(raylet)ESC[0m A worker died or was killed while executing a task by an unexpected system error. To troubleshoot the problem, check the logs for the dead worker. RayTask
 ID: ffffffffffffffff268bbf7e251c226eef173be001000000 Worker ID: 0a04e75572848f8edda22427a258283dfdee451cc8858a13c0c1bd7e Node ID: 6461451955b118f9eefdf9dfc3b248c3ce82bd7e1809df449be56a98 Worker IP address: 172.21.
0.18 Worker port: 40797 Worker PID: 14390 Worker exit type: SYSTEM_ERROR Worker exit detail: Worker unexpectedly exits with a connection error code 2. End of file. There are some potential root causes. (1) The proc
ess is killed by SIGKILL by OOM killer due to high memory usage. (2) ray stop --force is called. (3) The worker is crashed unexpectedly due to SIGSEGV or other unexpected errors.
llm-inference-mixtral-20240612-2-1  | ESC[36m(RayWorkerWrapper pid=14862)ESC[0m INFO 06-14 10:33:31 model_runner.py:965] Graph capturing finished in 13 secs.ESC[32m [repeated 2x across cluster]ESC[0m
llm-inference-mixtral-20240612-2-1  | ERROR 06-14 10:36:20 async_llm_engine.py:535] Engine iteration timed out. This should never happen!
llm-inference-mixtral-20240612-2-1  | ERROR 06-14 10:36:20 async_llm_engine.py:52] Engine background task failed

The only change compared to the previously working setup was the VLLM version upgrade and enabling prefix caching. The model served is Mixtral 8x7B (unquantized).

Note that CUDA_VISIBLE_DEVICES was set to 0,1,2,3 with tensor_parallel=4 to only use the first four GPUs (the machine has eight in total).

Update: no issues with v0.4.3

[youkaichao]

can you follow https://docs.vllm.ai/en/latest/getting_started/debugging.html to investigate which Python function causes the crash?

[simon-mo]

Also please try the latest patch v0.5.0.post1 which might fix one of the root cause

[mpoemsl]

Thanks for the quick response from both of you! Through experiments, I managed to further narrow the issue down - it only occurs when enable_prefix_caching=True and only the first time a prefix is re-used.

@simon-mo I'm already using the pip version v0.5.0.post1

@youkaichao Here's the output with the env vars from your link set:

llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148] Error executing method start_worker_execution_loop. This might cause deadlock in distributed execution.
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148] Traceback (most recent call last):
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148]   File "/usr/local/lib/python3.11/dist-packages/vllm/worker/worker_base.py", line 140, in execute_method
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148]     return executor(*args, **kwargs)
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148]            ^^^^^^^^^^^^^^^^^^^^^^^^^
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148]   File "/usr/local/lib/python3.11/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148]     return func(*args, **kwargs)
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148]            ^^^^^^^^^^^^^^^^^^^^^
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148]   File "/usr/local/lib/python3.11/dist-packages/vllm/worker/worker.py", line 294, in start_worker_execution_loop
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148]     while self._execute_model_non_driver():
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148]           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148]   File "/usr/local/lib/python3.11/dist-packages/vllm/worker/worker.py", line 317, in _execute_model_non_driver
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148]     self.model_runner.execute_model(None, self.gpu_cache)
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148]   File "/usr/local/lib/python3.11/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148]     return func(*args, **kwargs)
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148]            ^^^^^^^^^^^^^^^^^^^^^
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148]   File "/usr/local/lib/python3.11/dist-packages/vllm/worker/model_runner.py", line 749, in execute_model
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148]     hidden_states = model_executable(
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148]                     ^^^^^^^^^^^^^^^^^
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148]   File "/usr/local/lib/python3.11/dist-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148]     return self._call_impl(*args, **kwargs)
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148]   File "/usr/local/lib/python3.11/dist-packages/torch/nn/modules/module.py", line 1541, in _call_impl
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148]     return forward_call(*args, **kwargs)
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148]   File "/usr/local/lib/python3.11/dist-packages/vllm/model_executor/models/mixtral.py", line 535, in forward
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148]     hidden_states = self.model(input_ids, positions, kv_caches,
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148]                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148]   File "/usr/local/lib/python3.11/dist-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148]     return self._call_impl(*args, **kwargs)
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148]   File "/usr/local/lib/python3.11/dist-packages/torch/nn/modules/module.py", line 1541, in _call_impl
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148]     return forward_call(*args, **kwargs)
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148]   File "/usr/local/lib/python3.11/dist-packages/vllm/model_executor/models/mixtral.py", line 468, in forward
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148]     hidden_states, residual = layer(positions, hidden_states,
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148]                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148]   File "/usr/local/lib/python3.11/dist-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148]     return self._call_impl(*args, **kwargs)
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148]   File "/usr/local/lib/python3.11/dist-packages/torch/nn/modules/module.py", line 1541, in _call_impl
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148]     return forward_call(*args, **kwargs)
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148]   File "/usr/local/lib/python3.11/dist-packages/vllm/model_executor/models/mixtral.py", line 414, in forward
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148]     hidden_states = self.self_attn(
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148]                     ^^^^^^^^^^^^^^^
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148]   File "/usr/local/lib/python3.11/dist-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148]     return self._call_impl(*args, **kwargs)
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148]   File "/usr/local/lib/python3.11/dist-packages/torch/nn/modules/module.py", line 1541, in _call_impl
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148]     return forward_call(*args, **kwargs)
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148]   File "/usr/local/lib/python3.11/dist-packages/vllm/model_executor/models/mixtral.py", line 363, in forward
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148]     attn_output = self.attn(q, k, v, kv_cache, attn_metadata)
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148]                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148]   File "/usr/local/lib/python3.11/dist-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148]     return self._call_impl(*args, **kwargs)
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148]   File "/usr/local/lib/python3.11/dist-packages/torch/nn/modules/module.py", line 1541, in _call_impl
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148]     return forward_call(*args, **kwargs)
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148]   File "/usr/local/lib/python3.11/dist-packages/vllm/attention/layer.py", line 89, in forward
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148]     return self.impl.forward(query, key, value, kv_cache, attn_metadata,
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148]   File "/usr/local/lib/python3.11/dist-packages/vllm/attention/backends/flash_attn.py", line 339, in forward
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148]     output[:num_prefill_tokens] = flash_attn_varlen_func(
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148]                                   ^^^^^^^^^^^^^^^^^^^^^^^
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148]   File "/usr/local/lib/python3.11/dist-packages/vllm_flash_attn/flash_attn_interface.py", line 1099, in flash_attn_varlen_func
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148]     return FlashAttnVarlenFunc.apply(
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148]            ^^^^^^^^^^^^^^^^^^^^^^^^^^
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148]   File "/usr/local/lib/python3.11/dist-packages/torch/autograd/function.py", line 598, in apply
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148]     return super().apply(*args, **kwargs)  # type: ignore[misc]
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148]   File "/usr/local/lib/python3.11/dist-packages/vllm_flash_attn/flash_attn_interface.py", line 596, in forward
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148]     out, q, k, v, out_padded, softmax_lse, S_dmask, rng_state = _flash_attn_varlen_forward(
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148]                                                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148]   File "/usr/local/lib/python3.11/dist-packages/vllm_flash_attn/flash_attn_interface.py", line 88, in _flash_attn_varlen_forward
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148]     out, q, k, v, out_padded, softmax_lse, S_dmask, rng_state = flash_attn_cuda.varlen_fwd(
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148]                                                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148] RuntimeError: CUDA error: an illegal memory access was encountered
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148] Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14390) ERROR 06-14 17:57:30 worker_base.py:148]
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14863) INFO 06-14 17:52:48 model_runner.py:965] Graph capturing finished in 129 secs. [repeated 2x across cluster]
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14743) [rank2]:[E ProcessGroupNCCL.cpp:1414] [PG 2 Rank 2] Process group watchdog thread terminated with exception: CUDA error: an illegal memory access was encountered
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14743) Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14743)
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14743) Exception raised from c10_cuda_check_implementation at ../c10/cuda/CUDAException.cpp:43 (most recent call first):
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14743) frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f8140773897 in /usr/local/lib/python3.11/dist-packages/torch/lib/libc10.so)
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14743) frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&) + 0x64 (0x7f8140723b25 in /usr/local/lib/python3.11/dist-packages/torch/lib/libc10.so)
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14743) frame #2: c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x118 (0x7f81580c5718 in /usr/local/lib/python3.11/dist-packages/torch/lib/libc10_cuda.so)
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14743) frame #3: c10d::ProcessGroupNCCL::WorkNCCL::finishedGPUExecutionInternal() const + 0x56 (0x7f51b1978e36 in /usr/local/lib/python3.11/dist-packages/torch/lib/libtorch_cuda.so)
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14743) frame #4: c10d::ProcessGroupNCCL::WorkNCCL::isCompleted() + 0x58 (0x7f51b197cf38 in /usr/local/lib/python3.11/dist-packages/torch/lib/libtorch_cuda.so)
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14743) frame #5: c10d::ProcessGroupNCCL::watchdogHandler() + 0x77c (0x7f51b19825ac in /usr/local/lib/python3.11/dist-packages/torch/lib/libtorch_cuda.so)
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14743) frame #6: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x10c (0x7f51b198331c in /usr/local/lib/python3.11/dist-packages/torch/lib/libtorch_cuda.so)
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14743) frame #7: <unknown function> + 0xd6df4 (0x7f816b330df4 in /usr/lib/x86_64-linux-gnu/libstdc++.so.6)
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14743) frame #8: <unknown function> + 0x8609 (0x7f816d250609 in /usr/lib/x86_64-linux-gnu/libpthread.so.0)
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14743) frame #9: clone + 0x43 (0x7f816d38a353 in /usr/lib/x86_64-linux-gnu/libc.so.6)
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14743)
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14743) [2024-06-14 17:57:30,390 E 14743 15025] logging.cc:101: Unhandled exception: N3c1016DistBackendErrorE. what(): [PG 2 Rank 2] Process group watchdog thread terminated with exception: CUDA error: an illegal memory access was encountered
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14743) Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14743)
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14743) Exception raised from c10_cuda_check_implementation at ../c10/cuda/CUDAException.cpp:43 (most recent call first):
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14743) frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f8140773897 in /usr/local/lib/python3.11/dist-packages/torch/lib/libc10.so)
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14743) frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&) + 0x64 (0x7f8140723b25 in /usr/local/lib/python3.11/dist-packages/torch/lib/libc10.so)
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14743) frame #2: c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x118 (0x7f81580c5718 in /usr/local/lib/python3.11/dist-packages/torch/lib/libc10_cuda.so)
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14743) frame #3: c10d::ProcessGroupNCCL::WorkNCCL::finishedGPUExecutionInternal() const + 0x56 (0x7f51b1978e36 in /usr/local/lib/python3.11/dist-packages/torch/lib/libtorch_cuda.so)
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14743) frame #4: c10d::ProcessGroupNCCL::WorkNCCL::isCompleted() + 0x58 (0x7f51b197cf38 in /usr/local/lib/python3.11/dist-packages/torch/lib/libtorch_cuda.so)
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14743) frame #5: c10d::ProcessGroupNCCL::watchdogHandler() + 0x77c (0x7f51b19825ac in /usr/local/lib/python3.11/dist-packages/torch/lib/libtorch_cuda.so)
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14743) frame #6: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x10c (0x7f51b198331c in /usr/local/lib/python3.11/dist-packages/torch/lib/libtorch_cuda.so)
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14743) frame #7: <unknown function> + 0xd6df4 (0x7f816b330df4 in /usr/lib/x86_64-linux-gnu/libstdc++.so.6)
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14743) frame #8: <unknown function> + 0x8609 (0x7f816d250609 in /usr/lib/x86_64-linux-gnu/libpthread.so.0)
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14743) frame #9: clone + 0x43 (0x7f816d38a353 in /usr/lib/x86_64-linux-gnu/libc.so.6)
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14743)
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14743) Exception raised from ncclCommWatchdog at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1418 (most recent call first):
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14743) frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f8140773897 in /usr/local/lib/python3.11/dist-packages/torch/lib/libc10.so)
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14743) frame #1: <unknown function> + 0xe32e33 (0x7f51b1605e33 in /usr/local/lib/python3.11/dist-packages/torch/lib/libtorch_cuda.so)
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14743) frame #2: <unknown function> + 0xd6df4 (0x7f816b330df4 in /usr/lib/x86_64-linux-gnu/libstdc++.so.6)
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14743) frame #3: <unknown function> + 0x8609 (0x7f816d250609 in /usr/lib/x86_64-linux-gnu/libpthread.so.0)
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14743) frame #4: clone + 0x43 (0x7f816d38a353 in /usr/lib/x86_64-linux-gnu/libc.so.6)
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14743)
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14743) [2024-06-14 17:57:30,413 E 14743 15025] logging.cc:108: Stack trace:
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14743)  /usr/local/lib/python3.11/dist-packages/ray/_raylet.so(+0x101867a) [0x7f816c45e67a] ray::operator<<()
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14743) /usr/local/lib/python3.11/dist-packages/ray/_raylet.so(+0x101b138) [0x7f816c461138] ray::TerminateHandler()
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14743) /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xaa37c) [0x7f816b30437c]
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14743) /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xaa3e7) [0x7f816b3043e7]
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14743) /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xaa36f) [0x7f816b30436f]
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14743) /usr/local/lib/python3.11/dist-packages/torch/lib/libtorch_cuda.so(+0xe32ee4) [0x7f51b1605ee4] c10d::ProcessGroupNCCL::ncclCommWatchdog()
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14743) /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xd6df4) [0x7f816b330df4]
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14743) /usr/lib/x86_64-linux-gnu/libpthread.so.0(+0x8609) [0x7f816d250609] start_thread
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14743) /usr/lib/x86_64-linux-gnu/libc.so.6(clone+0x43) [0x7f816d38a353] __clone
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14743)
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14743) *** SIGABRT received at time=1718387850 on cpu 221 ***
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14743) PC: @     0x7f816d2ae00b  (unknown)  raise
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14743)     @     0x7f816d2ae090       2192  (unknown)
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14743)     @     0x7f816b30437c  (unknown)  (unknown)
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14743)     @     0x7f816b304090  (unknown)  (unknown)
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14743) [2024-06-14 17:57:30,413 E 14743 15025] logging.cc:365: *** SIGABRT received at time=1718387850 on cpu 221 ***
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14743) [2024-06-14 17:57:30,413 E 14743 15025] logging.cc:365: PC: @     0x7f816d2ae00b  (unknown)  raise
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14743) [2024-06-14 17:57:30,414 E 14743 15025] logging.cc:365:     @     0x7f816d2ae090       2192  (unknown)
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14743) [2024-06-14 17:57:30,414 E 14743 15025] logging.cc:365:     @     0x7f816b30437c  (unknown)  (unknown)
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14743) [2024-06-14 17:57:30,414 E 14743 15025] logging.cc:365:     @     0x7f816b304090  (unknown)  (unknown)
llm-inference-mixtral-20240612-2-1  | (RayWorkerWrapper pid=14743) Fatal Python error: Aborted

[youkaichao]

@robertgshaw2-neuralmagic for prefix caching.

[mpoemsl]

Will try this when I have the chance: https://github.com/vllm-project/vllm/issues/5376#issuecomment-2179257676

[ashgold]

This is happening to me as well. GPU: H100 tp: 4 engine: ray model: llama-65b fine-tuned model In vLLM v0.5.0.post1, applying fp8 quantized model and --enable-prefix-caching at the same time causes this symptom. Using only fp8 quantized model, or non quantized model and --enable-prefix-caching option does not cause this symptom. Also, it doesn't happen 100% of the time on every inference, it happens suddenly in the middle of a load.

Here's the error log I have.

[rank2]:[E ProcessGroupNCCL.cpp:1414] [PG 2 Rank 2] Process group watchdog thread terminated with exception: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

Exception raised from c10_cuda_check_implementation at ../c10/cuda/CUDAException.cpp:43 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7fdd9c09e897 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&) + 0x64 (0x7fdd9c04eb25 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so)
frame #2: c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x118 (0x7fdd9c176718 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10_cuda.so)
frame #3: c10d::ProcessGroupNCCL::WorkNCCL::finishedGPUExecutionInternal() const + 0x56 (0x7fdd4fe4ae36 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
frame #4: c10d::ProcessGroupNCCL::WorkNCCL::isCompleted() + 0x58 (0x7fdd4fe4ef38 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
frame #5: c10d::ProcessGroupNCCL::watchdogHandler() + 0x77c (0x7fdd4fe545ac in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
frame #6: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x10c (0x7fdd4fe5531c in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
frame #7: <unknown function> + 0xdc253 (0x7fdd9b8b0253 in /usr/lib/x86_64-linux-gnu/libstdc++.so.6)
frame #8: <unknown function> + 0x94ac3 (0x7fdd9cd50ac3 in /usr/lib/x86_64-linux-gnu/libc.so.6)
frame #9: <unknown function> + 0x126850 (0x7fdd9cde2850 in /usr/lib/x86_64-linux-gnu/libc.so.6)

[rank1]:[E ProcessGroupNCCL.cpp:1414] [PG 2 Rank 1] Process group watchdog thread terminated with exception: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

Exception raised from c10_cuda_check_implementation at ../c10/cuda/CUDAException.cpp:43 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7fb3ecc9e897 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&) + 0x64 (0x7fb3ecc4eb25 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so)
frame #2: c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x118 (0x7fb3ecd76718 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10_cuda.so)
frame #3: c10d::ProcessGroupNCCL::WorkNCCL::finishedGPUExecutionInternal() const + 0x56 (0x7fb3a0a4ae36 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
frame #4: c10d::ProcessGroupNCCL::WorkNCCL::isCompleted() + 0x58 (0x7fb3a0a4ef38 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
frame #5: c10d::ProcessGroupNCCL::watchdogHandler() + 0x77c (0x7fb3a0a545ac in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
frame #6: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x10c (0x7fb3a0a5531c in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
frame #7: <unknown function> + 0xdc253 (0x7fb3ec4b0253 in /usr/lib/x86_64-linux-gnu/libstdc++.so.6)
frame #8: <unknown function> + 0x94ac3 (0x7fb3ed957ac3 in /usr/lib/x86_64-linux-gnu/libc.so.6)
frame #9: <unknown function> + 0x126850 (0x7fb3ed9e9850 in /usr/lib/x86_64-linux-gnu/libc.so.6)

terminate called after throwing an instance of 'c10::DistBackendError'
  what():  [PG 2 Rank 2] Process group watchdog thread terminated with exception: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

Exception raised from c10_cuda_check_implementation at ../c10/cuda/CUDAException.cpp:43 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7fdd9c09e897 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&) + 0x64 (0x7fdd9c04eb25 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so)
frame #2: c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x118 (0x7fdd9c176718 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10_cuda.so)
frame #3: c10d::ProcessGroupNCCL::WorkNCCL::finishedGPUExecutionInternal() const + 0x56 (0x7fdd4fe4ae36 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
frame #4: c10d::ProcessGroupNCCL::WorkNCCL::isCompleted() + 0x58 (0x7fdd4fe4ef38 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
frame #5: c10d::ProcessGroupNCCL::watchdogHandler() + 0x77c (0x7fdd4fe545ac in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
frame #6: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x10c (0x7fdd4fe5531c in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
frame #7: <unknown function> + 0xdc253 (0x7fdd9b8b0253 in /usr/lib/x86_64-linux-gnu/libstdc++.so.6)
frame #8: <unknown function> + 0x94ac3 (0x7fdd9cd50ac3 in /usr/lib/x86_64-linux-gnu/libc.so.6)
frame #9: <unknown function> + 0x126850 (0x7fdd9cde2850 in /usr/lib/x86_64-linux-gnu/libc.so.6)

Exception raised from ncclCommWatchdog at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1418 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7fdd9c09e897 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so)
frame #1: <unknown function> + 0xe32e33 (0x7fdd4fad7e33 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
frame #2: <unknown function> + 0xdc253 (0x7fdd9b8b0253 in /usr/lib/x86_64-linux-gnu/libstdc++.so.6)
frame #3: <unknown function> + 0x94ac3 (0x7fdd9cd50ac3 in /usr/lib/x86_64-linux-gnu/libc.so.6)
frame #4: <unknown function> + 0x126850 (0x7fdd9cde2850 in /usr/lib/x86_64-linux-gnu/libc.so.6)

terminate called after throwing an instance of 'c10::DistBackendError'
  what():  [PG 2 Rank 1] Process group watchdog thread terminated with exception: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

Exception raised from c10_cuda_check_implementation at ../c10/cuda/CUDAException.cpp:43 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7fb3ecc9e897 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&) + 0x64 (0x7fb3ecc4eb25 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so)
frame #2: c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x118 (0x7fb3ecd76718 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10_cuda.so)
frame #3: c10d::ProcessGroupNCCL::WorkNCCL::finishedGPUExecutionInternal() const + 0x56 (0x7fb3a0a4ae36 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
frame #4: c10d::ProcessGroupNCCL::WorkNCCL::isCompleted() + 0x58 (0x7fb3a0a4ef38 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
frame #5: c10d::ProcessGroupNCCL::watchdogHandler() + 0x77c (0x7fb3a0a545ac in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
frame #6: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x10c (0x7fb3a0a5531c in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
frame #7: <unknown function> + 0xdc253 (0x7fb3ec4b0253 in /usr/lib/x86_64-linux-gnu/libstdc++.so.6)
frame #8: <unknown function> + 0x94ac3 (0x7fb3ed957ac3 in /usr/lib/x86_64-linux-gnu/libc.so.6)
frame #9: <unknown function> + 0x126850 (0x7fb3ed9e9850 in /usr/lib/x86_64-linux-gnu/libc.so.6)

Exception raised from ncclCommWatchdog at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1418 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7fb3ecc9e897 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so)
frame #1: <unknown function> + 0xe32e33 (0x7fb3a06d7e33 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
frame #2: <unknown function> + 0xdc253 (0x7fb3ec4b0253 in /usr/lib/x86_64-linux-gnu/libstdc++.so.6)
frame #3: <unknown function> + 0x94ac3 (0x7fb3ed957ac3 in /usr/lib/x86_64-linux-gnu/libc.so.6)
frame #4: <unknown function> + 0x126850 (0x7fb3ed9e9850 in /usr/lib/x86_64-linux-gnu/libc.so.6)

(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226] Exception in worker VllmWorkerProcess while processing method start_worker_execution_loop: CUDA error: an illegal memory access was encountered
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226] CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226] For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226] Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226] , Traceback (most recent call last):
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226]   File "/usr/local/lib/python3.10/dist-packages/vllm/executor/multiproc_worker_utils.py", line 223, in _run_worker_process
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226]     output = executor(*args, **kwargs)
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226]   File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226]     return func(*args, **kwargs)
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226]   File "/usr/local/lib/python3.10/dist-packages/vllm/worker/worker.py", line 294, in start_worker_execution_loop
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226]     while self._execute_model_non_driver():
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226]   File "/usr/local/lib/python3.10/dist-packages/vllm/worker/worker.py", line 317, in _execute_model_non_driver
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226]     self.model_runner.execute_model(None, self.gpu_cache)
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226]   File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226]     return func(*args, **kwargs)
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226]   File "/usr/local/lib/python3.10/dist-packages/vllm/worker/model_runner.py", line 749, in execute_model
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226]     hidden_states = model_executable(
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226]   File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226]     return self._call_impl(*args, **kwargs)
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226]   File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1541, in _call_impl
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226]     return forward_call(*args, **kwargs)
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226]   File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/llama.py", line 371, in forward
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226]     hidden_states = self.model(input_ids, positions, kv_caches,
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226]   File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226]     return self._call_impl(*args, **kwargs)
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226]   File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1541, in _call_impl
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226]     return forward_call(*args, **kwargs)
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226]   File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/llama.py", line 288, in forward
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226]     hidden_states, residual = layer(
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226]   File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226]     return self._call_impl(*args, **kwargs)
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226]   File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1541, in _call_impl
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226]     return forward_call(*args, **kwargs)
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226]   File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/llama.py", line 227, in forward
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226]     hidden_states = self.self_attn(
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226]   File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226]     return self._call_impl(*args, **kwargs)
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226]   File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1541, in _call_impl
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226]     return forward_call(*args, **kwargs)
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226]   File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/llama.py", line 161, in forward
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226]     attn_output = self.attn(q, k, v, kv_cache, attn_metadata)
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226]   File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226]     return self._call_impl(*args, **kwargs)
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226]   File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1541, in _call_impl
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226]     return forward_call(*args, **kwargs)
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226]   File "/usr/local/lib/python3.10/dist-packages/vllm/attention/layer.py", line 89, in forward
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226]     return self.impl.forward(query, key, value, kv_cache, attn_metadata,
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226]   File "/usr/local/lib/python3.10/dist-packages/vllm/attention/backends/flash_attn.py", line 339, in forward
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226]     output[:num_prefill_tokens] = flash_attn_varlen_func(
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226]   File "/usr/local/lib/python3.10/dist-packages/vllm_flash_attn/flash_attn_interface.py", line 1099, in flash_attn_varlen_func
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226]     return FlashAttnVarlenFunc.apply(
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226]   File "/usr/local/lib/python3.10/dist-packages/torch/autograd/function.py", line 598, in apply
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226]     return super().apply(*args, **kwargs)  # type: ignore[misc]
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226]   File "/usr/local/lib/python3.10/dist-packages/vllm_flash_attn/flash_attn_interface.py", line 596, in forward
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226]     out, q, k, v, out_padded, softmax_lse, S_dmask, rng_state = _flash_attn_varlen_forward(
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226]   File "/usr/local/lib/python3.10/dist-packages/vllm_flash_attn/flash_attn_interface.py", line 88, in _flash_attn_varlen_forward
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226]     out, q, k, v, out_padded, softmax_lse, S_dmask, rng_state = flash_attn_cuda.varlen_fwd(
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226] RuntimeError: CUDA error: an illegal memory access was encountered
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226] CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226] For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226] Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226]
(VllmWorkerProcess pid=1309) ERROR 06-20 14:07:59 multiproc_worker_utils.py:226]
[rank3]:[E ProcessGroupNCCL.cpp:1414] [PG 2 Rank 3] Process group watchdog thread terminated with exception: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

Exception raised from c10_cuda_check_implementation at ../c10/cuda/CUDAException.cpp:43 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f600937a897 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&) + 0x64 (0x7f600932ab25 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so)
frame #2: c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x118 (0x7f60097ac718 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10_cuda.so)
frame #3: c10d::ProcessGroupNCCL::WorkNCCL::finishedGPUExecutionInternal() const + 0x56 (0x7f5fbd04ae36 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
frame #4: c10d::ProcessGroupNCCL::WorkNCCL::isCompleted() + 0x58 (0x7f5fbd04ef38 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
frame #5: c10d::ProcessGroupNCCL::watchdogHandler() + 0x77c (0x7f5fbd0545ac in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
frame #6: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x10c (0x7f5fbd05531c in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
frame #7: <unknown function> + 0xdc253 (0x7f6008ab0253 in /usr/lib/x86_64-linux-gnu/libstdc++.so.6)
frame #8: <unknown function> + 0x94ac3 (0x7f600a0beac3 in /usr/lib/x86_64-linux-gnu/libc.so.6)
frame #9: <unknown function> + 0x126850 (0x7f600a150850 in /usr/lib/x86_64-linux-gnu/libc.so.6)

terminate called after throwing an instance of 'c10::DistBackendError'
  what():  [PG 2 Rank 3] Process group watchdog thread terminated with exception: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

Exception raised from c10_cuda_check_implementation at ../c10/cuda/CUDAException.cpp:43 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f600937a897 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&) + 0x64 (0x7f600932ab25 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so)
frame #2: c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x118 (0x7f60097ac718 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10_cuda.so)
frame #3: c10d::ProcessGroupNCCL::WorkNCCL::finishedGPUExecutionInternal() const + 0x56 (0x7f5fbd04ae36 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
frame #4: c10d::ProcessGroupNCCL::WorkNCCL::isCompleted() + 0x58 (0x7f5fbd04ef38 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
frame #5: c10d::ProcessGroupNCCL::watchdogHandler() + 0x77c (0x7f5fbd0545ac in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
frame #6: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x10c (0x7f5fbd05531c in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
frame #7: <unknown function> + 0xdc253 (0x7f6008ab0253 in /usr/lib/x86_64-linux-gnu/libstdc++.so.6)
frame #8: <unknown function> + 0x94ac3 (0x7f600a0beac3 in /usr/lib/x86_64-linux-gnu/libc.so.6)
frame #9: <unknown function> + 0x126850 (0x7f600a150850 in /usr/lib/x86_64-linux-gnu/libc.so.6)

Exception raised from ncclCommWatchdog at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1418 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f600937a897 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so)
frame #1: <unknown function> + 0xe32e33 (0x7f5fbccd7e33 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
frame #2: <unknown function> + 0xdc253 (0x7f6008ab0253 in /usr/lib/x86_64-linux-gnu/libstdc++.so.6)
frame #3: <unknown function> + 0x94ac3 (0x7f600a0beac3 in /usr/lib/x86_64-linux-gnu/libc.so.6)
frame #4: <unknown function> + 0x126850 (0x7f600a150850 in /usr/lib/x86_64-linux-gnu/libc.so.6)

ERROR 06-20 14:08:58 async_llm_engine.py:535] Engine iteration timed out. This should never happen!
ERROR 06-20 14:08:58 async_llm_engine.py:52] Engine background task failed
ERROR 06-20 14:08:58 async_llm_engine.py:52] Traceback (most recent call last):
ERROR 06-20 14:08:58 async_llm_engine.py:52]   File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 506, in engine_step
ERROR 06-20 14:08:58 async_llm_engine.py:52]     request_outputs = await self.engine.step_async()
ERROR 06-20 14:08:58 async_llm_engine.py:52]   File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 235, in step_async
ERROR 06-20 14:08:58 async_llm_engine.py:52]     output = await self.model_executor.execute_model_async(
ERROR 06-20 14:08:58 async_llm_engine.py:52]   File "/usr/local/lib/python3.10/dist-packages/vllm/executor/distributed_gpu_executor.py", line 166, in execute_model_async
ERROR 06-20 14:08:58 async_llm_engine.py:52]     return await self._driver_execute_model_async(execute_model_req)
ERROR 06-20 14:08:58 async_llm_engine.py:52]   File "/usr/local/lib/python3.10/dist-packages/vllm/executor/multiproc_gpu_executor.py", line 149, in _driver_execute_model_async
ERROR 06-20 14:08:58 async_llm_engine.py:52]     return await self.driver_exec_model(execute_model_req)
ERROR 06-20 14:08:58 async_llm_engine.py:52] asyncio.exceptions.CancelledError
ERROR 06-20 14:08:58 async_llm_engine.py:52]
ERROR 06-20 14:08:58 async_llm_engine.py:52] During handling of the above exception, another exception occurred:
ERROR 06-20 14:08:58 async_llm_engine.py:52]
ERROR 06-20 14:08:58 async_llm_engine.py:52] Traceback (most recent call last):
ERROR 06-20 14:08:58 async_llm_engine.py:52]   File "/usr/lib/python3.10/asyncio/tasks.py", line 456, in wait_for
ERROR 06-20 14:08:58 async_llm_engine.py:52]     return fut.result()
ERROR 06-20 14:08:58 async_llm_engine.py:52] asyncio.exceptions.CancelledError
ERROR 06-20 14:08:58 async_llm_engine.py:52]
ERROR 06-20 14:08:58 async_llm_engine.py:52] The above exception was the direct cause of the following exception:
ERROR 06-20 14:08:58 async_llm_engine.py:52]
ERROR 06-20 14:08:58 async_llm_engine.py:52] Traceback (most recent call last):
ERROR 06-20 14:08:58 async_llm_engine.py:52]   File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 42, in _log_task_completion
ERROR 06-20 14:08:58 async_llm_engine.py:52]     return_value = task.result()
ERROR 06-20 14:08:58 async_llm_engine.py:52]   File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 532, in run_engine_loop
ERROR 06-20 14:08:58 async_llm_engine.py:52]     has_requests_in_progress = await asyncio.wait_for(
ERROR 06-20 14:08:58 async_llm_engine.py:52]   File "/usr/lib/python3.10/asyncio/tasks.py", line 458, in wait_for
ERROR 06-20 14:08:58 async_llm_engine.py:52]     raise exceptions.TimeoutError() from exc
ERROR 06-20 14:08:58 async_llm_engine.py:52] asyncio.exceptions.TimeoutError
ERROR:asyncio:Exception in callback functools.partial(<function _log_task_completion at 0x7f8796cbea70>, error_callback=<bound method AsyncLLMEngine._error_callback of <vllm.engine.async_llm_engine.AsyncLLMEngine object at 0x7f884971aad0>>)
handle: <Handle functools.partial(<function _log_task_completion at 0x7f8796cbea70>, error_callback=<bound method AsyncLLMEngine._error_callback of <vllm.engine.async_llm_engine.AsyncLLMEngine object at 0x7f884971aad0>>)>
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 506, in engine_step
    request_outputs = await self.engine.step_async()
  File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 235, in step_async
    output = await self.model_executor.execute_model_async(
  File "/usr/local/lib/python3.10/dist-packages/vllm/executor/distributed_gpu_executor.py", line 166, in execute_model_async
    return await self._driver_execute_model_async(execute_model_req)
  File "/usr/local/lib/python3.10/dist-packages/vllm/executor/multiproc_gpu_executor.py", line 149, in _driver_execute_model_async
    return await self.driver_exec_model(execute_model_req)
asyncio.exceptions.CancelledError

[robertgshaw2-neuralmagic]

@ashgold does the model have quantized KV cache or just the layers?

[ashgold]

@robertgshaw2-neuralmagic just layers, not with --kv-cache-dtype argument.

[e3oroush]

I had the same issue, I guess the following two features have conflicts with each other:

• sliding window
• prefix cache

So we have to disable one. It means, we have to use --disable-sliding-window --enable-prefix-caching

[KrishnaM251]

Hi @mpoemsl, have you resolved the issue? If so, what steps did you take to do so? If not, can you please provide steps to reproduce it using the following template (replacing the italicized values with your values)?

• GPUs
  • 8x A6000
• Client code
  • _python3 benchmarks/benchmark_prefixcaching.py --model meta-llama/Meta-Llama-3-70B-Instruct --dataset-path ShareGPT.json --enable-prefix-caching --num-prompts 20 --repeat-count 5 --input-length-range 128:256
• Server code
  • _python -m vllm.entrypoints.openai.apiserver --model meta-llama/Meta-Llama-3-70B-Instruct --gpu-memory-utilization 0.40 --tensor-parallel-size 8 --max-model-len 2048 --trust-remote-code --enable-prefix-caching --max-num-seqs 128
• Env Vars
  • _export VLLM_ATTENTION_BACKEND=FLASHATTN
  • _export CUDA_VISIBLEDEVICES=0,1,2,3,4,5,6,7,8
• Version(s) Tested
  • v0.5.2

[kuangdao]

I had the same issue in deepseek, i use enable_prefix_caching=True , fp8 and tp = 2 vllm version is 0.5.1

[George-ao]

Is this issue solved in the latest version?

[kuangdao]

0.5.3 still meet error

[weimakeit]

same issue with vllm == 0.5.4 with APC enabled

[ashgold]

Will try this when I have the chance: #5376 (comment)

hi guys. "export VLLM_ATTENTION_BACKEND=XFORMERS" fixs this problem.

[skejriwal44]

VLLM_ATTENTION_BACKEND=XFORMERS gives suboptimal latencies - and prefix caching seems to not give any gains. Has anyone experienced this - and knows the reason? profile_xformers.csv profile.csv

[skejriwal44]

@simon-mo if you could please share some insights, it would be immensely helpful.

[Columpio]

I have the same issue with vllm v0.6.3.post1 using official docker.

The issue reproduces every time I send my 64 requests. When sending 1-4 requests, everything is ok. Hope it helps in debugging.

Output of `python collect_env.py` outside docker

``` Collecting environment information... WARNING 11-14 11:48:43 _custom_ops.py:19] Failed to import from vllm._C with ModuleNotFoundError("No module named 'vllm._C'") /home/columpio/ai-inference/vllm/vllm/connections.py:8: RuntimeWarning: Failed to read commit hash: No module named 'vllm._version' from vllm.version import __version__ as VLLM_VERSION PyTorch version: 2.4.0+cu121 Is debug build: False CUDA used to build PyTorch: 12.1 ROCM used to build PyTorch: N/A OS: Ubuntu 22.04.3 LTS (x86_64) GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 Clang version: Could not collect CMake version: Could not collect Libc version: glibc-2.35 Python version: 3.10.12 (main, Sep 11 2024, 15:47:36) [GCC 11.4.0] (64-bit runtime) Python platform: Linux-5.15.0-124-generic-x86_64-with-glibc2.35 Is CUDA available: True CUDA runtime version: 12.6.77 CUDA_MODULE_LOADING set to: LAZY GPU models and configuration: GPU 0: NVIDIA H100 PCIe GPU 1: NVIDIA H100 PCIe GPU 2: NVIDIA H100 PCIe GPU 3: NVIDIA H100 PCIe GPU 4: NVIDIA H100 PCIe GPU 5: NVIDIA H100 PCIe GPU 6: NVIDIA H100 PCIe GPU 7: NVIDIA H100 PCIe Nvidia driver version: 560.35.03 cuDNN version: Could not collect HIP runtime version: N/A MIOpen runtime version: N/A Is XNNPACK available: True CPU: Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Address sizes: 45 bits physical, 48 bits virtual Byte Order: Little Endian CPU(s): 64 On-line CPU(s) list: 0-63 Vendor ID: AuthenticAMD Model name: AMD EPYC 9374F 32-Core Processor CPU family: 25 Model: 17 Thread(s) per core: 1 Core(s) per socket: 32 Socket(s): 2 Stepping: 1 BogoMIPS: 7688.44 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext invpcid_single ssbd ibrs ibpb ibrs_enhanced vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves avx512_bf16 clzero wbnoinvd arat avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq rdpid overflow_recov succor fsrm flush_l1d Hypervisor vendor: VMware Virtualization type: full L1d cache: 2 MiB (64 instances) L1i cache: 2 MiB (64 instances) L2 cache: 64 MiB (64 instances) L3 cache: 64 MiB (2 instances) NUMA node(s): 2 NUMA node0 CPU(s): 0-31 NUMA node1 CPU(s): 32-63 Vulnerability Gather data sampling: Not affected Vulnerability Itlb multihit: Not affected Vulnerability L1tf: Not affected Vulnerability Mds: Not affected Vulnerability Meltdown: Not affected Vulnerability Mmio stale data: Not affected Vulnerability Reg file data sampling: Not affected Vulnerability Retbleed: Not affected Vulnerability Spec rstack overflow: Mitigation; safe RET Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Vulnerability Spectre v2: Mitigation; Enhanced / Automatic IBRS; IBPB conditional; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected Vulnerability Srbds: Not affected Vulnerability Tsx async abort: Not affected Versions of relevant libraries: [pip3] numpy==1.26.4 [pip3] nvidia-cublas-cu12==12.1.3.1 [pip3] nvidia-cuda-cupti-cu12==12.1.105 [pip3] nvidia-cuda-nvrtc-cu12==12.1.105 [pip3] nvidia-cuda-runtime-cu12==12.1.105 [pip3] nvidia-cudnn-cu12==9.1.0.70 [pip3] nvidia-cufft-cu12==11.0.2.54 [pip3] nvidia-curand-cu12==10.3.2.106 [pip3] nvidia-cusolver-cu12==11.4.5.107 [pip3] nvidia-cusparse-cu12==12.1.0.106 [pip3] nvidia-ml-py==12.560.30 [pip3] nvidia-nccl-cu12==2.20.5 [pip3] nvidia-nvjitlink-cu12==12.6.77 [pip3] nvidia-nvtx-cu12==12.1.105 [pip3] pyzmq==26.2.0 [pip3] torch==2.4.0 [pip3] torchvision==0.19.0 [pip3] transformers==4.46.2 [pip3] triton==3.0.0 [conda] Could not collect ROCM Version: Could not collect Neuron SDK Version: N/A vLLM Version: N/A (dev) vLLM Build Flags: CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled GPU Topology: GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7 CPU Affinity NUMA Affinity GPU NUMA ID GPU0 X NV12 PHB PHB PHB PHB PHB PHB 0-63 0-1 N/A GPU1 NV12 X PHB PHB PHB PHB PHB PHB 0-63 0-1 N/A GPU2 PHB PHB X PHB PHB PHB PHB NV12 0-63 0-1 N/A GPU3 PHB PHB PHB X NV12 PHB PHB PHB 0-63 0-1 N/A GPU4 PHB PHB PHB NV12 X PHB PHB PHB 0-63 0-1 N/A GPU5 PHB PHB PHB PHB PHB X NV12 PHB 0-63 0-1 N/A GPU6 PHB PHB PHB PHB PHB NV12 X PHB 0-63 0-1 N/A GPU7 PHB PHB NV12 PHB PHB PHB PHB X 0-63 0-1 N/A Legend: X = Self SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI) NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU) PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge) PIX = Connection traversing at most a single PCIe bridge NV# = Connection traversing a bonded set of # NVLinks ```

Output of `python collect_env.py` inside docker

``` Collecting environment information... WARNING 11-14 00:53:09 _custom_ops.py:19] Failed to import from vllm._C with ModuleNotFoundError("No module named 'vllm._C'") /ai-inference/vllm/vllm/connections.py:8: RuntimeWarning: Failed to read commit hash: No module named 'vllm._version' from vllm.version import __version__ as VLLM_VERSION PyTorch version: 2.4.0+cu121 Is debug build: False CUDA used to build PyTorch: 12.1 ROCM used to build PyTorch: N/A OS: Ubuntu 22.04.4 LTS (x86_64) GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 Clang version: Could not collect CMake version: Could not collect Libc version: glibc-2.35 Python version: 3.12.7 (main, Oct 1 2024, 08:52:12) [GCC 11.4.0] (64-bit runtime) Python platform: Linux-5.15.0-124-generic-x86_64-with-glibc2.35 Is CUDA available: True CUDA runtime version: Could not collect CUDA_MODULE_LOADING set to: LAZY GPU models and configuration: GPU 0: NVIDIA H100 PCIe GPU 1: NVIDIA H100 PCIe GPU 2: NVIDIA H100 PCIe GPU 3: NVIDIA H100 PCIe GPU 4: NVIDIA H100 PCIe GPU 5: NVIDIA H100 PCIe GPU 6: NVIDIA H100 PCIe GPU 7: NVIDIA H100 PCIe Nvidia driver version: 560.35.03 cuDNN version: Could not collect HIP runtime version: N/A MIOpen runtime version: N/A Is XNNPACK available: True CPU: Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Address sizes: 45 bits physical, 48 bits virtual Byte Order: Little Endian CPU(s): 64 On-line CPU(s) list: 0-63 Vendor ID: AuthenticAMD Model name: AMD EPYC 9374F 32-Core Processor CPU family: 25 Model: 17 Thread(s) per core: 1 Core(s) per socket: 32 Socket(s): 2 Stepping: 1 BogoMIPS: 7688.44 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext invpcid_single ssbd ibrs ibpb ibrs_enhanced vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves avx512_bf16 clzero wbnoinvd arat avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq rdpid overflow_recov succor fsrm flush_l1d Hypervisor vendor: VMware Virtualization type: full L1d cache: 2 MiB (64 instances) L1i cache: 2 MiB (64 instances) L2 cache: 64 MiB (64 instances) L3 cache: 64 MiB (2 instances) NUMA node(s): 2 NUMA node0 CPU(s): 0-31 NUMA node1 CPU(s): 32-63 Vulnerability Gather data sampling: Not affected Vulnerability Itlb multihit: Not affected Vulnerability L1tf: Not affected Vulnerability Mds: Not affected Vulnerability Meltdown: Not affected Vulnerability Mmio stale data: Not affected Vulnerability Reg file data sampling: Not affected Vulnerability Retbleed: Not affected Vulnerability Spec rstack overflow: Mitigation; safe RET Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Vulnerability Spectre v2: Mitigation; Enhanced / Automatic IBRS; IBPB conditional; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected Vulnerability Srbds: Not affected Vulnerability Tsx async abort: Not affected Versions of relevant libraries: [pip3] flashinfer==0.1.6+cu121torch2.4 [pip3] numpy==1.26.4 [pip3] nvidia-cublas-cu12==12.1.3.1 [pip3] nvidia-cuda-cupti-cu12==12.1.105 [pip3] nvidia-cuda-nvrtc-cu12==12.1.105 [pip3] nvidia-cuda-runtime-cu12==12.1.105 [pip3] nvidia-cudnn-cu12==9.1.0.70 [pip3] nvidia-cufft-cu12==11.0.2.54 [pip3] nvidia-curand-cu12==10.3.2.106 [pip3] nvidia-cusolver-cu12==11.4.5.107 [pip3] nvidia-cusparse-cu12==12.1.0.106 [pip3] nvidia-ml-py==12.560.30 [pip3] nvidia-nccl-cu12==2.20.5 [pip3] nvidia-nvjitlink-cu12==12.6.77 [pip3] nvidia-nvtx-cu12==12.1.105 [pip3] pyzmq==26.2.0 [pip3] torch==2.4.0 [pip3] torchvision==0.19.0 [pip3] transformers==4.45.2 [pip3] triton==3.0.0 [conda] Could not collect ROCM Version: Could not collect Neuron SDK Version: N/A vLLM Version: N/A (dev) vLLM Build Flags: CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled GPU Topology: GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7 CPU Affinity NUMA Affinity GPU NUMA ID GPU0 X NV12 PHB PHB PHB PHB PHB PHB 0-63 0-1 N/A GPU1 NV12 X PHB PHB PHB PHB PHB PHB 0-63 0-1 N/A GPU2 PHB PHB X PHB PHB PHB PHB NV12 0-63 0-1 N/A GPU3 PHB PHB PHB X NV12 PHB PHB PHB 0-63 0-1 N/A GPU4 PHB PHB PHB NV12 X PHB PHB PHB 0-63 0-1 N/A GPU5 PHB PHB PHB PHB PHB X NV12 PHB 0-63 0-1 N/A GPU6 PHB PHB PHB PHB PHB NV12 X PHB 0-63 0-1 N/A GPU7 PHB PHB NV12 PHB PHB PHB PHB X 0-63 0-1 N/A Legend: X = Self SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI) NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU) PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge) PIX = Connection traversing at most a single PCIe bridge NV# = Connection traversing a bonded set of # NVLinks ```

What I run:

```sh GPU_COUNT=8 MODEL_PATH=Qwen2.5-Coder-32B-Instruct QUANT_TYPE=bfloat16 PIPELINE_PARALLEL=1 PORT=9000 MAX_BATCH_SIZE=32 MAX_INPUT_TOKENS=20000 MAX_OUTPUT_TOKENS=10000 max_num_batched_tokens=64000 SERVED_MODEL_NAME=ensemble VLLM_ATTENTION_BACKEND=FLASH_ATTN docker run --rm -it --runtime nvidia --gpus all \ -v /home/models/huggingface/:/models \ -v /home/columpio/ai-inference:/ai-inference \ -p ${PORT}:${PORT} \ --ipc=host \ -e VLLM_ATTENTION_BACKEND=${VLLM_ATTENTION_BACKEND} \ -e VLLM_LOGGING_LEVEL=DEBUG \ -e CUDA_LAUNCH_BLOCKING=1 \ -e NCCL_DEBUG=TRACE \ -e VLLM_TRACE_FUNCTION=1 \ vllm/vllm-openai:v0.6.3.post1 \ --port ${PORT} \ --tensor-parallel-size ${GPU_COUNT} \ --pipeline-parallel-size ${PIPELINE_PARALLEL} \ --dtype ${QUANT_TYPE} \ --gpu-memory-utilization 0.9 \ --max-num-seqs ${MAX_BATCH_SIZE} \ --max-model-len $((MAX_INPUT_TOKENS + MAX_OUTPUT_TOKENS)) \ --max_num_batched_tokens ${max_num_batched_tokens} \ --served-model-name ${SERVED_MODEL_NAME} \ --model /models/${MODEL_PATH} 2>&1 | tee vllm.log ```

Then I send 64 requests with 15000 tokens each from another server. I get the following log: vllm2.log.

I also tried modifying the above parameters a bit and here is what I get. For all metrics I take median.

Parameters	The issue reproduces?	TTFT, sec.	e2e latency, sec.	tokens/sec.	symbols/sec.	ITL, sec.
VLLM_ATTENTION_BACKEND=FLASH_ATTN	yes
VLLM_ATTENTION_BACKEND=FLASH_ATTN --disable-sliding-window	yes
VLLM_ATTENTION_BACKEND=FLASH_ATTN --enforce-eager	no, but after 6 minutes requests still have not finished
VLLM_ATTENTION_BACKEND=ROCM_FLASH	Other issue: Keyword argument waves_per_eu was specified but unrecognised
VLLM_ATTENTION_BACKEND=ROCM_FLASH VLLM_USE_TRITON_FLASH_ATTN=0	Other issue: Error in calling custom op paged_attention_rocm: '_OpNamespace' '_rocm_C' object has no attribute 'paged_attention', possibly related to #9319
VLLM_ATTENTION_BACKEND=XFORMERS	Other issue: could not broadcast input array from shape (895,) into shape (512,)
VLLM_ATTENTION_BACKEND=TORCH_SDPA	Torch SDPA backend is only used for the CPU device
VLLM_ATTENTION_BACKEND=FLASHINFER	no	175	378	4.2	19.9	0.114