[Bug]: fixie-ai/ultravox-v0_4 throws Out of Memory

erkintelnyx commented 3 weeks ago

Your current environment

The output of `python collect_env.py`

```text PyTorch version: 2.6.0.dev20240918+rocm6.2 Is debug build: False CUDA used to build PyTorch: N/A ROCM used to build PyTorch: 6.2.41133-dd7f95766 OS: Ubuntu 20.04.6 LTS (x86_64) GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0 Clang version: 18.0.0git (https://github.com/RadeonOpenCompute/llvm-project roc-6.2.0 24292 26466ce804ac523b398608f17388eb6d605a3f09) CMake version: version 3.26.4 Libc version: glibc-2.31 Python version: 3.9.19 (main, May 6 2024, 19:43:03) [GCC 11.2.0] (64-bit runtime) Python platform: Linux-5.15.0-117-generic-x86_64-with-glibc2.31 Is CUDA available: True CUDA runtime version: Could not collect CUDA_MODULE_LOADING set to: LAZY GPU models and configuration: AMD Instinct MI100 (gfx908:sramecc+:xnack-) Nvidia driver version: Could not collect cuDNN version: Could not collect HIP runtime version: 6.2.41133 MIOpen runtime version: 3.2.0 Is XNNPACK available: True CPU: Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian Address sizes: 48 bits physical, 48 bits virtual CPU(s): 16 On-line CPU(s) list: 0-15 Thread(s) per core: 1 Core(s) per socket: 16 Socket(s): 1 NUMA node(s): 1 Vendor ID: AuthenticAMD CPU family: 25 Model: 1 Model name: AMD EPYC 7713 64-Core Processor Stepping: 1 CPU MHz: 2000.000 BogoMIPS: 4000.00 Virtualization: AMD-V Hypervisor vendor: KVM Virtualization type: full L1d cache: 1 MiB L1i cache: 1 MiB L2 cache: 8 MiB L3 cache: 16 MiB NUMA node0 CPU(s): 0-15 Vulnerability Gather data sampling: Not affected Vulnerability Itlb multihit: Not affected Vulnerability L1tf: Not affected Vulnerability Mds: Not affected Vulnerability Meltdown: Not affected Vulnerability Mmio stale data: Not affected Vulnerability Reg file data sampling: Not affected Vulnerability Retbleed: Not affected Vulnerability Spec rstack overflow: Mitigation; safe RET Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Vulnerability Spectre v2: Mitigation; Retpolines; IBPB conditional; IBRS_FW; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected Vulnerability Srbds: Not affected Vulnerability Tsx async abort: Not affected Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm rep_good nopl cpuid extd_apicid tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy svm cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw perfctr_core invpcid_single ssbd ibrs ibpb stibp vmmcall fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves clzero xsaveerptr wbnoinvd arat npt lbrv nrip_save tsc_scale vmcb_clean flushbyasid pausefilter pfthreshold v_vmsave_vmload vgif umip pku ospke vaes vpclmulqdq rdpid fsrm arch_capabilities Versions of relevant libraries: [pip3] mypy==1.8.0 [pip3] mypy-extensions==1.0.0 [pip3] numpy==1.26.4 [pip3] optree==0.9.1 [pip3] pytorch-triton-rocm==3.1.0+5fe38ffd73 [pip3] pyzmq==26.2.0 [pip3] torch==2.6.0.dev20240918+rocm6.2 [pip3] torchvision==0.20.0.dev20240918+rocm6.2 [pip3] transformers==4.46.0 [pip3] triton==3.0.0 [conda] No relevant packages ROCM Version: 6.2.41133-dd7f95766 Neuron SDK Version: N/A vLLM Version: 0.6.3.post2.dev116+g55137e8e.d20241029 vLLM Build Flags: CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled GPU Topology: ============================ ROCm System Management Interface ============================ ================================ Weight between two GPUs ================================= GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7 GPU0 0 15 15 15 40 40 40 40 GPU1 15 0 15 15 40 40 40 40 GPU2 15 15 0 15 40 40 40 40 GPU3 15 15 15 0 40 40 40 40 GPU4 40 40 40 40 0 15 15 15 GPU5 40 40 40 40 15 0 15 15 GPU6 40 40 40 40 15 15 0 15 GPU7 40 40 40 40 15 15 15 0 ================================= Hops between two GPUs ================================== GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7 GPU0 0 1 1 1 2 2 2 2 GPU1 1 0 1 1 2 2 2 2 GPU2 1 1 0 1 2 2 2 2 GPU3 1 1 1 0 2 2 2 2 GPU4 2 2 2 2 0 1 1 1 GPU5 2 2 2 2 1 0 1 1 GPU6 2 2 2 2 1 1 0 1 GPU7 2 2 2 2 1 1 1 0 =============================== Link Type between two GPUs =============================== GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7 GPU0 0 XGMI XGMI XGMI PCIE PCIE PCIE PCIE GPU1 XGMI 0 XGMI XGMI PCIE PCIE PCIE PCIE GPU2 XGMI XGMI 0 XGMI PCIE PCIE PCIE PCIE GPU3 XGMI XGMI XGMI 0 PCIE PCIE PCIE PCIE GPU4 PCIE PCIE PCIE PCIE 0 XGMI XGMI XGMI GPU5 PCIE PCIE PCIE PCIE XGMI 0 XGMI XGMI GPU6 PCIE PCIE PCIE PCIE XGMI XGMI 0 XGMI GPU7 PCIE PCIE PCIE PCIE XGMI XGMI XGMI 0 ======================================= Numa Nodes ======================================= GPU[0] : (Topology) Numa Node: 0 GPU[0] : (Topology) Numa Affinity: -1 GPU[1] : (Topology) Numa Node: 0 GPU[1] : (Topology) Numa Affinity: -1 GPU[2] : (Topology) Numa Node: 0 GPU[2] : (Topology) Numa Affinity: -1 GPU[3] : (Topology) Numa Node: 0 GPU[3] : (Topology) Numa Affinity: -1 GPU[4] : (Topology) Numa Node: 0 GPU[4] : (Topology) Numa Affinity: -1 GPU[5] : (Topology) Numa Node: 0 GPU[5] : (Topology) Numa Affinity: -1 GPU[6] : (Topology) Numa Node: 0 GPU[6] : (Topology) Numa Affinity: -1 GPU[7] : (Topology) Numa Node: 0 GPU[7] : (Topology) Numa Affinity: -1 ================================== End of ROCm SMI Log =================================== ```

Model Input Dumps

No response

🐛 Describe the bug

When I try to deploy the model, it throws OOM:

vllm serve fixie-ai/ultravox-v0_4 --tensor-parallel-size 2 --max-model-len 120000 --trust-remote-code

(VllmWorkerProcess pid=35734) INFO 11-06 17:51:01 model_runner.py:1066] Loading model weights took 8.1626 GB
INFO 11-06 17:51:01 model_runner.py:1066] Loading model weights took 8.1626 GB
(VllmWorkerProcess pid=35734) /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/modules/linear.py:125: UserWarning: Attempting to use hipBLASLt on an unsupported architecture! Overriding blas backend to hipblas (Triggered internally at ../aten/src/ATen/Context.cpp:288.)
(VllmWorkerProcess pid=35734)   return F.linear(input, self.weight, self.bias)
(VllmWorkerProcess pid=35734) INFO 11-06 17:51:06 model_runner_base.py:120] Writing input of failed execution to /tmp/err_execute_model_input_20241106-175106.pkl...
/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/modules/linear.py:125: UserWarning: Attempting to use hipBLASLt on an unsupported architecture! Overriding blas backend to hipblas (Triggered internally at ../aten/src/ATen/Context.cpp:288.)
  return F.linear(input, self.weight, self.bias)
INFO 11-06 17:51:07 model_runner_base.py:120] Writing input of failed execution to /tmp/err_execute_model_input_20241106-175107.pkl...
(VllmWorkerProcess pid=35734) INFO 11-06 17:51:07 model_runner_base.py:149] Completed writing input of failed execution to /tmp/err_execute_model_input_20241106-175106.pkl.
(VllmWorkerProcess pid=35734) ERROR 11-06 17:51:07 multiproc_worker_utils.py:229] Exception in worker VllmWorkerProcess while processing method determine_num_available_blocks.
(VllmWorkerProcess pid=35734) ERROR 11-06 17:51:07 multiproc_worker_utils.py:229] Traceback (most recent call last):
(VllmWorkerProcess pid=35734) ERROR 11-06 17:51:07 multiproc_worker_utils.py:229]   File "/vllm-workspace/vllm/worker/model_runner_base.py", line 116, in _wrapper
(VllmWorkerProcess pid=35734) ERROR 11-06 17:51:07 multiproc_worker_utils.py:229]     return func(*args, **kwargs)
(VllmWorkerProcess pid=35734) ERROR 11-06 17:51:07 multiproc_worker_utils.py:229]   File "/vllm-workspace/vllm/worker/model_runner.py", line 1657, in execute_model
(VllmWorkerProcess pid=35734) ERROR 11-06 17:51:07 multiproc_worker_utils.py:229]     hidden_or_intermediate_states = model_executable(
(VllmWorkerProcess pid=35734) ERROR 11-06 17:51:07 multiproc_worker_utils.py:229]   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
(VllmWorkerProcess pid=35734) ERROR 11-06 17:51:07 multiproc_worker_utils.py:229]     return self._call_impl(*args, **kwargs)
(VllmWorkerProcess pid=35734) ERROR 11-06 17:51:07 multiproc_worker_utils.py:229]   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
(VllmWorkerProcess pid=35734) ERROR 11-06 17:51:07 multiproc_worker_utils.py:229]     return forward_call(*args, **kwargs)
(VllmWorkerProcess pid=35734) ERROR 11-06 17:51:07 multiproc_worker_utils.py:229]   File "/vllm-workspace/vllm/model_executor/models/ultravox.py", line 468, in forward
(VllmWorkerProcess pid=35734) ERROR 11-06 17:51:07 multiproc_worker_utils.py:229]     audio_embeddings = self._process_audio_input(audio_input)
(VllmWorkerProcess pid=35734) ERROR 11-06 17:51:07 multiproc_worker_utils.py:229]   File "/vllm-workspace/vllm/model_executor/models/ultravox.py", line 420, in _process_audio_input
(VllmWorkerProcess pid=35734) ERROR 11-06 17:51:07 multiproc_worker_utils.py:229]     flattened_embeddings = self._audio_features_to_embeddings(
(VllmWorkerProcess pid=35734) ERROR 11-06 17:51:07 multiproc_worker_utils.py:229]   File "/vllm-workspace/vllm/model_executor/models/ultravox.py", line 380, in _audio_features_to_embeddings
(VllmWorkerProcess pid=35734) ERROR 11-06 17:51:07 multiproc_worker_utils.py:229]     audio_features = self.audio_tower(audio_input)
(VllmWorkerProcess pid=35734) ERROR 11-06 17:51:07 multiproc_worker_utils.py:229]   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
(VllmWorkerProcess pid=35734) ERROR 11-06 17:51:07 multiproc_worker_utils.py:229]     return self._call_impl(*args, **kwargs)
(VllmWorkerProcess pid=35734) ERROR 11-06 17:51:07 multiproc_worker_utils.py:229]   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
(VllmWorkerProcess pid=35734) ERROR 11-06 17:51:07 multiproc_worker_utils.py:229]     return forward_call(*args, **kwargs)
(VllmWorkerProcess pid=35734) ERROR 11-06 17:51:07 multiproc_worker_utils.py:229]   File "/vllm-workspace/vllm/model_executor/models/ultravox.py", line 320, in forward
(VllmWorkerProcess pid=35734) ERROR 11-06 17:51:07 multiproc_worker_utils.py:229]     layer_outputs = encoder_layer(
(VllmWorkerProcess pid=35734) ERROR 11-06 17:51:07 multiproc_worker_utils.py:229]   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
(VllmWorkerProcess pid=35734) ERROR 11-06 17:51:07 multiproc_worker_utils.py:229]     return self._call_impl(*args, **kwargs)
(VllmWorkerProcess pid=35734) ERROR 11-06 17:51:07 multiproc_worker_utils.py:229]   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
(VllmWorkerProcess pid=35734) ERROR 11-06 17:51:07 multiproc_worker_utils.py:229]     return forward_call(*args, **kwargs)
(VllmWorkerProcess pid=35734) ERROR 11-06 17:51:07 multiproc_worker_utils.py:229]   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 618, in forward
(VllmWorkerProcess pid=35734) ERROR 11-06 17:51:07 multiproc_worker_utils.py:229]     hidden_states, attn_weights, _ = self.self_attn(
(VllmWorkerProcess pid=35734) ERROR 11-06 17:51:07 multiproc_worker_utils.py:229]   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
(VllmWorkerProcess pid=35734) ERROR 11-06 17:51:07 multiproc_worker_utils.py:229]     return self._call_impl(*args, **kwargs)
(VllmWorkerProcess pid=35734) ERROR 11-06 17:51:07 multiproc_worker_utils.py:229]   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
(VllmWorkerProcess pid=35734) ERROR 11-06 17:51:07 multiproc_worker_utils.py:229]     return forward_call(*args, **kwargs)
(VllmWorkerProcess pid=35734) ERROR 11-06 17:51:07 multiproc_worker_utils.py:229]   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 545, in forward
(VllmWorkerProcess pid=35734) ERROR 11-06 17:51:07 multiproc_worker_utils.py:229]     attn_output = torch.nn.functional.scaled_dot_product_attention(
(VllmWorkerProcess pid=35734) ERROR 11-06 17:51:07 multiproc_worker_utils.py:229] torch.OutOfMemoryError: HIP out of memory. Tried to allocate 34.33 GiB. GPU 1 has a total capacity of 31.98 GiB of which 10.55 GiB is free. Of the allocated memory 20.23 GiB is allocated by PyTorch, and 784.37 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_HIP_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
(VllmWorkerProcess pid=35734) ERROR 11-06 17:51:07 multiproc_worker_utils.py:229]
(VllmWorkerProcess pid=35734) ERROR 11-06 17:51:07 multiproc_worker_utils.py:229] The above exception was the direct cause of the following exception:
(VllmWorkerProcess pid=35734) ERROR 11-06 17:51:07 multiproc_worker_utils.py:229]
(VllmWorkerProcess pid=35734) ERROR 11-06 17:51:07 multiproc_worker_utils.py:229] Traceback (most recent call last):
(VllmWorkerProcess pid=35734) ERROR 11-06 17:51:07 multiproc_worker_utils.py:229]   File "/vllm-workspace/vllm/executor/multiproc_worker_utils.py", line 223, in _run_worker_process
(VllmWorkerProcess pid=35734) ERROR 11-06 17:51:07 multiproc_worker_utils.py:229]     output = executor(*args, **kwargs)
(VllmWorkerProcess pid=35734) ERROR 11-06 17:51:07 multiproc_worker_utils.py:229]   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
(VllmWorkerProcess pid=35734) ERROR 11-06 17:51:07 multiproc_worker_utils.py:229]     return func(*args, **kwargs)
(VllmWorkerProcess pid=35734) ERROR 11-06 17:51:07 multiproc_worker_utils.py:229]   File "/vllm-workspace/vllm/worker/worker.py", line 223, in determine_num_available_blocks
(VllmWorkerProcess pid=35734) ERROR 11-06 17:51:07 multiproc_worker_utils.py:229]     self.model_runner.profile_run()
(VllmWorkerProcess pid=35734) ERROR 11-06 17:51:07 multiproc_worker_utils.py:229]   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
(VllmWorkerProcess pid=35734) ERROR 11-06 17:51:07 multiproc_worker_utils.py:229]     return func(*args, **kwargs)
(VllmWorkerProcess pid=35734) ERROR 11-06 17:51:07 multiproc_worker_utils.py:229]   File "/vllm-workspace/vllm/worker/model_runner.py", line 1304, in profile_run
(VllmWorkerProcess pid=35734) ERROR 11-06 17:51:07 multiproc_worker_utils.py:229]     self.execute_model(model_input, kv_caches, intermediate_tensors)
(VllmWorkerProcess pid=35734) ERROR 11-06 17:51:07 multiproc_worker_utils.py:229]   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
(VllmWorkerProcess pid=35734) ERROR 11-06 17:51:07 multiproc_worker_utils.py:229]     return func(*args, **kwargs)
(VllmWorkerProcess pid=35734) ERROR 11-06 17:51:07 multiproc_worker_utils.py:229]   File "/vllm-workspace/vllm/worker/model_runner_base.py", line 152, in _wrapper
(VllmWorkerProcess pid=35734) ERROR 11-06 17:51:07 multiproc_worker_utils.py:229]     raise type(err)(
(VllmWorkerProcess pid=35734) ERROR 11-06 17:51:07 multiproc_worker_utils.py:229] torch.OutOfMemoryError: Error in model execution (input dumped to /tmp/err_execute_model_input_20241106-175106.pkl): HIP out of memory. Tried to allocate 34.33 GiB. GPU 1 has a total capacity of 31.98 GiB of which 10.55 GiB is free. Of the allocated memory 20.23 GiB is allocated by PyTorch, and 784.37 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_HIP_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
INFO 11-06 17:51:07 model_runner_base.py:149] Completed writing input of failed execution to /tmp/err_execute_model_input_20241106-175107.pkl.
Process SpawnProcess-1:
ERROR 11-06 17:51:07 multiproc_worker_utils.py:116] Worker VllmWorkerProcess pid 35734 died, exit code: -15
INFO 11-06 17:51:07 multiproc_worker_utils.py:120] Killing local vLLM worker processes
Traceback (most recent call last):
  File "/vllm-workspace/vllm/worker/model_runner_base.py", line 116, in _wrapper
    return func(*args, **kwargs)
  File "/vllm-workspace/vllm/worker/model_runner.py", line 1657, in execute_model
    hidden_or_intermediate_states = model_executable(
  File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
  File "/vllm-workspace/vllm/model_executor/models/ultravox.py", line 468, in forward
    audio_embeddings = self._process_audio_input(audio_input)
  File "/vllm-workspace/vllm/model_executor/models/ultravox.py", line 420, in _process_audio_input
    flattened_embeddings = self._audio_features_to_embeddings(
  File "/vllm-workspace/vllm/model_executor/models/ultravox.py", line 380, in _audio_features_to_embeddings
    audio_features = self.audio_tower(audio_input)
  File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
  File "/vllm-workspace/vllm/model_executor/models/ultravox.py", line 320, in forward
    layer_outputs = encoder_layer(
  File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 618, in forward
    hidden_states, attn_weights, _ = self.self_attn(
  File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 545, in forward
    attn_output = torch.nn.functional.scaled_dot_product_attention(
torch.OutOfMemoryError: HIP out of memory. Tried to allocate 34.33 GiB. GPU 0 has a total capacity of 31.98 GiB of which 10.12 GiB is free. Of the allocated memory 20.23 GiB is allocated by PyTorch, and 784.37 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_HIP_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/conda/envs/py_3.9/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/opt/conda/envs/py_3.9/lib/python3.9/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/vllm-workspace/vllm/engine/multiprocessing/engine.py", line 397, in run_mp_engine
    engine = MQLLMEngine.from_engine_args(engine_args=engine_args,
  File "/vllm-workspace/vllm/engine/multiprocessing/engine.py", line 147, in from_engine_args
    return cls(ipc_path=ipc_path,
  File "/vllm-workspace/vllm/engine/multiprocessing/engine.py", line 83, in __init__
    self.engine = LLMEngine(*args, **kwargs)
  File "/vllm-workspace/vllm/engine/llm_engine.py", line 352, in __init__
    self._initialize_kv_caches()
  File "/vllm-workspace/vllm/engine/llm_engine.py", line 489, in _initialize_kv_caches
    self.model_executor.determine_num_available_blocks())
  File "/vllm-workspace/vllm/executor/distributed_gpu_executor.py", line 39, in determine_num_available_blocks
    num_blocks = self._run_workers("determine_num_available_blocks", )
  File "/vllm-workspace/vllm/executor/multiproc_gpu_executor.py", line 192, in _run_workers
    driver_worker_output = driver_worker_method(*args, **kwargs)
  File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
  File "/vllm-workspace/vllm/worker/worker.py", line 223, in determine_num_available_blocks
    self.model_runner.profile_run()
  File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
  File "/vllm-workspace/vllm/worker/model_runner.py", line 1304, in profile_run
    self.execute_model(model_input, kv_caches, intermediate_tensors)
  File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
  File "/vllm-workspace/vllm/worker/model_runner_base.py", line 152, in _wrapper
    raise type(err)(
torch.OutOfMemoryError: Error in model execution (input dumped to /tmp/err_execute_model_input_20241106-175107.pkl): HIP out of memory. Tried to allocate 34.33 GiB. GPU 0 has a total capacity of 31.98 GiB of which 10.12 GiB is free. Of the allocated memory 20.23 GiB is allocated by PyTorch, and 784.37 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_HIP_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
[rank0]:[W1106 17:51:08.187078063 ProcessGroupNCCL.cpp:1250] Warning: WARNING: process group has NOT been destroyed before we destruct ProcessGroupNCCL. On normal program exit, the application should call destroy_process_group to ensure that any pending NCCL operations have finished in this process. In rare cases this process can exit before this point and block the progress of another member of the process group. This constraint has always been present,  but this warning has only been added since PyTorch 2.4 (function operator())
Traceback (most recent call last):
  File "/opt/conda/envs/py_3.9/bin/vllm", line 33, in <module>
    sys.exit(load_entry_point('vllm', 'console_scripts', 'vllm')())
  File "/vllm-workspace/vllm/scripts.py", line 195, in main
    args.dispatch_function(args)
  File "/vllm-workspace/vllm/scripts.py", line 41, in serve
    uvloop.run(run_server(args))
  File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/uvloop/__init__.py", line 82, in run
    return loop.run_until_complete(wrapper())
  File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
  File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/uvloop/__init__.py", line 61, in wrapper
    return await main
  File "/vllm-workspace/vllm/entrypoints/openai/api_server.py", line 552, in run_server
    async with build_async_engine_client(args) as engine_client:
  File "/opt/conda/envs/py_3.9/lib/python3.9/contextlib.py", line 181, in __aenter__
    return await self.gen.__anext__()
  File "/vllm-workspace/vllm/entrypoints/openai/api_server.py", line 107, in build_async_engine_client
    async with build_async_engine_client_from_engine_args(
  File "/opt/conda/envs/py_3.9/lib/python3.9/contextlib.py", line 181, in __aenter__
    return await self.gen.__anext__()
  File "/vllm-workspace/vllm/entrypoints/openai/api_server.py", line 194, in build_async_engine_client_from_engine_args
    raise RuntimeError(
RuntimeError: Engine process failed to start

The model was working on a previous commit (08287ef6751e79a89bf4f060f5f9545560a6de12).

Is it because whisper and llama are trying be placed on the same GPU (this also doesn't work for tensor parallel size 4)? If so, where would be the part that can be tweaked for those to be scheduled to separate GPUs (if that is possible)?

Before submitting a new issue...

[X] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

DarkLight1337 commented 3 weeks ago

@youkaichao would the recent changes to torch.compile affect memory usage?

youkaichao commented 3 weeks ago

torch.compile is not enabled by default. we have all the wheels for every commit, and you can bisect to find which one caused the problem, see https://docs.vllm.ai/en/latest/getting_started/installation.html#install-the-latest-code

vllm-project / vllm