search

vllm-project / vllm

A high-throughput and memory-efficient inference and serving engine for LLMs
https://docs.vllm.ai
Apache License 2.0
29.96k stars 4.53k forks source link

[Bug]: torch.OutOfMemoryError: CUDA out of memory. #8721

Closed tristandevs closed 1 month ago

tristandevs commented 1 month ago

Your current environment

The output of `python collect_env.py` ```text Your output of `python collect_env.py` here ``` GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 Clang version: Could not collect CMake version: version 3.29.3 Libc version: glibc-2.35 Python version: 3.10.12 (main, Sep 11 2024, 15:47:36) [GCC 11.4.0] (64-bit runtime) Python platform: Linux-6.8.0-1014-azure-x86_64-with-glibc2.35 Is CUDA available: True CUDA runtime version: 12.4.131 CUDA_MODULE_LOADING set to: LAZY GPU models and configuration: GPU 0: NVIDIA A100 80GB PCIe Nvidia driver version: 555.42.06 cuDNN version: Could not collect HIP runtime version: N/A MIOpen runtime version: N/A Is XNNPACK available: True CPU: Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Address sizes: 48 bits physical, 48 bits virtual Byte Order: Little Endian CPU(s): 24 On-line CPU(s) list: 0-23 Vendor ID: AuthenticAMD Model name: AMD EPYC 7V13 64-Core Processor CPU family: 25 Model: 1 Thread(s) per core: 1 Core(s) per socket: 24 Socket(s): 1 Stepping: 1 BogoMIPS: 4890.87 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc r ep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves user_shstk clzero x saveerptr rdpru arat umip vaes vpclmulqdq rdpid fsrm Hypervisor vendor: Microsoft Virtualization type: full L1d cache: 768 KiB (24 instances) L1i cache: 768 KiB (24 instances) L2 cache: 12 MiB (24 instances) L3 cache: 96 MiB (3 instances) NUMA node(s): 1 NUMA node0 CPU(s): 0-23 Vulnerability Gather data sampling: Not affected Vulnerability Itlb multihit: Not affected Vulnerability L1tf: Not affected Vulnerability Mds: Not affected Vulnerability Meltdown: Not affected Vulnerability Mmio stale data: Not affected Vulnerability Reg file data sampling: Not affected Vulnerability Retbleed: Not affected Vulnerability Spec rstack overflow: Vulnerable: Safe RET, no microcode Vulnerability Spec store bypass: Vulnerable Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Vulnerability Spectre v2: Mitigation; Retpolines; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected Vulnerability Srbds: Not affected Vulnerability Tsx async abort: Not affected Versions of relevant libraries: [pip3] numpy==1.26.4 [pip3] nvidia-cublas-cu12==12.1.3.1 [pip3] nvidia-cuda-cupti-cu12==12.1.105 [pip3] nvidia-cuda-nvrtc-cu12==12.1.105 [pip3] nvidia-cuda-runtime-cu12==12.1.105 [pip3] nvidia-cudnn-cu12==9.1.0.70 [pip3] nvidia-cufft-cu12==11.0.2.54 [pip3] nvidia-curand-cu12==10.3.2.106 [pip3] nvidia-cusolver-cu12==11.4.5.107 [pip3] nvidia-cusparse-cu12==12.1.0.106 [pip3] nvidia-ml-py==12.550.52 [pip3] nvidia-nccl-cu12==2.20.5 [pip3] nvidia-nvjitlink-cu12==12.4.127 [pip3] nvidia-nvtx-cu12==12.1.105 [pip3] pyzmq==26.0.3 [pip3] torch==2.4.0 [pip3] torchvision==0.19.0 [pip3] transformers==4.45.0.dev0 [pip3] triton==3.0.0 [pip3] vllm-nccl-cu12==2.18.1.0.4.0 [conda] Could not collect ROCM Version: Could not collect Neuron SDK Version: N/A vLLM Version: 0.6.1.post2@9ba0817ff1eb514f51cc6de9cb8e16c98d6ee44f vLLM Build Flags: CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled GPU Topology: GPU0 NIC0 CPU Affinity NUMA Affinity GPU NUMA ID GPU0 X SYS 0-23 0 N/A NIC0 SYS X Legend: X = Self SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI) NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU) PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge) PIX = Connection traversing at most a single PCIe bridge NV# = Connection traversing a bonded set of # NVLinks NIC Legend: NIC0: mlx5_0

Model Input Dumps

No response

🐛 Describe the bug

When I run python3 -m vllm.entrypoints.openai.api_server --model shuttleai/shuttle-3-GPTQ-Int4 --dtype bfloat16 --api-key 123456 --tensor-parallel-size 1 --gpu-memory-utilization 0.90 --served-model-name shuttle-2.5, I get an OOM error.

shuttle@ShuttleAI:~$ python3 -m vllm.entrypoints.openai.api_server --model shuttleai/shuttle-3-GPTQ-Int4 --dtype bfloat16 --api-key Shuttle1010AI --tensor-parallel-size 1 --gpu-memory-utilization 0.90 --served-model-name shuttle-2.5 --max-num-seqs 32 --max-model-len 4096
INFO 09-23 00:28:06 api_server.py:495] vLLM API server version 0.6.1.post2
INFO 09-23 00:28:06 api_server.py:496] args: Namespace(host=None, port=8000, uvicorn_log_level='info', allow_credentials=False, allowed_origins=['*'], allowed_methods=['*'], allowed_headers=['*'], api_key='Shuttle1010AI', lora_modules=None, prompt_adapters=None, chat_template=None, response_role='assistant', ssl_keyfile=None, ssl_certfile=None, ssl_ca_certs=None, ssl_cert_reqs=0, root_path=None, middleware=[], return_tokens_as_token_ids=False, disable_frontend_multiprocessing=False, enable_auto_tool_choice=False, tool_call_parser=None, model='shuttleai/shuttle-3-GPTQ-Int4', tokenizer=None, skip_tokenizer_init=False, revision=None, code_revision=None, tokenizer_revision=None, tokenizer_mode='auto', trust_remote_code=False, download_dir=None, load_format='auto', config_format='auto', dtype='bfloat16', kv_cache_dtype='auto', quantization_param_path=None, max_model_len=4096, guided_decoding_backend='outlines', distributed_executor_backend=None, worker_use_ray=False, pipeline_parallel_size=1, tensor_parallel_size=1, max_parallel_loading_workers=None, ray_workers_use_nsight=False, block_size=16, enable_prefix_caching=False, disable_sliding_window=False, use_v2_block_manager=False, num_lookahead_slots=0, seed=0, swap_space=4, cpu_offload_gb=0, gpu_memory_utilization=0.9, num_gpu_blocks_override=None, max_num_batched_tokens=None, max_num_seqs=32, max_logprobs=20, disable_log_stats=False, quantization=None, rope_scaling=None, rope_theta=None, enforce_eager=False, max_context_len_to_capture=None, max_seq_len_to_capture=8192, disable_custom_all_reduce=False, tokenizer_pool_size=0, tokenizer_pool_type='ray', tokenizer_pool_extra_config=None, limit_mm_per_prompt=None, enable_lora=False, max_loras=1, max_lora_rank=16, lora_extra_vocab_size=256, lora_dtype='auto', long_lora_scaling_factors=None, max_cpu_loras=None, fully_sharded_loras=False, enable_prompt_adapter=False, max_prompt_adapters=1, max_prompt_adapter_token=0, device='auto', num_scheduler_steps=1, scheduler_delay_factor=0.0, enable_chunked_prefill=None, speculative_model=None, speculative_model_quantization=None, num_speculative_tokens=None, speculative_draft_tensor_parallel_size=None, speculative_max_model_len=None, speculative_disable_by_batch_size=None, ngram_prompt_lookup_max=None, ngram_prompt_lookup_min=None, spec_decoding_acceptance_method='rejection_sampler', typical_acceptance_sampler_posterior_threshold=None, typical_acceptance_sampler_posterior_alpha=None, disable_logprobs_during_spec_decoding=None, model_loader_extra_config=None, ignore_patterns=[], preemption_mode=None, 
served_model_name=['shuttle-2.5'], qlora_adapter_name_or_path=None, otlp_traces_endpoint=None, collect_detailed_traces=None, disable_async_output_proc=False, override_neuron_config=None, disable_log_requests=False, max_log_len=None)
config.json: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 694/694 [00:00<00:00, 6.04MB/s]INFO 09-23 00:28:06 api_server.py:162] Multiprocessing frontend to use ipc:///tmp/710bc47a-e931-4c86-a6d9-230688c970d1 for RPC Path.
INFO 09-23 00:28:06 api_server.py:178] Started engine process with PID 4044
INFO 09-23 00:28:10 llm_engine.py:223] Initializing an LLM engine (v0.6.1.post2) with config: model='shuttleai/shuttle-3-GPTQ-Int4', speculative_config=None, tokenizer='shuttleai/shuttle-3-GPTQ-Int4', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, rope_scaling=None, rope_theta=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=4096, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=0, served_model_name=shuttle-2.5, use_v2_block_manager=False, num_scheduler_steps=1, enable_prefix_caching=False, use_async_output_proc=True)
tokenizer_config.json: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4.97k/4.97k [00:00<00:00, 42.0MB/s]tokenizer.json: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 7.03M/7.03M [00:00<00:00, 25.0MB/s]special_tokens_map.json: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 613/613 [00:00<00:00, 8.60MB/s]generation_config.json: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 248/248 [00:00<00:00, 2.56MB/s]INFO 09-23 00:28:11 model_runner.py:997] Starting to load model shuttleai/shuttle-3-GPTQ-Int4...
Process SpawnProcess-1:
Traceback (most recent call last):
  File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/home/shuttle/.local/lib/python3.10/site-packages/vllm/entrypoints/openai/rpc/server.py", line 236, in run_rpc_server
    server = AsyncEngineRPCServer(async_engine_args, usage_context, rpc_path)
  File "/home/shuttle/.local/lib/python3.10/site-packages/vllm/entrypoints/openai/rpc/server.py", line 34, in __init__
    self.engine = AsyncLLMEngine.from_engine_args(
  File "/home/shuttle/.local/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 573, in from_engine_args
    engine = cls(
  File "/home/shuttle/.local/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 473, in __init__
    self.engine = self._engine_class(*args, **kwargs)
  File "/home/shuttle/.local/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 257, in __init__
    super().__init__(*args, **kwargs)
  File "/home/shuttle/.local/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 317, in __init__
    self.model_executor = executor_class(
  File "/home/shuttle/.local/lib/python3.10/site-packages/vllm/executor/executor_base.py", line 47, in __init__
    self._init_executor()
  File "/home/shuttle/.local/lib/python3.10/site-packages/vllm/executor/gpu_executor.py", line 40, in _init_executor
    self.driver_worker.load_model()
  File "/home/shuttle/.local/lib/python3.10/site-packages/vllm/worker/worker.py", line 183, in load_model
    self.model_runner.load_model()
  File "/home/shuttle/.local/lib/python3.10/site-packages/vllm/worker/model_runner.py", line 999, in load_model
    self.model = get_model(model_config=self.model_config,
  File "/home/shuttle/.local/lib/python3.10/site-packages/vllm/model_executor/model_loader/__init__.py", line 19, in get_model
    return loader.load_model(model_config=model_config,
  File "/home/shuttle/.local/lib/python3.10/site-packages/vllm/model_executor/model_loader/loader.py", line 358, in load_model
    model = _initialize_model(model_config, self.load_config,
  File "/home/shuttle/.local/lib/python3.10/site-packages/vllm/model_executor/model_loader/loader.py", line 172, in _initialize_model
    return build_model(
  File "/home/shuttle/.local/lib/python3.10/site-packages/vllm/model_executor/model_loader/loader.py", line 157, in build_model
    return model_class(config=hf_config,
  File "/home/shuttle/.local/lib/python3.10/site-packages/vllm/model_executor/models/qwen2.py", line 341, in __init__
    self.model = Qwen2Model(config, cache_config, quant_config)
  File "/home/shuttle/.local/lib/python3.10/site-packages/vllm/model_executor/models/qwen2.py", line 243, in __init__
    self.start_layer, self.end_layer, self.layers = make_layers(
  File "/home/shuttle/.local/lib/python3.10/site-packages/vllm/model_executor/models/utils.py", line 248, in make_layers
    [PPMissingLayer() for _ in range(start_layer)] + [
  File "/home/shuttle/.local/lib/python3.10/site-packages/vllm/model_executor/models/utils.py", line 249, in <listcomp>
    maybe_offload_to_cpu(layer_fn(prefix=f"{prefix}.{idx}"))
  File "/home/shuttle/.local/lib/python3.10/site-packages/vllm/model_executor/models/qwen2.py", line 245, in <lambda>
    lambda prefix: Qwen2DecoderLayer(config=config,
  File "/home/shuttle/.local/lib/python3.10/site-packages/vllm/model_executor/models/qwen2.py", line 184, in __init__
    self.mlp = Qwen2MLP(
  File "/home/shuttle/.local/lib/python3.10/site-packages/vllm/model_executor/models/qwen2.py", line 69, in __init__
   self.down_proj = RowParallelLinear(intermediate_size,
  File "/home/shuttle/.local/lib/python3.10/site-packages/vllm/model_executor/layers/linear.py", line 975, in __init__
    self.quant_method.create_weights(
  File "/home/shuttle/.local/lib/python3.10/site-packages/vllm/model_executor/layers/linear.py", line 122, in create_weights
    weight = Parameter(torch.empty(sum(output_partition_sizes),
  File "/home/shuttle/.local/lib/python3.10/site-packages/torch/utils/_device.py", line 79, in __torch_function__
    return func(*args, **kwargs)
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 462.00 MiB. GPU 0 has a total capacity of 79.26 GiB of which 43.62 MiB is free. Including non-PyTorch memory, this process has 79.20 GiB memory in use. Of the allocated memory 78.72 GiB is allocated by PyTorch, and 1.64 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
ERROR 09-23 00:28:16 api_server.py:188] RPCServer process died before responding to readiness probe

I am pretty sure my GPU memory is enough to hold the model. The error happens before the model even started downloading, and I have tested it on my other a100 server, and another 4x a100 server and still get the same error.

I have tried setting --max-num-seqs 32 --max-model-len 4096 and lowering the GPU utilization, however, I still get the same error.

Before submitting a new issue...

tristandevs commented 1 month ago

Attempting to set PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True gives error Warning: expandable_segments not supported on this platform (function operator()) even with nightly pytorch build.

youkaichao commented 1 month ago

you need to pass correct quantization method to vllm. it looks like it is loading the model as if it is a normal qwen 2 model.

yangalan123 commented 1 month ago

May I ask what is the "lower GPU utilization" you use here? I have an ongoing PR trying to explain why we should try to set GPU_memory_utilization as small as possible: #8541 -- basically, GPU_memory_utilization only accounts for model_weight + KV_cache + CUDA Graph, and does not count running-time activation memory. I once came across this problem and I set GPU_memory_utilization to be as small as just a bit over model_weight (0.5 roughly for 70B Llama-3 fp16 model on 4A100 80G cards. A simple calculation: 70 2 = 140 GB needed, which is smaller than 80 4 0.5 = 160 GB) and no more OOM happened. It seems your model is some Qwen2 70B model quantized version by checking the model config on Huggingface, and I think under int4, you need at least 140/4=35GB to host the model.

By the way, it seems a bit weird that you use an int-4 quantized model with bf16 precision. Also, when you run your model on a 4*A100 card, did you set tensor_parallel_size=4 to use 4 GPUs?

tristandevs commented 1 month ago

Passing quantization gptq works.

The lower utilization I used was passing --gpu-memory-utilization 0.80.

When testing the 4x A100 I forgot to pass the right tensor size.

Thank you!

SnzFor16Min commented 1 month ago

No other GPU processes were running and I set tensor_parallel_size correctly, but still got the same OOM error from the same line weight = Parameter(torch.empty(sum(output_partition_sizes). After I set --enforce-eager=False to capture CUDA graphs, the problem seems to go away.