[Bug] ValueError: The memory capacity is unbalanced

[chuangzhidan]

Checklist

• [X] 1. I have searched related issues but cannot get the expected help.
• [ ] 2. The bug has not been fixed in the latest version.
• [X] 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.
• [X] 4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/sgl-project/sglang/discussions/new/choose Otherwise, it will be closed.
• [X] 5. Please use English, otherwise it will be closed.

Describe the bug

========== == CUDA ==

CUDA Version 12.1.1

Container image Copyright (c) 2016-2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.

This container image and its contents are governed by the NVIDIA Deep Learning Container License. By pulling and using the container, you accept the terms and conditions of this license: https://developer.nvidia.com/ngc/nvidia-deep-learning-container-license

A copy of this license is made available in this container at /NGC-DL-CONTAINER-LICENSE for your convenience.

[22:07:11] server_args=ServerArgs(model_path='/workspace/model/Qwen2.5-72-int4', tokenizer_path='/workspace/model/Qwen2.5-72-int4', tokenizer_mode='auto', skip_tokenizer_init=False, load_format='auto', dtype='auto', kv_cache_dtype='auto', trust_remote_code=False, context_length=None, quantization='gptq_marlin', served_model_name='/workspace/model/Qwen2.5-72-int4', chat_template=None, is_embedding=False, host='0.0.0.0', port=30000, additional_ports=[30001, 30002, 30003, 30004], mem_fraction_static=0.9, max_running_requests=None, max_num_reqs=None, max_total_tokens=None, chunked_prefill_size=8192, max_prefill_tokens=16384, schedule_policy='lpm', schedule_conservativeness=1.0, tp_size=2, stream_interval=1, random_seed=245707448, log_level='info', log_level_http=None, log_requests=False, show_time_cost=False, api_key=None, file_storage_pth='SGLang_storage', dp_size=1, load_balance_method='round_robin', disable_flashinfer=False, disable_flashinfer_sampling=False, disable_radix_cache=False, disable_regex_jump_forward=False, disable_cuda_graph=True, disable_cuda_graph_padding=False, disable_disk_cache=False, disable_custom_all_reduce=False, enable_mixed_chunk=False, enable_torch_compile=False, enable_p2p_check=False, enable_mla=False, triton_attention_reduce_in_fp32=False, nccl_init_addr=None, nnodes=1, node_rank=None) [22:07:13 TP0] Init nccl begin. [22:07:16 TP1] Init nccl begin. [22:07:25 TP1] Exception in run_tp_server: Traceback (most recent call last): File "/sgl-workspace/sglang/python/sglang/srt/managers/tp_worker.py", line 885, in run_tp_server model_server = ModelTpServer( File "/sgl-workspace/sglang/python/sglang/srt/managers/tp_worker.py", line 99, in init self.model_runner = ModelRunner( File "/sgl-workspace/sglang/python/sglang/srt/model_executor/model_runner.py", line 109, in init min_per_gpu_memory = self.init_torch_distributed() File "/sgl-workspace/sglang/python/sglang/srt/model_executor/model_runner.py", line 158, in init_torch_distributed raise ValueError( ValueError: The memory capacity is unbalanced. Some GPUs may be occupied by other processes.

Process Process-1:1: Traceback (most recent call last): File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap self.run() File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run self._target(*self._args, self._kwargs) File "/sgl-workspace/sglang/python/sglang/srt/managers/tp_worker.py", line 885, in run_tp_server model_server = ModelTpServer( File "/sgl-workspace/sglang/python/sglang/srt/managers/tp_worker.py", line 99, in init self.model_runner = ModelRunner( File "/sgl-workspace/sglang/python/sglang/srt/model_executor/model_runner.py", line 109, in init min_per_gpu_memory = self.init_torch_distributed() File "/sgl-workspace/sglang/python/sglang/srt/model_executor/model_runner.py", line 158, in init_torch_distributed raise ValueError( ValueError: The memory capacity is unbalanced**

memory usage is listed bellowed(A800*2, 80G): +-----------------------------------------------------------------------------------------+ | NVIDIA-SMI 560.35.03 Driver Version: 560.35.03 CUDA Version: 12.6 | |-----------------------------------------+------------------------+----------------------+ | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+========================+======================| | 0 NVIDIA A800 80GB PCIe On | 00000000:17:00.0 Off | 0 | | N/A 42C P0 68W / 300W | 48857MiB / 81920MiB | 0% Default | | | | Disabled | +-----------------------------------------+------------------------+----------------------+ | 1 NVIDIA A800 80GB PCIe On | 00000000:31:00.0 Off | 0 | | N/A 41C P0 69W / 300W | 38316MiB / 81920MiB | 0% Default | | | | Disabled | +-----------------------------------------+------------------------+----------------------+

Reproduction

docker run --gpus all -it -p 30000:30000 -v ~/.cache/huggingface:/root/.cache/huggingface -v /media/data/xgp:/workspace -v /media/data/llm:/workspace/model --env "HF_TOKEN=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" --env CUDA_VISIBLE_DEVICES=0,1 --ipc=host lmsysorg/sglang:latest python3 -m sglang.launch_server --model-path /workspace/model/Qwen2.5-72-int4 --host 0.0.0.0 --port 30000 --quantization gptq_marlin --mem-fraction-static 0.9 --disable-cuda-graph --tp-size 2

Environment

Python: 3.12.4 | packaged by Anaconda, Inc. | (main, Jun 18 2024, 15:12:24) [GCC 11.2.0] CUDA available: True GPU 0,1: NVIDIA A800 80GB PCIe GPU 0,1 Compute Capability: 8.0 CUDA_HOME: /usr NVCC: Cuda compilation tools, release 11.5, V11.5.119 CUDA Driver Version: 560.35.03 PyTorch: 2.4.0+cu121 sglang: 0.3.0 flashinfer: Module Not Found triton: 3.0.0 transformers: 4.44.2 requests: 2.32.3 tqdm: 4.66.5 numpy: 1.26.4 aiohttp: 3.10.5 fastapi: 0.113.0 hf_transfer: Module Not Found huggingface_hub: 0.24.6 interegular: 0.3.3 packaging: 24.1 PIL: 10.4.0 psutil: 6.0.0 pydantic: 2.9.0 uvicorn: 0.30.6 uvloop: 0.20.0 zmq: 26.2.0 vllm: 0.6.0 multipart: Module Not Found openai: 1.43.1 anthropic: Module Not Found litellm: Module Not Found NVIDIA Topology: GPU0 GPU1 CPU Affinity NUMA Affinity GPU NUMA ID GPU0 X NODE 0-23,48-71 0 N/A GPU1 NODE X 0-23,48-71 0 N/A

Legend:

X = Self SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI) NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU) PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge) PIX = Connection traversing at most a single PCIe bridge NV# = Connection traversing a bonded set of # NVLinks

[merrymercy]

The memory capacity is unbalanced. Some GPUs may be occupied by other processes.

Please kill all existing processes that occupy the GPU memory.

[chuangzhidan]

The memory capacity is unbalanced. Some GPUs may be occupied by other processes.

Please kill all existing processes that occupy the GPU memory.

yes， but it's not friendly and there is enough memory. vllm doesn't have this problem