[Bug] qwen272b- lmDeploy hanged after 900 requests

ChunyiY commented 3 months ago

Checklist

[X] 1. I have searched related issues but cannot get the expected help.
[X] 2. The bug has not been fixed in the latest version.
[X] 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.

Describe the bug

I ran this code and did a load test based on my code, testing qwen2-72b. I am on a remote server with 4 A100 gpus to deploy this. I used locust as the platform to run my load test, I tried multiple user counts such as 100, 50 ,10, but they all hanged up after completing 900 requests.

After 900 requests, the model stopped processing new requests. 服务器在处理了900个请求之后挂住，不知道应该如何解决？？

Reproduction

import subprocess
import time
from fastapi import FastAPI, HTTPException, Request
from pydantic import BaseModel
from transformers import AutoTokenizer
import torch
import httpx
import asyncio
from concurrent.futures import ProcessPoolExecutor
import logging
import multiprocessing
from collections import deque

app = FastAPI()

class PredictRequest(BaseModel):
    prompt: str

class PredictResponse(BaseModel):
    output: str

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

model_path = "/home/bob.yang1/Qwen-7B-Chat"
num_gpus = int(os.environ.get("NUM_GPUS", 4))

tokenizer = AutoTokenizer.from_pretrained(model_path)

os.environ["PATH"] += os.pathsep + "/home/bob.yang1/.local/bin"

def start_lmdeploy_server():
    command = (
        f"lmdeploy serve api_server {model_path} "
        f"--server-name 0.0.0.0 --server-port 23333 "
        f"--backend turbomind --tp {num_gpus} "
    )
    logger.info(f"Starting lmdeploy server with command: {command}")
    process = subprocess.Popen(command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
    while True:
        line = process.stdout.readline().decode('utf-8').strip()
        logger.info(line)
        if "API server started" in line or "HINT:" in line:
            logger.info("lmdeploy server is ready.")
            break
        if process.poll() is not None:
            logger.error("lmdeploy server failed to start.")
            raise RuntimeError("lmdeploy server failed to start.")
        time.sleep(1)

start_lmdeploy_server()

class ModelManager:
    def __init__(self, model_path, num_instances):
        self.instances = [self.create_instance() for _ in range(num_instances)]
        self.lock = asyncio.Lock()

    def create_instance(self):
        return {"model_path": model_path}

    async def get_instance(self, idx):
        async with self.lock:
            return self.instances[idx % len(self.instances)]

model_manager = ModelManager(model_path, num_gpus)

max_queue_size = 30
request_queue = deque(maxlen=max_queue_size)
processed_requests = 0

@app.post("/predict", response_model=PredictResponse)
async def predict(data: PredictRequest):
    global processed_requests
    global request_queue

    if len(request_queue) >= max_queue_size:
        request_queue = deque(list(request_queue)[:20], maxlen=max_queue_size)
        logger.warning("服务器拥挤，稍后再试")
        raise HTTPException(status_code=503, detail="服务器拥挤，稍后再试")

    request_queue.append(data.prompt)
    logger.info(f"Queue size: {len(request_queue)}")

    try:
        torch.cuda.empty_cache()
        messages = [
            {"role": "system", "content": "You are a helpful assistant."},
            {"role": "user", "content": data.prompt}
        ]
        instance = await model_manager.get_instance(int(hash(data.prompt)))
        async with httpx.AsyncClient(timeout=httpx.Timeout(60.0), limits=httpx.Limits(max_connections=100, max_keepalive_connections=20)) as client:
            async with client.stream("POST", f"http://localhost:23333/v1/chat/completions", json={
                "model": instance["model_path"],
                "messages": messages,
                "temperature": 0.7,
                "top_p": 0.8,
                "max_tokens": 524,
                "repetition_penalty": 1.05
            }) as response:
                if response.status_code == 200:
                    output_text = ""
                    async for chunk in response.aiter_text():
                        output_text += chunk
                    if request_queue:
                        request_queue.popleft()
                    processed_requests += 1
                    logger.info(f"Processed requests: {processed_requests}, Pending requests: {len(request_queue)}")
                    return {"output": output_text}
                else:
                    result = await response.json()
                    if request_queue:
                        request_queue.popleft()
                    logger.info(f"Processed requests: {processed_requests}, Pending requests: {len(request_queue)}")
                    raise HTTPException(status_code=response.status_code, detail=result)
    except torch.cuda.OutOfMemoryError as e:
        logger.error(f"CUDA out of memory: {e}")
        if request_queue:
            request_queue.popleft()
        logger.info(f"Processed requests: {processed_requests}, Pending requests: {len(request_queue)}")
        raise HTTPException(status_code=500, detail="CUDA out of memory")
    except httpx.RequestError as e:
        logger.error(f"Request error: {e}")
        if request_queue:
            request_queue.popleft()
        logger.info(f"Processed requests: {processed_requests}, Pending requests: {len(request_queue)}")
        raise HTTPException(status_code=500, detail="Request error")
    except Exception as e:
        logger.error(f"Unexpected error: {e}")
        if request_queue:
            request_queue.popleft()
        logger.info(f"Processed requests: {processed_requests}, Pending requests: {len(request_queue)}")
        raise HTTPException(status_code=500, detail=str(e))

@app.middleware("http")
async def add_process_time_header(request: Request, call_next):
    start_time = time.time()
    response = await call_next(request)
    process_time = time.time() - start_time
    response.headers["X-Process-Time"] = str(process_time)
    return response

@app.on_event("startup")
async def startup_event():
    asyncio.create_task(health_check_and_restart())

async def health_check_and_restart():
    while True:
        await asyncio.sleep(300)
        try:
            async with httpx.AsyncClient() as client:
                response = await client.get("http://localhost:23333/health")
                if response.status_code != 200:
                    logger.warning("Health check failed, restarting lmdeploy server.")
                    start_lmdeploy_server()
        except Exception as e:
            logger.error(f"Health check exception: {e}")
            start_lmdeploy_server()

def run_app():
    import uvicorn
    import uvloop

    uvloop.install()

    loop = asyncio.get_event_loop()
    executor = ProcessPoolExecutor(max_workers=num_gpus)
    loop.set_default_executor(executor)

    uvicorn.run(app, host="0.0.0.0", port=9000)

if __name__ == "__main__":
    num_workers = num_gpus
    processes = []
    for _ in range(num_workers):
        p = multiprocessing.Process(target=run_app)
        p.start()
        processes.append(p)

    for p in processes:
        p.join()

Environment

/home/bob.yang1/.local/bin/lmdeploy check_env
/home/bob.yang1/.local/lib/python3.10/site-packages/_distutils_hack/__init__.py:55: UserWarning: Reliance on distutils from stdlib is deprecated. Users must rely on setuptools to provide the distutils module. Avoid importing distutils or import setuptools first, and avoid setting SETUPTOOLS_USE_DISTUTILS=stdlib. Register concerns at Build software better, together
  warnings.warn(
sys.platform: linux
Python: 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0]
CUDA available: True
MUSA available: False
numpy_random_seed: 2147483648
GPU 0,1,2,3: NVIDIA A100-SXM4-80GB
CUDA_HOME: /usr/local/cuda
NVCC: Cuda compilation tools, release 12.2, V12.2.128
GCC: x86_64-linux-gnu-gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
PyTorch: 2.5.0.dev20240720+cu121
PyTorch compiling details: PyTorch built with:
  - GCC 9.3
  - C++ Version: 201703
  - Intel(R) oneAPI Math Kernel Library Version 2024.2-Product Build 20240605 for Intel(R) 64 architecture applications
  - Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67)
  - OpenMP 201511 (a.k.a. OpenMP 4.5)
  - LAPACK is enabled (usually provided by MKL)
  - NNPACK is enabled
  - CPU capability usage: AVX2
  - CUDA Runtime 12.1
  - NVCC architecture flags: -gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_90,code=sm_90
  - CuDNN 90.1  (built against CUDA 12.4)
  - Magma 2.6.1
  - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=12.1, CUDNN_VERSION=9.1.0, CXX_COMPILER=/opt/rh/devtoolset-9/root/usr/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=0 -fabi-version=11 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=pedantic -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=2.5.0, USE_CUDA=ON, USE_CUDNN=ON, USE_CUSPARSELT=1, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=1, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, 

TorchVision: 0.20.0.dev20240722+cu121
LMDeploy: 0.5.1+
transformers: 4.42.4
gradio: Not Found
fastapi: 0.111.0
pydantic: 2.8.2
triton: 3.0.0+dedb7bdf33
NVIDIA Topology: 
        GPU0    GPU1    GPU2    GPU3    NIC0    NIC1    NIC2    CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0     X      NV12    NV12    NV12    SYS     PXB     SYS     0-31    0               N/A
GPU1    NV12     X      NV12    NV12    SYS     PXB     SYS     0-31    0               N/A
GPU2    NV12    NV12     X      NV12    SYS     SYS     PXB     96-127  3               N/A
GPU3    NV12    NV12    NV12     X      SYS     SYS     PXB     96-127  3               N/A
NIC0    SYS     SYS     SYS     SYS      X      SYS     SYS
NIC1    PXB     PXB     SYS     SYS     SYS      X      SYS
NIC2    SYS     SYS     PXB     PXB     SYS     SYS      X 

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

NIC Legend:

  NIC0: mlx5_0
  NIC1: mlx5_1
  NIC2: mlx5_2

Error traceback

No response

irexyc commented 3 months ago

lmdeploy 服务端的日志是怎样的呢？

Wondersui commented 1 month ago

I have encountered similar problems, feeling that the health_check block is the default return 200, sometimes the request is too high service crash will not automatically restart

github-actions[bot] commented 3 weeks ago

This issue is marked as stale because it has been marked as invalid or awaiting response for 7 days without any further response. It will be closed in 5 days if the stale label is not removed or if there is no further response.

github-actions[bot] commented 2 weeks ago

This issue is closed because it has been stale for 5 days. Please open a new issue if you have similar issues or you have any new updates now.

InternLM / lmdeploy