希望服务稳定性提升

[yinghaodang]

我部署了xinference，版本是0.9.0. 一旦对显存的请求过大，服务就会hang住。 表现为：刷新UI没有任何模型显示，所有正在运行的模型全部异常退出，无法自动恢复。 日志信息表示：Out of memory.

目前解决方案：杀死容器，重新部署。

我心目中的服务，当自身负载过高时，有序处理当下任务，临时拒绝额外请求。保证服务可用。

[qinxuye]

日志完整的能贴一下吗

[yinghaodang]

我复现一下我的操作过程，方便复现。

version: '3.8'

services:
  xinference-local:
    image: xprobe/xinference:v0.9.0
    container_name: xinference-local
    ports:
      - 9998:9997
    environment:
      - XINFERENCE_MODEL_SRC=modelscope
      - XINFERENCE_HOME=/root/MODEL_PATH
    volumes:
      - ${MODEL_PATH}:/root/MODEL_PATH
    restart: always
    shm_size: '128g'
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    command: xinference-local -H 0.0.0.0 --log-level debug
    networks:
      - xinference-local
networks:
  xinference-local:
    driver: bridge
    ipam:
      driver: default
      config:
        - subnet: "172.30.2.0/24"

${MODEL_PATH}是一个文件夹，里面的子文件夹是模型文件夹。 我的显卡是4张英伟达的A100，然后在部署页面选择部署了72B int4量化的Qwen(这个是使用Xinference平台下载的模型)。同样方法测试过Baichuan2-7b-chat模型，都是独占4张卡，都是选择pytorch的模型。 下面是测试大模型使用的代码

 from ragas import evaluate

from langchain.chat_models import ChatOpenAI
from ragas.llms.base import LangchainLLMWrapper

inference_server_url = ""

# create vLLM Langchain instance
chat = ChatOpenAI(
    model="Baichuan2-13B-Chat",
    openai_api_key="no-key",
    openai_api_base=inference_server_url,
    max_tokens=1024,
    temperature=0.5,
)

# use the Ragas LangchainLLM wrapper to create a RagasLLM instance
vllm = LangchainLLMWrapper(chat)

from ragas.metrics import (
    context_precision,
    answer_relevancy,
    faithfulness,
    context_recall,
    context_relevancy,
    answer_correctness,
    answer_similarity
)
from ragas.metrics.critique import harmfulness

# change the LLM

faithfulness.llm = vllm
context_relevancy.llm = vllm
context_recall.llm = vllm
context_precision.llm = vllm
answer_similarity.llm = vllm
answer_relevancy.llm = vllm # Invalid key: 0 is out of bounds for size 0
answer_correctness.llm = vllm
harmfulness.llm = vllm

from langchain.embeddings import HuggingFaceEmbeddings

modelPath = "bge-large-zh"
# Create a dictionary with model configuration options, specifying to use the CPU for computations
model_kwargs = {'device':'cpu'}
# Create a dictionary with encoding options, specifically setting 'normalize_embeddings' to False
encode_kwargs = {'normalize_embeddings': True}
# Initialize an instance of HuggingFaceEmbeddings with the specified parameters
embeddings = HuggingFaceEmbeddings(
    model_name=modelPath,     # Provide the pre-trained model's path
    model_kwargs=model_kwargs, # Pass the model configuration options
    encode_kwargs=encode_kwargs # Pass the encoding options
)

answer_relevancy.embeddings = embeddings
answer_correctness.embeddings = embeddings
answer_similarity.embeddings = embeddings

# evaluate
from ragas import evaluate

result = evaluate(
    dataset,
    metrics=[context_precision], # 1
)

print(result)

测试代码大概是这个意思，删删改改多次，可能有bug。核心调用大模型的其实就是最后一句result = evaluate( dataset, metrics=[context_precision], # 1 ) 我测试的时候dataset大概只有100条数据，后台观察nvidia-smi，会发现显存占用率一直攀升，直到占满显存，显存占用归0. 后台日志就是普通的Out of memory.(现在卡还在其他用途，之后有空闲了再贴) 然后Web前端表现为Hang住，无法查看正在运行的模型，也无法重新部署模型。

[oubeichen]

所以是怎么解决的