ValueError: The model's max seq len (32768) is larger than the maximum number of tokens that can be stored in KV cache (15248). Try increasing `gpu_memory_utilization` or decreasing `max_model_len` when initializing the engine.

[guiniao]

提交前必须检查以下项目 | The following items must be checked before submission

• [X] 请确保使用的是仓库最新代码（git pull），一些问题已被解决和修复。 | Make sure you are using the latest code from the repository (git pull), some issues have already been addressed and fixed.
• [X] 我已阅读项目文档和FAQ章节并且已在Issue中对问题进行了搜索，没有找到相似问题和解决方案 | I have searched the existing issues / discussions

问题类型 | Type of problem

启动命令 | Startup command

操作系统 | Operating system

Linux

详细描述问题 | Detailed description of the problem

使用vllm方式部署api，使用的模型是qwen1.5-14b-chat .env配置如下：

PORT=8000

model related

MODEL_NAME=qwen MODEL_PATH=./models/qwen-1.5-14b-chat PROMPT_NAME= EMBEDDING_NAME=

device related

GPU设备并行化策略

DEVICE_MAP=auto

GPU数量

NUM_GPUs=2

api related

API_PREFIX=/v1

vllm related

ENGINE=vllm TRUST_REMOTE_CODE=true TOKENIZE_MODE=slow TENSOR_PARALLEL_SIZE=1

开启半精度，可以加快运行速度、减少GPU占用

DTYPE=half

API_KEY，此处随意填一个字符串即可

OPENAI_API_KEY=

启动 python server.py时报错 如下： ValueError: The model's max seq len (32768) is larger than the maximum number of tokens that can be stored in KV cache (15248). Try increasing gpu_memory_utilization or decreasing max_model_len when initializing the engine.

看了一些处理方案python server.py --max-model-len 24320，也不起作用

另外我设置了NUM_GPUs=2，显示还是只用了一张卡

Dependencies

# 请在此处粘贴依赖情况
# Please paste the dependencies here

peft 0.10.0 sentence-transformers 2.6.1 torch 2.1.2 transformers 4.39.3 transformers-stream-generator 0.0.5

运行日志或截图 | Runtime logs or screenshots

# 请在此处粘贴运行日志
# Please paste the run log here

[xusenlinzy]

更新一下项目代码，然后更改下面的配置

TENSOR_PARALLEL_SIZE=2 # GPU数量 MODEL_NAME=qwen2 PROMPT_NAME=qwen2

如果显存不够启动32k的话，可以设置CONTEXT_LEN=8192