max_tokens、temperature、top_p参数无法省略

[fevenor]

参考示例发送请求：

{
    "model": "gpt-3.5-turbo",
    "messages": [{"role": "user", "content": "Hello!"}]
}

返回报错：

{
    "detail": [
        {
            "type": "missing",
            "loc": [
                "body",
                "max_tokens"
            ],
            "msg": "Field required",
            "input": {
                "model": "gpt-3.5-turbo",
                "messages": [
                    {
                        "role": "user",
                        "content": "Hello!"
                    }
                ]
            },
            "url": "https://errors.pydantic.dev/2.0.3/v/missing"
        },
        {
            "type": "missing",
            "loc": [
                "body",
                "temperature"
            ],
            "msg": "Field required",
            "input": {
                "model": "gpt-3.5-turbo",
                "messages": [
                    {
                        "role": "user",
                        "content": "Hello!"
                    }
                ]
            },
            "url": "https://errors.pydantic.dev/2.0.3/v/missing"
        },
        {
            "type": "missing",
            "loc": [
                "body",
                "top_p"
            ],
            "msg": "Field required",
            "input": {
                "model": "gpt-3.5-turbo",
                "messages": [
                    {
                        "role": "user",
                        "content": "Hello!"
                    }
                ]
            },
            "url": "https://errors.pydantic.dev/2.0.3/v/missing"
        }
    ]
}

从错误看，max_tokens、temperature、top_p三个参数无法省略。

补全参数发送请求：

{
    "model": "gpt-3.5-turbo",
    "messages": [{"role": "user", "content": "Hello!"}],
    "max_tokens": 16,
    "temperature": 1,
    "top_p": 1
}

返回成功：

{
    "id": "chatcmpl-77PZm95TtxE0oYLRx3cxa6HtIDI7s",
    "object": "chat.completion",
    "created": 1682000966,
    "model": "gpt-3.5-turbo-0301",
    "usage": {
        "prompt_tokens": 0,
        "completion_tokens": 0,
        "total_tokens": 0
    },
    "choices": [
        {
            "message": {
                "role": "assistant",
                "content": "Hello! How can I help you today?"
            },
            "finish_reason": "stop",
            "index": 0
        }
    ]
}

[ninehills]

我这里重新初始化项目后无法复现，请参考如下提供完成的 curl 命令参数：

$ curl https://66c3-34-91-222-112.ngrok.io/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer token1" \          
  -d '{
    "model": "gpt-3.5-turbo",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'
{"id":"chatcmpl-77PZm95TtxE0oYLRx3cxa6HtIDI7s","object":"chat.completion","created":1682000966,"model":"gpt-3.5-turbo-0301","usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0},"choices":[{"message":{"role":"assistant","content":"Hello! How can I assist you today?"},"finish_reason":"stop","index":0}]}% 

[fevenor]

运行环境：Ubuntu 22.04.2 on WSL Python 3.10.6

启动命令：

CUDA_VISIBLE_DEVICES=0 python3 main.py --port 8080 --llm_model chatglm-6b-int8

本地执行curl：

curl -X POST http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -H "Authorization: Bearer token" -d '{"model": "gpt-3.5-turbo","messages": [{"role": "user", "content": "Hello!"}]}'
{"detail":[{"type":"missing","loc":["body","max_tokens"],"msg":"Field required","input":{"model":"gpt-3.5-turbo","messages":[{"role":"user","content":"Hello!"}]},"url":"https://errors.pydantic.dev/2.0.3/v/missing"},{"type":"missing","loc":["body","temperature"],"msg":"Field required","input":{"model":"gpt-3.5-turbo","messages":[{"role":"user","content":"Hello!"}]},"url":"https://errors.pydantic.dev/2.0.3/v/missing"},{"type":"missing","loc":["body","top_p"],"msg":"Field required","input":{"model":"gpt-3.5-turbo","messages":[{"role":"user","content":"Hello!"}]},"url":"https://errors.pydantic.dev/2.0.3/v/missing"}]}

[ninehills]

运行环境：Ubuntu 22.04.2 on WSL Python 3.10.6

确定是最新版本的代码么，之前的某个版本应该是有类似的问题。可以head -30 app.py 看下app.py 的内容.

如果确定代码相同的话，就辛苦 pip freeze | grep pydantic 看下 pydantic 的版本

[fevenor]

是5月27日的版本

head -30 app.py内容：

#!/usr/bin/env python3
# -*- coding: utf-8 -*-
import json
from typing import List, Optional, Any

from fastapi import FastAPI, HTTPException, Request, status, BackgroundTasks
from fastapi.middleware.cors import CORSMiddleware
from fastapi.responses import JSONResponse
from pydantic import BaseModel
from sse_starlette.sse import EventSourceResponse

from context import context
from utils import torch_gc

app = FastAPI()

app.add_middleware(
    CORSMiddleware,
    allow_origins=['*'],
    allow_credentials=True,
    allow_methods=['*'],
    allow_headers=['*'],
)

class Message(BaseModel):
    role: str
    content: str

sha1sum app.py结果

39cd0dee41e2d7b5d5f96e9c68e1f9d79ab62c90  app.py

pip freeze | grep pydantic结果

pydantic==2.0.3
pydantic_core==2.3.0

[ninehills]

curl -X POST http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -H "Authorization: Bearer token" -d '{"model": "gpt-3.5-turbo","messages": [{"role": "user", "content": "Hello!"}]}'

确定是 pydantic 版本的问题，我修改了 requirements.txt 限制 pydantic 版本为 1.x 可以解决