drikster80
commented
1 month ago I believe I was able to find a solution to this. It is related to OpenAI-Python #1454
Not sure why it works with fastapi 0.112.2 but fails in 0.113.0
Problem line:
https://github.com/vllm-project/vllm/blob/main/vllm/entrypoints/openai/api_server.py#L286
async def create_chat_completion(request: ChatCompletionRequest,
raw_request: Request):Confirmed Fix:
async def create_chat_completion(request: Annotated[dict, ChatCompletionRequest],
raw_request: Request):I'll make a PR on this and reference the issue. Can also add some try/catch with TypeAdapter validation, unless it's seen as unnecessary or impacts performance.
arshadshk
pachewise
tiangolo
DarkLight1337
darthhexx
Your current environment
The output of `python collect_env.py`
```text Collecting environment information... WARNING 09-05 21:11:49 cuda.py:22] You are using a deprecated `pynvml` package. Please install `nvidia-ml-py` instead, and make sure to uninstall `pynvml`. When both of them are installed, `pynvml` will take precedence and cause errors. See https://pypi.org/project/pynvml for more information. WARNING 09-05 21:11:49 _custom_ops.py:18] Failed to import from vllm._C with ModuleNotFoundError("No module named 'vllm._C'") /vllm/vllm/connections.py:8: RuntimeWarning: Failed to read commit hash: No module named 'vllm.commit_id' from vllm.version import __version__ as VLLM_VERSION PyTorch version: 2.4.0a0+3bcc3cddb5.nv24.07 Is debug build: False CUDA used to build PyTorch: 12.5 ROCM used to build PyTorch: N/A OS: Ubuntu 22.04.4 LTS (aarch64) GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 Clang version: Could not collect CMake version: version 3.30.0 Libc version: glibc-2.35 Python version: 3.10.12 (main, Jul 29 2024, 16:56:48) [GCC 11.4.0] (64-bit runtime) Python platform: Linux-6.5.0-1024-nvidia-64k-aarch64-with-glibc2.35 Is CUDA available: True CUDA runtime version: 12.5.82 CUDA_MODULE_LOADING set to: LAZY GPU models and configuration: GPU 0: NVIDIA GH200 480GB Nvidia driver version: 560.35.03 cuDNN version: Probably one of the following: /usr/lib/aarch64-linux-gnu/libcudnn.so.9.2.1 /usr/lib/aarch64-linux-gnu/libcudnn_adv.so.9.2.1 /usr/lib/aarch64-linux-gnu/libcudnn_cnn.so.9.2.1 /usr/lib/aarch64-linux-gnu/libcudnn_engines_precompiled.so.9.2.1 /usr/lib/aarch64-linux-gnu/libcudnn_engines_runtime_compiled.so.9.2.1 /usr/lib/aarch64-linux-gnu/libcudnn_graph.so.9.2.1 /usr/lib/aarch64-linux-gnu/libcudnn_heuristic.so.9.2.1 /usr/lib/aarch64-linux-gnu/libcudnn_ops.so.9.2.1 HIP runtime version: N/A MIOpen runtime version: N/A Is XNNPACK available: True CPU: Architecture: aarch64 CPU op-mode(s): 64-bit Byte Order: Little Endian CPU(s): 72 On-line CPU(s) list: 0-71 Vendor ID: ARM Model name: Neoverse-V2 Model: 0 Thread(s) per core: 1 Core(s) per socket: 72 Socket(s): 1 Stepping: r0p0 Frequency boost: disabled CPU max MHz: 3492.0000 CPU min MHz: 81.0000 BogoMIPS: 2000.00 Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm ssbs sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh bti L1d cache: 4.5 MiB (72 instances) L1i cache: 4.5 MiB (72 instances) L2 cache: 72 MiB (72 instances) L3 cache: 114 MiB (1 instance) NUMA node(s): 9 NUMA node0 CPU(s): 0-71 NUMA node1 CPU(s): NUMA node2 CPU(s): NUMA node3 CPU(s): NUMA node4 CPU(s): NUMA node5 CPU(s): NUMA node6 CPU(s): NUMA node7 CPU(s): NUMA node8 CPU(s): Vulnerability Gather data sampling: Not affected Vulnerability Itlb multihit: Not affected Vulnerability L1tf: Not affected Vulnerability Mds: Not affected Vulnerability Meltdown: Not affected Vulnerability Mmio stale data: Not affected Vulnerability Retbleed: Not affected Vulnerability Spec rstack overflow: Not affected Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl Vulnerability Spectre v1: Mitigation; __user pointer sanitization Vulnerability Spectre v2: Not affected Vulnerability Srbds: Not affected Vulnerability Tsx async abort: Not affected Versions of relevant libraries: [pip3] numpy==1.24.4 [pip3] nvidia-cudnn-frontend==1.5.1 [pip3] nvidia-dali-cuda120==1.39.0 [pip3] nvidia-ml-py==12.560.30 [pip3] nvidia-modelopt==0.13.0 [pip3] nvidia-nvimgcodec-cu12==0.2.0.7 [pip3] nvidia-pyindex==1.0.9 [pip3] onnx==1.16.0 [pip3] optree==0.12.1 [pip3] pynvml==11.4.1 [pip3] pytorch-triton==3.0.0+989adb9a2 [pip3] pyzmq==26.0.3 [pip3] torch==2.4.0a0+3bcc3cddb5.nv24.7 [pip3] torch-tensorrt==2.5.0a0 [pip3] torchvision==0.19.0a0 [pip3] transformers==4.44.2 [pip3] triton==3.0.0 [conda] Could not collect ROCM Version: Could not collect Neuron SDK Version: N/A vLLM Version: 0.6.0@COMMIT_HASH_PLACEHOLDER vLLM Build Flags: CUDA Archs: 9.0+PTX; ROCm: Disabled; Neuron: Disabled GPU Topology: GPU0 NIC0 NIC1 CPU Affinity NUMA Affinity GPU NUMA ID GPU0 X NODE NODE 0-71 0 1 NIC0 NODE X PIX NIC1 NODE PIX X Legend: X = Self SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI) NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU) PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge) PIX = Connection traversing at most a single PCIe bridge NV# = Connection traversing a bonded set of # NVLinks NIC Legend: NIC0: mlx5_0 NIC1: mlx5_1 ```🐛 Describe the bug
FastAPI released 0.113.0 about 5 hours ago. This release has a major refactor of Pydantic support. It appears this causes a Pydantic failure with the OpenAI-API calling.
Confirmed that reverting to FastAPI 0.112.2 resolves the problem (
pip install fastapi==0.112.2).Here are logs on the failure:
Before submitting a new issue...