search

abetlen / llama-cpp-python

Python bindings for llama.cpp
https://llama-cpp-python.readthedocs.io
MIT License
8.02k stars 951 forks source link

Recurrent errors in server mode #1541

Open mhoangvslev opened 4 months ago

mhoangvslev commented 4 months ago

It happens when I bombards the server with many requests. The errors are either RuntimeError, e.g., RuntimeError: Could not convert data into a valid instance of typing.Optional[typing.List[openai.types.chat.chat_completion_message_tool_call.ChatCompletionMessageToolCall]]), or RecursionError, e.g., exceeded while calling a Python object

abetlen commented 4 months ago

Hi @mhoangvslev can you provide more information on the chat format, model, and an example request.