[Feature]: Streaming prompt errors should log partial content

BerriAI / litellm

Python SDK, Proxy Server to call 100+ LLM APIs using the OpenAI format - [Bedrock, Azure, OpenAI, VertexAI, Cohere, Anthropic, Sagemaker, HuggingFace, Replicate, Groq]

https://docs.litellm.ai/docs/

Other

12.2k stars 1.42k forks source link

[Feature]: Streaming prompt errors should log partial content #4605

Open Manouchehri opened 2 months ago

Manouchehri commented 2 months ago

What happened?

In my opinion, if a streaming prompt errors out, we should still log all the content that we streamed so far to Langfuse. Right now, we only see the error message in the output in Langfuse.

Related ish to #4604 (see screenshot).

Relevant log output

No response

Twitter / LinkedIn details

https://twitter.com/DaveManouchehri

krrishdholakia commented 2 months ago

@Manouchehri how would you expect this to be logged?

is it individual generations?

Manouchehri commented 2 months ago

Could we keep the error message in Status Message, and just return the partial response in assistant/the messages object? This is what I would like to see: