search

brainlid / langchain

Elixir implementation of an AI focused LangChain style framework.
https://hexdocs.pm/langchain/
Other
668 stars 72 forks source link

Handle Anthropic overloaded errors during streaming #193

Open doomspork opened 3 days ago

doomspork commented 3 days ago

As we're making more use of streaming we're seeing more of these errors recently:

[error] Unsupported event received when parsing Anthropic response: "event: error\ndata: {\"type\":\"error\",\"error\":{\"details\":null,\"type\":\"overloaded_error\",\"message\":\"Overloaded\"}        }"

The result is this error logged but the code seems to proceed like everything is okay. I would expect/prefer an error were raised so we could retry the conversation versus leaving it in a partial and incomplete state.

I started spelunking today but it looks like handling this could be a decent sized changes so I thought it was best to bring it here for discussion.

Anthropic documentation around streaming errors for reference: https://docs.anthropic.com/en/api/messages-streaming#error-events

brainlid commented 3 days ago

Just saw this myself this morning! Thanks for reporting it.

brainlid commented 2 days ago

There's more to handling this than I've done so far. It helps that they keep making it reproducible. :man_facepalming:

doomspork commented 2 days ago

I haven't had a chance to pull your changes and do my own testing but I will here shortly.

It's unfortunate how frequently their API is overloaded recently.

brainlid commented 2 days ago

Yeah. I’m going to take a different approach. I’ll try raising an exception and catching it in the LLMChain.

So no, it’s not working great yet.

On Fri, Nov 22, 2024 at 7:37 AM Sean Callan @.***> wrote:

I haven't had a chance to pull your changes and do my own testing but I will here shortly.

It's unfortunate how frequently their API is overloaded recently.

— Reply to this email directly, view it on GitHub https://github.com/brainlid/langchain/issues/193#issuecomment-2493909068, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAGFQGGHCD4C2EKJRPPBGJT2B46TPAVCNFSM6AAAAABSHHUKYSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIOJTHEYDSMBWHA . You are receiving this because you modified the open/close state.Message ID: @.***>

brainlid commented 2 days ago

@doomspork I pushed up a hopefully more complete fix.

When it happens again, the LLMChain.run(chain, ...) result will be: {:error, updated_chain, %LangChain.LangChainError{} = error}.

Where the error has a type of "overloaded_error" (that's what they return).

If the pattern holds, it will be down again tomorrow morning (but it's the weekend, so perhaps not). I'll be watching for it too though.

doomspork commented 2 days ago

Sounds good @brainlid! Thanks for tackling this one 🙏 I'll let you know what I see 🫡