search

Quansight / ragna

RAG orchestration framework ⛵️
https://ragna.chat
BSD 3-Clause "New" or "Revised" License
177 stars 22 forks source link

Ability to pass error messages from API to UI #409

Open nenb opened 4 months ago

nenb commented 4 months ago

Feature description

The ability to pass any error message that is raised in the API to the UI. Currently this is not possible as StreamingResponse is used which results in peer closed connection without sending complete message body being raised in the UI for all API errors.

Value and/or benefit

Better user experience eg ability to determine more easily whether an issue is UI or API-related.

Anything else?

No response

pmeier commented 4 months ago

All for it. The message (or chunks in case of streaming) that we are returning always have a role attached to them:

https://github.com/Quansight/ragna/blob/3cef0f7da1f2ed90e5d0618bcad82f824d00dc5a/ragna/deploy/_api/schemas.py#L54-L57

Currently MessageRole.ASSISTANT is hardcoded for this

https://github.com/Quansight/ragna/blob/3cef0f7da1f2ed90e5d0618bcad82f824d00dc5a/ragna/core/_rag.py#L223-L227

We could switch the role to MessageRole.SYSTEM in case we hit an error. With that we could send the error message either instead of the answer or as last chunk when streaming. If we do that, we could make it protocol, i.e. document, that if one receives a system message when one from the assistant is expected, this indicates an error and it should be displayed as such.

Thoughts?

nenb commented 4 months ago

If we do that, we could make it protocol, i.e. document, that if one receives a system message when one from the assistant is expected, this indicates an error and it should be displayed as such.

It seems a little bit complicated/non-standard, but I would be okay with it.

In practice, whenever I hit an error, it always comes from trying to generate the first chunk. If the first chunk is successful, then I don't remember hitting an error (although it's of course possible).

This was the motivation for #410 - test the first chunk, and if there is an error then return a standard 500 response with details. I suspect that this will catch the vast majority of errors, and is what seemed to be recommended by others when I did a quick web-search.

It might be useful for others who only consume the web API (ie don't use our UI) to have the ability to error before starting streaming.

Happy for you to make a decision on how to proceed.