EinStack / glide

🐦 A open blazing-fast simple model gateway for rapid development of production GenAI apps
https://docs.einstack.ai/glide/
Apache License 2.0
113 stars 18 forks source link

🐛 [Streaming Chat] Improve error handling on bad chat request messages #272

Open roma-glushko opened 2 weeks ago

roma-glushko commented 2 weeks ago

A Glide's streaming channel gets closed on chat requests that doesn't pass validation. The issue is in the way we handle errors right now:

https://github.com/EinStack/glide/blob/bca8c8f2d2f19986ce2d064743e2147e58f466d0/pkg/api/http/handlers.go#L139-L148

For example, you can open a streaming chat connection (via Postman, for example):

{{base_url}}/language/{{routerID}}/chatStream

Then, send it just an empty message like:

{

}

You would get an error like:

Screenshot 2024-06-13 at 22 40 16

Finally, if you try to send a valid message now:

"message": {
    "role": "user",
    "content": "Write an essay about ecommerce in a modern era"
  }

It would not really work.