[Connectors] Generalise handling of streaming responses

elastic / kibana

Your window into the Elastic Stack

https://www.elastic.co/products/kibana

Other

19.69k stars 8.24k forks source link

[Connectors] Generalise handling of streaming responses #169088

Open dgieselaar opened 1 year ago

dgieselaar commented 1 year ago

In https://github.com/elastic/kibana/pull/168440, we are implementing token count tracking for streaming responses for OpenAI inside of the framework. This is a small iteration but ideally this should be generalised. Per @mikecote:

We should add a capability for connector types to report extra data within their code (instead of the framework)
Streaming responses should generally be handled, not just for the gen-ai connector

elasticmachine commented 1 year ago

Pinging @elastic/response-ops (Team:ResponseOps)