search

cequence-io / openai-scala-client

Scala client for OpenAI API
MIT License
191 stars 20 forks source link

Implement model router for "Streamed" service #67

Closed yahor-filipchyk closed 5 months ago

yahor-filipchyk commented 5 months ago

Currently the OpenAIServiceAdapters#chatCompletionRouter only works with instances of type OpenAIChatCompletionService. It would be nice also to have a router for OpenAIChatCompletionStreamedServiceExtra.

I am looking to add Anthropic to the list of supported models. Most of our user experiences require streaming. Since OpenAIAnthropicChatCompletionService already implements both OpenAIAnthropicChatCompletionService and OpenAIChatCompletionStreamedServiceExtra I imagine it wouldn't be hard to support model routing for streamed requests?

peterbanda commented 5 months ago

Hey @yahor-filipchyk , it's definitely possible! Check out these two examples:

I agree it would be better to incorporate the routing of streaming functions to adapters abstraction. It's on my list :)

yahor-filipchyk commented 5 months ago

@peterbanda perfect! Didn't realize OpenAIChatCompletionStreamedServiceRouter was already available!