quarkiverse / quarkus-langchain4j

Quarkus Langchain4j extension
https://docs.quarkiverse.io/quarkus-langchain4j/dev/index.html
Apache License 2.0
131 stars 79 forks source link

Support custom `StreamingChatLanguageModel` suppliers #842

Closed dastrobu closed 2 weeks ago

dastrobu commented 3 weeks ago

RegisterAiService currently allows adding a chatLanguageModelSupplier. Unfortunately, I could not find a straightforward way of implementing a custom Supplier<StreamingChatLanguageModel>.

I'd suggest adding an additional property streamingChatLanguageModelSupplier to RegisterAiService, so we can use custom models.

I could not find a viable alternative for providing my own implementation, except for writing my own extension (like in quarkus-langchain4j-azure-openai), which seems to be overkill for that task.

I also think quarkus-langchain4j would profit from that feature, as it would allow to add more test cases with StreamingChatLanguageModels, when a custom model can be implemented like in:

public class StreamingTest {

    static class MyModelSupplier implements Supplier<StreamingChatLanguageModel> {
        @Override
        public StreamingChatLanguageModel get() {
            return (messages, handler) -> {
                handler.onComplete(new Response<>(new AiMessage("42")));
            };
        }
    }

    @RegisterAiService(streamingChatLanguageModelSupplier = MyModelSupplier.class)
    interface MyService {
        @UserMessage("What is the Answer to the Ultimate Question of Life, the Universe, and Everything?")
        Multi<String> chat();
    }

// ...
}
geoand commented 3 weeks ago

Good point!