search

quarkiverse / quarkus-langchain4j

Quarkus Langchain4j extension
https://docs.quarkiverse.io/quarkus-langchain4j/dev/index.html
Apache License 2.0
133 stars 79 forks source link

Allow the injection of TokenCountEstimator #697

Closed andreadimaio closed 3 months ago

andreadimaio commented 3 months ago

Today it is not possible to inject the TokenCountEstimator, this could be useful to estimate the number of tokens.

geoand commented 3 months ago

Where would one inject that?

andreadimaio commented 3 months ago

The idea is to have the same configuration of the ChatLanguageModel, in this case the developers can inject the interface where they prefer.

For example, suppose I have two AIServices configured with watsonx, one with llama3 and the other with mixstral. The developer could inject the TokenCountEstimator for both models and choose which to use based on the token counts.

This is an idea of what I have in mind LINK.

geoand commented 3 months ago

We can certainly do that and it does makes sense!

andreadimaio commented 3 months ago

Ok, I'm going to share what I wrote (I have more tests to add) for the end of the day.

geoand commented 3 months ago

Thanks!