DevoxxGenie is a plugin for IntelliJ IDEA that uses local LLM's (Ollama, LMStudio, GPT4All and Jan) and Cloud based LLMs to help review, test, explain your project code.
Different LLM providers use different tokenizers, which can lead to varying token counts for the same text. To address this, we'll need to modify our implementation to use the appropriate tokenizer based on the selected Cloud LLM
stephanj
commented
1 week ago
Different LLM providers use different tokenizers, which can lead to varying token counts for the same text. To address this, we'll need to modify our implementation to use the appropriate tokenizer based on the selected Cloud LLM