Tiendil
commented
2 months ago Required features:
- Universal estimation of the amount of tokens before making the request.
- Good enough splitting requests if they exceed the context size.
- Tracking API keys status.
- Using different API keys for different requests.
- Using different LLM providers/models for different requests.
- async/await support.
- Protection from mistakes in global settings or, better, turning off global settings.
After examining top libs (LangChain, LLamaIndex, haystack):
- Code quality of
LLamaIndexandhaystackis under question (significant bugs found after short investigation, I'll link them later).LangChainis better, but I can not tell if it is good enough. - There is a lack of functionality for most of the required features in all of the libs.
- When the functionality is present, it does not look like it is well implemented.
- All libs provide a lot of not required for our case functionality.
Therefore, we'll not use unified libs.
Instead, we'll implement our simplified interface for interaction with LLMs in a way specific to our project.
The first implementation will support OpenAI and Gemini providers. Later, integration with OLLame will be added (by request or when needed).
To become more agile in using different LLM providers and open-source solutions, we should migrate from using direct OpenAI clients to a unified intermediate library like LangChain.
Also, there are requests to allow alternatives to OpenAI; for example, see gh-168.
We could provide a way to construct LLM agents via configs by splitting them into three parts: text preprocessor, LLM request config, and tags parser.
We may need different keys for different processors to track usage (even for a single API provider).