Closed synio-wesley closed 1 month ago
Hello, I open a post about that . https://github.com/theodo-group/LLPhant/issues/173 but I decided to close. After conversation with some people, this approach is not relevant and we can loose some information about the context. After we can take in consideration the cost of llm will decrease. Look what s happen with gpt4-o mini. In this case not sure it's really important to include this kind of feature. That's my opinion.
@ClicShopping I'm not talking about base64 compression, but rather optimizing the prompt somehow by re-writing it (for example using an LLM as well). And I'm also more interested in re-ranking, which has proven more successful.
In any case, I think this is a valid feature request to want to hook a custom transformer right after documents have been retrieved, so we can re-rank them, or do anything else with them to optimize it, which can be dependent on your specific requirements.
ok understood
Hey @synio-wesley , In langchain we do a custom Chain for that. SO we could create another QA class with what you want to do and keep this one straightforward. What do you think?
@MaximeThoonsen I'm not sure if creating a QA class could simplify things. I think that we should keep both updated when we'll change one of them.
I think that @synio-wesley proposal of adding a RetrievedDocumentsTransformer
seems fair enough. I'll try to propose an implementation in the next few days
I do have some code that I am using myself right now. I could maybe share it as an example but the code needs some tweaking and tests before it is ready.. I'm on holiday right now for 2 weeks so will have to see when I have some time.
merged. All good for you @synio-wesley ?
What I want to achieve basically is re-ranking and prompt compression, before adding the retrieved docs to the context.
I read that it could drastically improve RAG performance. I think right now the
QuestionAnswering
class does not let us hook into modifying the list of retrieved docs.To at least support a third party to do this, we could maybe add something similar to the
QueryTransformer
with the defaultIdentityTransformer
which can be passed when constructing aQuestionAnswering
object, but instead it would be something likeRetrievedDocumentsTransformer
with a default (ornull
) transformer that does nothing.Because I think this is not supported yet without modifying LLPhant?
Then later we can also add some
RetrievedDocumentsTransformer
classes like one for re-ranking and one for prompt compression. And then we can also create a multi-transformer that allows us to define multiple other transformers, so we can combine multiple transformers and run them one by one.