feat(rag): Auto-RAG

[gaocegege]

/kind feature

Ref https://arxiv.org/pdf/2404.01037.pdf

Auto-RAG: The idea of automatically optimizing RAG systems, akin to Auto-ML’s approach in traditional machine learning, presents a significant opportunity for future exploration. Currently, selecting the optimal configuration of RAG components — e.g., chunking strategies, window sizes, and parameters within rerankers — relies on manual experimentation and intuition. An automated system could systematically explore a vast space of RAG configurations and select the very best model (Markr.AI, 2024).

RAG requires some hyperparameters e.g. chunking strategies, and window sizes for sentence window retrieval. It should be done automatically.

---

Love this feature? Give it a 👍 We prioritize the features with the most 👍

[gaocegege]

Maybe we could add an example to showcase how to use Katib and LlamaIndex to AutoRAG.

Not sure if there is any new feature to be implemented.

[gaocegege]

Related: https://github.com/Marker-Inc-Korea/AutoRAG

[tariq-hasan]

Are you thinking of adding an example that uses the proposed tuning API for LLMs to demonstrate Auto-RAG?

[gaocegege]

@tariq-hasan It should work. But I do not have the bandwidth for it. I'm simply presenting the idea for consideration at this point.

[andreyvelich]

Thanks for creating this @gaocegege. Are there any differences to optimize these HPs for RAG (e.g. chunking strategies and window sizes) compare to our current optimization flow with Experiment -> Suggestion -> Trials? I guess, Trials can consume prompt and produce the metrics.

[gaocegege]

The workflow should be similar. I think. We could make a demo based on llama index to see if there is anything we miss.