Feature request: LlaVa and advanced RAG integration

[bw-Deejee]

Hey. I love the project. However one thing that is missing here, as well as in pretty much all other open RAG projects are advanced features which would lift RAG to a new level:

• LLaVA Many documents contain a mixture of content types, including text, tables, and images. Current level of RAG often fails with tables, diagramms and especially images. Therefore, missing out on a lot of valueable information in PDFs and Powerpoint slides. LLaVA is the open source attempt to offer something like GPT-4V(ision). Project page, Langchain RAG implementation cookbook

• Advanced RAG enhancements:

  • Self Querying Retrieval
  • Parent Document Retriever
  • Hybird Search
  • Contextual Compressors & Filters
  • HyDE
  • ... Sam Witteveen has great ressources about each of those.

[pseudotensor]

Already present:

• HYDE: Can activate on CLI or UI.
• Self Querying Retrieval: Already present with SEARCH agent, but as with all langchain agent stuff it's fragile even after many of my own improvements, esp. for open source.

Planned:

• LLaVa: https://github.com/h2oai/h2ogpt/issues/774
• Hybrid: Requires multiple retrievers setup differently than h2ogpt is currently

Not planned:

• Parent Document Retriever: Not particularly useful as chunking-expansion strategy, not main issue in RAG
• Contextual Compressors & Filters: Not particularly useful as filtering strategy, not main issue in RAG

I watch most of his videos, and some really good ideas or demonstrations, but some of those approaches are not helping with the main problems in RAG that memGPT etc. solve

[bw-Deejee]

Thank you