QuivrHQ / quivr

Opiniated RAG for integrating GenAI in your apps 🧠 Focus on your product rather than the RAG. Easy integration in existing products with customisation! Any LLM: GPT4, Groq, Llama. Any Vectorstore: PGVector, Faiss. Any Files. Anyway you want.
https://core.quivr.com
Other
36.52k stars 3.57k forks source link

Enable use of embeddings from Vision Language models #3452

Open jacopo-chevallard opened 1 day ago

jacopo-chevallard commented 1 day ago

Currently, we use text embeddings. This is fine for textual documents, while it present obvious drawbacks for documents containing non-textual content (images, graphs, schemes, …).

An alternative, is to use Visual Language models such as ColPali (see also https://huggingface.co/blog/manu/colpali, https://danielvanstrien.xyz/posts/post-with-code/colpali-qdrant/2024-10-02_using_colpali_with_qdrant.html, https://blog.vespa.ai/retrieval-with-vision-language-models-colpali/, https://blog.vespa.ai/scaling-colpali-to-billions/)

image.png

linear[bot] commented 1 day ago

CORE-280 Enable use of embeddigns Vision Language models