rmusser01 / tldw

Too Long, Didn't Watch(TL/DW): Your Personal Research Multi-Tool - Open Source NotebookLM
Apache License 2.0
44 stars 2 forks source link

Improvement: RAG Search #48

Open rmusser01 opened 1 month ago

rmusser01 commented 1 month ago

lol.

Incase you're curious we use named entity recognition models to extract key words / phrases. Then use bm25 + vector search to identify the top results!

Links: https://www.elastic.co/search-labs/blog/semantic-reranking-with-retrievers https://towardsdatascience.com/text-to-knowledge-graph-made-easy-with-graph-maker-f3f890c0dbe8?gi=f557597a37c9 https://towardsdatascience.com/advanced-retrieval-augmented-generation-from-theory-to-llamaindex-implementation-4de1464a9930 https://techcommunity.microsoft.com/t5/microsoft-developer-community/doing-rag-vector-search-is-not-enough/ba-p/4161073 https://blog.gopenai.com/advanced-retrieval-augmented-generation-rag-techniques-5abad385ac66?gi=44b899a78d28 https://medium.com/@LakshmiNarayana_U/advanced-rag-techniques-in-ai-retrieval-a-deep-dive-into-the-chroma-course-d8b06118cde3 https://towardsdatascience.com/rag-vs-finetuning-which-is-the-best-tool-to-boost-your-llm-application-94654b1eaba7?gi=270e315b9b3f https://medium.com/the-ai-forum/rag-on-complex-pdf-using-llamaparse-langchain-and-groq-5b132bd1f9f3 https://www.cazton.com/blogs/technical/advanced-rag-techniques https://github.com/rahulnyk/knowledge_graph https://medium.com/@krtarunsingh/advanced-rag-techniques-unlocking-the-next-level-040c205b95bc https://towardsdatascience.com/how-to-convert-any-text-into-a-graph-of-concepts-110844f22a1a https://pub.towardsai.net/advanced-rag-techniques-an-illustrated-overview-04d193d8fec6?gi=6a258d737c25 https://arxiv.org/html/2401.18059v1 https://medium.com/@djangoist/how-to-create-accurate-llm-responses-on-large-code-repositories-presenting-cgrag-a-new-feature-of-e77c0ffe432d https://arxiv.org/pdf/2005.11401 https://arxiv.org/html/2403.19889v1 https://betterprogramming.pub/building-a-reddit-thread-summarizer-with-chatgpt-api-5b0dcd50b88e?gi=157029dc5472 https://jxnl.co/writing/2024/05/22/systematically-improving-your-rag/ https://winder.ai/llm-architecture-rag-implementation-design-patterns/ https://github.com/wordsmith-ai/hello-wordsmith https://www.llamaindex.ai/ https://gist.github.com/Donavan/62e238aa0a40ca88191255a070e356a2 https://arxiv.org/html/2404.07220v1 https://about.xethub.com/blog/you-dont-need-a-vector-database https://arxiv.org/abs/2312.10997 https://techcommunity.microsoft.com/t5/ai-azure-ai-services-blog/azure-ai-search-outperforming-vector-search-with-hybrid/ba-p/3929167 https://github.com/xetdata/RagIRBench https://medium.com/decodingml/the-4-advanced-rag-algorithms-you-must-know-to-implement-5d0c7f1199d2 https://github.com/OSU-NLP-Group/HippoRAG https://github.com/Azure-Samples/rag-postgres-openai-python https://github.com/Azure-Samples/rag-postgres-openai-python/blob/e30ea96ca11ca6578ca38d3428594bd98d704900/src/fastapi_app/postgres_searcher.py#L38 https://news.ycombinator.com/item?id=38759877 https://medium.com/@cdg2718/why-your-rag-doesnt-work-9755726dd1e9 https://www.timescale.com/blog/pgvector-vs-pinecone/ https://www.timescale.com/blog/how-we-made-postgresql-as-fast-as-pinecone-for-vector-data/ https://stephencollins.tech/posts/how-to-use-sqLite-to-store-and-query-vector-embeddings

BEAM search https://github.com/infiniflow/infinity https://docs.haystack.deepset.ai/docs/inmemorybm25retriever https://www.width.ai/post/what-is-beam-search

Existing implementations: https://github.com/abgulati/LARS/tree/v1.1 https://github.com/infiniflow/ragflow

https://huggingface.co/papers/2406.12824 https://arxiv.org/abs/2402.16406

rmusser01 commented 1 week ago

https://github.com/whitead/paper-qa https://blog.dagworks.io/p/building-a-conversational-graphdb https://github.com/pymupdf/RAG https://medium.com/microsoftazure/introducing-graphrag-with-langchain-and-neo4j-90446df17c1e https://medium.com/@Stan_DS/build-a-100-free-hallucination-free-secure-rag-chatbot-using-reranker-and-gpt4o-96c2eea24f95 https://blog.demir.io/advanced-rag-implementing-advanced-techniques-to-enhance-retrieval-augmented-generation-systems-0e07301e46f4?gi=9e3e04c616da https://superlinked.com/vectorhub/articles/advanced-retrieval-augmented-generation https://div.beehiiv.com/p/advanced-rag-series-retrieval https://towardsdatascience.com/how-to-reduce-embedding-size-and-increase-rag-retrieval-speed-7f903d3cecf7?gi=b5bcdb8dfecf https://generativeai.pub/advanced-rag-retrieval-strategy-query-rewriting-a1dd61815ff0 https://medium.com/intel-tech/tabular-data-rag-llms-improve-results-through-data-table-prompting-bcb42678914b https://generativeai.pub/advanced-rag-retrieval-strategies-hybrid-retrieval-997d39659720?gi=09c4b7d4c4c6 https://ai.gopubby.com/advanced-rag-retrieval-strategy-embedded-tables-fdb3e44003a5?gi=59a21f193fb7 https://arxiv.org/abs/2401.08406 https://github.com/MichaelMcCulloch/WikiDex https://huggingface.co/google/xtr-base-multilingual https://arxiv.org/abs/2304.01982 https://r2r-docs.sciphi.ai/introduction

rmusser01 commented 2 days ago

Eval Rag

https://docs.ragas.io/en/latest/index.html