search

infiniflow / ragflow

RAGFlow is an open-source RAG (Retrieval-Augmented Generation) engine based on deep document understanding.
https://ragflow.io
Apache License 2.0
18.18k stars 1.84k forks source link

[Bug]: Poor Performance of RAGFlow with Specialized Queries #2428

Open pure11011 opened 1 week ago

pure11011 commented 1 week ago

While RAGFlow performs reasonably well in general queries, it struggles with providing accurate and relevant responses when handling queries related to specific professional domains such as medicine. This issue affects the overall utility of the system for more specialized users who rely on accurate domain-specific information.

Steps to reproduce

Load a knowledge base with professional documents, such as medical literature.
Issue a complex query related to the domain (e.g., a specific medical diagnosis or legal principle).
Compare the generated responses to the expected or correct results.
KevinHuSh commented 1 week ago

Could you elaberate on a case?

Said-Apollo commented 1 week ago

While RAGFlow performs reasonably well in general queries, it struggles with providing accurate and relevant responses when handling queries related to specific professional domains such as medicine. This issue affects the overall utility of the system for more specialized users who rely on accurate domain-specific information.

Steps to reproduce

Load a knowledge base with professional documents, such as medical literature.
Issue a complex query related to the domain (e.g., a specific medical diagnosis or legal principle).
Compare the generated responses to the expected or correct results.

Experienced something similar (law&Compliance). However, we could improve accuracy by using better LLM's, using Reranker and also better prompts. Furthermore you could try out Raptor (in this Case, I'm not quite sure whether it really helps) or the new GraphRAG feature. Also, probably one of the best ways would be to use an embedding model for the medical domain. Embedding Models of OpenAI and others are typically good on general benchmarks, but not so for expert domains.