pseudotensor
commented
6 months ago If you can give me an example repro of your case using some set of public documents, where it shows sources are at bottom of list, I can take a look.
The sources are scored based upon standard semantic similarity search and the distance metric 'l2'. The LLM then decides what is most relevant from that. As you might imagine, that's why LLMs are so useful, is because they actually find the relevant info, when semantic similarity cannot fully. That's why we fill the context with various info the let the LLM decide.
There's no easy way to extract which text/sources the LLM used, that's an area of active research. In some cases a simple ngram check may help, but it also might hurt for non-keyword related cases. There may be a way to back-prop through the model to see why it decided, but nobody has figured that out for auto-regressive case, only classification.
If the user has to verify the answer, best is to present the sources more cleanly like in our Enterprise h2oGPT https://h2o.ai/resources/video/enterprise-h2ogpt/ that shows full page references with highlighted text regions. But that doesn't really solve your problem when there are numerous documents.
rohitnanda1443
Hello,
I've successfully implemented the LLM Model on a custom dataset within a controlled network, leveraging over 1000 PDF documents. The responses are comprehensive and informative. However, I've noticed a significant issue in the way sources are presented.
When a query is raised, the model accurately retrieves responses from relevant documents. However, the sources listed at the end do not seem to align with the provided response. The document mentioned in the response is consistently placed at the very bottom of the list of sources. Ideally, the relevant document should be listed first, enabling users to access the specific source document immediately.
This discrepancy undermines the primary goal of using the LLM. The intent is to present query details along with the corresponding source at the top of the list for quick access. Currently, users have to sift through the entire list of sources to find the document related to the response, which defeats the purpose of streamlining information retrieval from a pool of 1000 PDFs.
I am seeking suggestions on how to address this issue as it poses a significant obstacle in the effective deployment of the solution. Your assistance in resolving this matter is crucial, and I appreciate any insights or recommendations you can provide.
Thank you