Ongoing Improvement: Improve Accuracy of Results / Reduce Confabulation Rate

[rmusser01]

Title.

https://blog.streamlit.io/ai21_grounded_multi_doc_q-a/ https://arxiv.org/html/2404.09129v1 https://arxiv.org/pdf/2406.02543 https://huggingface.co/papers/2406.02543 https://huggingface.co/spaces/hallucinations-leaderboard/leaderboard https://github.com/hamelsmu/ft-drift https://hamel.dev/blog/posts/evals/ https://arxiv.org/abs/2009.01325 https://thetechoasis.beehiiv.com/p/eliminating-hallucinations-robots-imitate-us https://arxiv.org/abs/2407.16604 https://eugeneyan.com/writing/finetuning/ https://eugeneyan.com/writing/evals/ https://arxiv.org/pdf/2404.12272 https://arize.com/blog/breaking-down-evalgen-who-validates-the-validators/ https://eugeneyan.com/writing/finetuning/ https://www.nature.com/articles/s41586-024-07421-0 https://arxiv.org/abs/2406.15927 https://research.google/blog/halva-hallucination-attenuated-language-and-vision-assistant/ https://arxiv.org/abs/2407.08488v1 https://osu-nlp-group.github.io/AttributionBench/ https://arxiv.org/abs/2407.08488 https://arxiv.org/pdf/2407.03651 https://www.patronus.ai/blog/lynx-state-of-the-art-open-source-hallucination-detection-model https://arxiv.org/abs/2408.07852 https://www.turingpost.com/p/10-ways-to-process-long-context

[rmusser01]

Seems like the best thing currently short of fine-tuning/pre-training is asking a model specifically trained to identify confabulations.

[rmusser01]

Finetuning:

• https://eugeneyan.com/writing/finetuning/

Evals:

• https://github.com/yanhong-lbh/LLM-SelfReflection-Eval
• https://eugeneyan.com/writing/evals/

LLM As Judge:

• https://arxiv.org/pdf/2404.12272
• https://arize.com/blog/breaking-down-evalgen-who-validates-the-validators/

Detecting Hallucinations using Semantic Entropy:

• https://www.nature.com/articles/s41586-024-07421-0
• https://github.com/jlko/semantic_uncertainty
• https://github.com/jlko/long_hallucinations
• https://arxiv.org/abs/2406.15927

Lynx/patronus

• https://www.patronus.ai/blog/lynx-state-of-the-art-open-source-hallucination-detection-model
• https://huggingface.co/PatronusAI/Llama-3-Patronus-Lynx-8B-Instruct-Q4_K_M-GGUF
• https://github.com/NVIDIA/NeMo-Guardrails/blob/develop/docs/user_guides/community/patronus-lynx.md
• https://github.com/NVIDIA/NeMo-Guardrails/blob/develop/examples/configs/patronusai/prompts.yml
• https://arxiv.org/html/2407.08488v1

[rmusser01]

https://github.com/EdinburghNLP/awesome-hallucination-detection

[rmusser01]

https://arxiv.org/abs/2407.16557

[rmusser01]

Reasoning https://arxiv.org/abs/2407.19813 https://arxiv.org/abs/2408.06195 https://arxiv.org/abs/2407.13481 Output length impacts on reasoning https://arxiv.org/abs/2407.19825

[rmusser01]

https://arxiv.org/abs/2406.10279 https://huggingface.co/spaces/vectara/Hallucination-evaluation-leaderboard

[rmusser01]

https://arxiv.org/pdf/2409.18475 https://cleanlab.ai/blog/trustworthy-language-model/

[rmusser01]

https://arxiv.org/abs/2410.02707

[rmusser01]

https://www.lycee.ai/blog/rag-ragallucinations-and-how-to-fight-them

[rmusser01]

https://github.com/lechmazur/confabulations/

[rmusser01]

https://llm-editing.github.io/

[rmusser01]

https://arxiv.org/abs/2410.22071