search

h2oai / h2ogpt

Private chat with local GPT with document, images, video, etc. 100% private, Apache 2.0. Supports oLLaMa, Mixtral, llama.cpp, and more. Demo: https://gpt.h2o.ai/ https://gpt-docs.h2o.ai/
http://h2o.ai
Apache License 2.0
11.39k stars 1.25k forks source link

Can you add langsmith/wandb for tracing and ragas for evaluation metrics? #1529

Open vitalyshalumov opened 7 months ago

pseudotensor commented 6 months ago

I have some code in WIP for verifiers that include such things, but not done. RAGAS is ok, but it's a bit loose compared to specific checking of actual specific faqs like done here: https://github.com/h2oai/enterprise-h2ogpte/tree/main/rag_benchmark

vitalyshalumov commented 4 months ago

Thank you, My problem is not on a test set - it is on a per query metric: I want to let the user know the quality of the answer by showing him the metrics:

[Faithfulness] [Answer relevancy] [Context recall] [Context precision] [Context relevancy] [Context entity recall]

https://docs.ragas.io/en/stable/concepts/metrics/index.html