How can I use the HHEM model to evaluate my LLM after finetuning?

vectara / hallucination-leaderboard

Leaderboard Comparing LLM Performance at Producing Hallucinations when Summarizing Short Documents

https://vectara.com

Apache License 2.0

1.25k stars 50 forks source link

How can I use the HHEM model to evaluate my LLM after finetuning? #50

Open zjq0455 opened 4 months ago

zjq0455 commented 4 months ago

Thank you for your contributions!

I was wondering whether it is possible or how can I use the HHEM model to evaluate a LLM after finetuning using our specific dataset?

forrestbao commented 4 months ago

hi @zjq0455 thanks for your interest in HHEM. Yes, of course. Just note that HHEM's context window is 512 tokens. We are working on a more powerful open source version to remove this restrictions.