search

vectara / hallucination-leaderboard

Leaderboard Comparing LLM Performance at Producing Hallucinations when Summarizing Short Documents
https://vectara.com
Apache License 2.0
1.25k stars 50 forks source link

Generation parameters #9

Closed qmdnls closed 1 year ago

qmdnls commented 1 year ago

What parameters did you use to generate the summarizations on the various API services? In particular I would be interested in the generation temperature used since it seems to me that this quite important for a fair evaluation.

simonhughes22 commented 1 year ago

Sorry i thought we had mentioned this but we didn't. We used a temperature of 0 thoughout. I just updated the readme. The only other parameter of note is the model name which i mentioned in the readme under API details. You can also see the prompt we used for every model, which was passed as the system prompt where possible.