Fix EOS token bug on FastChat models (non DPO) - Githubissues

allenai / reward-bench

RewardBench: the first evaluation tool for reward models.

https://huggingface.co/spaces/allenai/reward-bench

Apache License 2.0

281 stars 28 forks source link

Fix EOS token bug on FastChat models (non DPO) #94

Closed natolambert closed 3 months ago

natolambert commented 3 months ago

Closes #90 Effected models to re-run (prev score --> new score):

openbmb/UltraRM-13b (69.53 --> 68.18)
OpenAssistant/oasst-rm-2.1-pythia-1.4b-epoch-2.5 (69.6 --> 69.49)
OpenAssistant/oasst-rm-2-pythia-6.9b-epoch-1 (62.05 --> 62.23)
OpenAssistant/reward-model-deberta-v3-large-v2 (57.49 --> 54.29)
weqweasdas/hh_rlhf_rm_open_llama_3b (54.92 --> 48.85)
berkeley-nest/Starling-RM-7B-alpha (71.42 --> 71.43)
PKU-Alignment/beaver-7b-v1.0-reward (47.62 --> 45.44)
PKU-Alignment/beaver-7b-v1.0-cost (58.25 --> 59.83)
IDEA-CCNL/Ziya-LLaMA-7B-Reward (63.43 --> 62.94)

These had a stronger effect on Tulu models, interestingly.