Bradley-Terry model removes lm head while saving

RLHFlow / RLHF-Reward-Modeling

Recipes to train reward model for RLHF.

https://rlhflow.github.io/

Apache License 2.0

612 stars 52 forks source link

Bradley-Terry model removes lm head while saving #22

Open Arnav0400 opened 1 month ago

Arnav0400 commented 1 month ago

Hello and thanks for your work!

While running bradley-terry-rm/llama3_rm.py the final saved model does not have a lm head as the script is using a AutoModelForSequenceClassification model and not CausalLM. Because of this the lm head is initialized from scratch while loading the saved model. Is this correct or do you manually add the lm head weights?

WeiXiongUST commented 1 month ago

Maybe you can test the saved model by the useful_code/eval_reward_bench_bt.py to see whether the model is saved correctly.