reward/chosen is decreasing

[zhangguoxin1]

Hi! I am fine-tuning LLaMA3 on the hh-rlhf dataset using SimPo and noticed that the reward/chosen reward is decreasing. Is this reasonable? `# SimPOTrainer arguments

bf16: true beta: 2.5 gamma: 1.4 per_device_train_batch_size: 2 per_device_eval_batch_size: 4 do_eval: true eval_strategy: steps eval_steps: 500 gradient_accumulation_steps: 8 gradient_checkpointing: true gradient_checkpointing_kwargs: use_reentrant: False learning_rate: 5.0e-5 num_train_epochs: 1 log_level: info logging_steps: 5 lr_scheduler_type: cosine max_length: 2048 max_prompt_length: 1800 optim: adamw_torch output_dir: outputs/llama-3-8b-instruct-simpo-hh run_name: llama-3-8b-instruct-simpo-hh force_use_ref_model: True push_to_hub: false save_strategy: "steps" save_steps: 500 remove_unused_columns: False save_total_limit: 20 seed: 42 warmup_ratio: 0.1 `

[zhangguoxin1]

I expected the reward/chosen to increase, but since the goal of SimPo is to maximize the difference between reward/chosen and reward/rejected, it is acceptable for reward/chosen to decrease to a certain extent. However, the extent of the decrease in reward/chosen seems a bit large compared to reward/chosen - reward/rejected.

[yumeng5]

Hi,

Yes, this is reasonable. The reward margin should increase but the reward on chosen responses may slightly decrease (and the reward on rejected decreases more rapidly). In general, we don't want the reward on chosen to decrease too much (as that implies the likelihood of chosen responses is decreasing), and you may use a larger beta or a lower learning rate to mitigate the decrease of reward on chosen responses.

Best, Yu

[zhangguoxin1]

get it!

Thanks for the quick reply.

[zhangguoxin1]

Hi, I used Simpo in my task with qwen2_7B (there are approximately 40,000 data entries), but the model generated repeated sentences and pre-trained data. The parameters are as follows:

pref_beta: 2.5
simpo_gamma: 1.0
learning_rate: 1.0e-6
num_train_epochs: 3.0

and I'm try use a larger beta=8.0

[xiamengzhou]

@zhangguoxin1 I think you should be using Qwen2-7B-Instruct rather than Qwen2-7B if you only running PO? Also I'd suggest that you use online data rather offline data that is generated by other models.

[yumeng5]

Hi @zhangguoxin1

In addition to the suggestions by Mengzhou, you may try the following as well:

• decrease the learning rate (we usually start learning rate search around 5e-7)
• reduce the number of training epochs (we generally train the model for only one epoch)

Best, Yu