ORPO

Updates (24.03.25)

• [X] Sample script for ORPOTrainer in 🤗TRL is added to trl/test_orpo_trainer_demo.py
• [X] New model, 🤗kaist-ai/mistral-orpo-capybara-7k, is added to 🤗ORPO Collection
• [X] Now you can try ORPO in 🤗TRL, Axolotl and LLaMA-Factory🔥
• [X] We are making general guideline for training LLMs with ORPO, stay tuned🔥
• [X] Mistral-ORPO-β achieved a 14.7% in the length-controlled (LC) win rate on official AlpacaEval Leaderboard🔥

This is the official repository for ORPO: Monolithic Preference Optimization without Reference Model. The detailed results in the paper can be found in:

• Open LLM Leaderboard
• AlpacaEval
• MT-Bench
• IFEval

Model Checkpoints

Our models trained with ORPO can be found in:

• [X] Mistral-ORPO-Capybara-7k: 🤗 kaist-ai/mistral-orpo-capybara-7k
• [X] Mistral-ORPO-⍺: 🤗 kaist-ai/mistral-orpo-alpha
• [X] Mistral-ORPO-β: 🤗 kaist-ai/mistral-orpo-beta

And the corresponding logs for the average log probabilities of chosen/rejected responses during training are reported in:

• [X] Mistral-ORPO-Capybara-7k: TBU
• [X] Mistral-ORPO-⍺: Wandb Report for Mistral-ORPO-⍺
• [X] Mistral-ORPO-β: Wandb Report for Mistral-ORPO-β

AlpacaEval

MT-Bench

IFEval

IFEval scores are measured with EleutherAI/lm-evaluation-harness by applying the chat template. The scores for Llama-2-Chat (70B), Zephyr-β (7B), and Mixtral-8X7B-Instruct-v0.1 are originally reported in this tweet.