Blog: Preference Tuning LLMs with Direct Preference Optimization Methods

[konabuta]

source: [Preference Tuning LLMs with Direct Preference Optimization Methods](https://huggingface.co/blog/pref-tuning)_

[konabuta]

Though RLHF is a major way to align LLMs, there are several algorithmes without reinforcement learning techniques. This blog introduces the following algorithms:

• Direct Preference Optimization (DPO)
• Indeity Preference Optimization (IPO)
• Kahneman-Tversky Optimization (KTO)

[konabuta]

DPO

• optimize a simple loss function of alignment function
• dataset of preference: ${(x, y_w, y_l)}$
  • $x$: prompt
  • $y_w$: preferred response
  • $y_l$: dispreferred response

Challenge in DPO

• Robustness:
  • easy to overfit
  • Identity Preference Optimization (IPO) - adding regularisation term to the DPO loss
• Dispensing with paried preference data altogher
  • Kahneman-Tversky Optimization (KTO) - define loss function for individual exampls with "good" or "bad" labels.