Closed nivibilla closed 3 weeks ago
PR : #1036
Thank you for sharing this! I need to dig into the paper a bit more, could you highlight the differences between the SimPO recipes and the DPO recipes?
Simpo loss doesn't seem to have a need for the reference log probs(similar to ORPO) and introduces a new hyperparameter gamma. Which I guess also means the model being aligned doesn't need to be SFT on the training set before the RL part.
This is a really good summary of it https://x.com/_philschmid/status/1794627683575316548?t=m-qwqK8Mm1tlIO-pyMpIPw&s=19
If the only difference between SimPO and DPO is the loss, maybe we could define the recipes.
From what I understand from their repo the loss seems to be the only difference but I haven't verified by training identical models yet. But yeah, not sure if there needs to be two recipes for something so similar. I was thinking to refactor it to "preference optimisation" recipe and have dpo and simpo as loss options.
Or alternatively, not touch anything and simply have some internal logic to adjust the loss given the loss in the config yaml.
https://arxiv.org/html/2405.14734v1
Claims to have better performance than all previous offline rl training methods.