DPO alignment doesn't work on Lora models as suggested

[Abe13]

You claim that "In practice, we find comparable performance for both full and LoRA fine-tuning, with the latter having the advantage of producing small adapter weights that are fast to upload and download from the Hugging Face Hub."

However, when I try the Lora model DPO-aligned LLM that you have trained, alignment-handbook/zephyr-7b-dpo-lora, I experience a total performance degradation. Here is an example of model output that seems confused:

Even the training loss indicates that the model has not learned much

Here is the training loss for the full model DPO alignment.

Would you please do a clarification? Is my observation different from what you have experienced?

Thanks

[lewtun]

Hello @Abe13 thanks for raising this issue! Yes, there's seems to be a discrepancy / regression that occurred during the porting of our internal codebase and we're currently working on tracking it down. See this issue for related discussion: https://github.com/huggingface/alignment-handbook/issues/45