search

huggingface / alignment-handbook

Robust recipes to align language models with human and AI preferences
https://huggingface.co/HuggingFaceH4
Apache License 2.0
4.28k stars 367 forks source link

SFT lora ends with higher loss #72

Open Randl opened 7 months ago

Randl commented 7 months ago

I've run the training without changing any hyperparameter except for batch size and gradient accumulation steps to match the global batch size on two machines. The first run is exactly as in repo, gets eval loss 1.0667: https://wandb.ai/evgeniizh/huggingface/runs/pskgg48d The second one adds warmup (https://github.com/huggingface/alignment-handbook/pull/31 https://github.com/huggingface/alignment-handbook/pull/71) and uses TRL from master (which fixes https://github.com/huggingface/alignment-handbook/issues/61) and gets eval loss of 1.0927 https://wandb.ai/evgeniizh/huggingface/runs/9ez7kl7s

The official SFT model gets much lower loss of 0.99 https://huggingface.co/alignment-handbook/zephyr-7b-sft-lora

Randl commented 7 months ago

Possibly related to https://github.com/huggingface/alignment-handbook/issues/45