search

huggingface / alignment-handbook

Robust recipes to align language models with human and AI preferences
https://huggingface.co/HuggingFaceH4
Apache License 2.0
4.53k stars 393 forks source link

Having troubles changing save_steps parameter for a resumed job. #195

Open antonpolishko opened 1 month ago

antonpolishko commented 1 month ago

I'm having next issue. Let say I'm starting a job 

ACCELERATE_LOG_LEVEL=info accelerate launch --config_file recipes/accelerate_configs/deepspeed_zero3.yaml scripts/run_sft.py recipes/zephyr-7b-beta/sft/config_full.yaml

In config_full.yaml I would have save_steps: 1000. At some point I would realize that 1000 is too frequent of a step to save, so I stop the job, edit config_full.yaml to have save_steps: 10000 and restart the job. The resume from checkpoint goes as planned, however I would still have checkpoints saved every 1000 steps (original parameter). What do I do wrong?

lewtun commented 1 month ago

Hmm this is a bit strange. If you look at the logs of your second run, do you see save_steps is set to 1000 or 10_000?