foundation-model-stack / fms-hf-tuning

🚀 Collection of tuning recipes with HuggingFace SFTTrainer and PyTorch FSDP.
Apache License 2.0
28 stars 48 forks source link

fix: remove fire.fire(main), replace with main() #323

Closed willmj closed 2 months ago

willmj commented 2 months ago

Description of the change

Remove fire.Fire(main) from if __name__ == "__main__":

fire.Fire caused a problem with the following CLI command:

python tuning/sft_trainer.py --model_name_or_path $MODEL_PATH --data_path $DATA_PATH --output_dir $OUTPUT_PATH --num_train_epochs 80 --per_device_train_batch_size 4 --per_device_eval_batch_size 4 --gradient_accumulation_steps 4 --save_strategy "epoch" --learning_rate 1e-4 --weight_decay 0. --warmup_ratio 0.03 --lr_scheduler_type "cosine" --logging_steps 1 --include_tokens_per_second --packing False --response_template " Label:" --dataset_text_field "output" --use_flash_attn True --tokenizer_name_or_path $MODEL_PATH --torch_dtype bfloat16 --peft_method "lora" --logging_strategy "epoch" --r 16 --lora_dropout 0.05 --lora_alpha 32 --target_modules "c_attn" "c_proj"

Because it could not handle multiple target_modules Just using main() solves this issue

Related issue number

47

How to verify the PR

Run

python tuning/sft_trainer.py --model_name_or_path $MODEL_PATH --data_path $DATA_PATH --output_dir $OUTPUT_PATH --num_train_epochs 80 --per_device_train_batch_size 4 --per_device_eval_batch_size 4 --gradient_accumulation_steps 4 --save_strategy "epoch" --learning_rate 1e-4 --weight_decay 0. --warmup_ratio 0.03 --lr_scheduler_type "cosine" --logging_steps 1 --include_tokens_per_second --packing False --response_template " Label:" --dataset_text_field "output" --use_flash_attn True --tokenizer_name_or_path $MODEL_PATH --torch_dtype bfloat16 --peft_method "lora" --logging_strategy "epoch" --r 16 --lora_dropout 0.05 --lora_alpha 32 --target_modules "c_attn" "c_proj"

With fire.Fire(main) then with main()

Was the PR tested

HarikrishnanBalagopal commented 2 months ago

Related https://github.com/foundation-model-stack/fms-hf-tuning/pull/303

willmj commented 2 months ago

324