Closed situjunhao closed 3 months ago
llamafactory
llamafactory-cli train \ --stage sft \ --do_train True \ --model_name_or_path THUDM/glm-4-9b-chat \ --preprocessing_num_workers 16 \ --finetuning_type lora \ --template glm4 \ --rope_scaling dynamic \ --flash_attn fa2 \ --dataset_dir data \ --dataset alpaca_zh \ --cutoff_len 4096 \ --learning_rate 5e-05 \ --num_train_epochs 3.0 \ --max_samples 100000 \ --per_device_train_batch_size 2 \ --gradient_accumulation_steps 8 \ --lr_scheduler_type cosine \ --max_grad_norm 1.0 \ --logging_steps 5 \ --save_steps 100 \ --warmup_steps 0 \ --optim adamw_torch \ --packing False \ --report_to none \ --output_dir saves/GLM-4-9B-Chat/lora/train_2024-08-07-11-28-01 \ --bf16 True \ --plot_loss True \ --ddp_timeout 180000000 \ --include_num_input_tokens_seen True \ --lora_rank 8 \ --lora_alpha 16 \ --lora_dropout 0 \ --use_dora True \ --lora_target all
No response
+1
LIUKAI0815
commented
2 months ago LIUKAI0815
commented
2 months ago
Reminder
System Info
llamafactoryversion: 0.8.4.dev0Reproduction
llamafactory-cli train \ --stage sft \ --do_train True \ --model_name_or_path THUDM/glm-4-9b-chat \ --preprocessing_num_workers 16 \ --finetuning_type lora \ --template glm4 \ --rope_scaling dynamic \ --flash_attn fa2 \ --dataset_dir data \ --dataset alpaca_zh \ --cutoff_len 4096 \ --learning_rate 5e-05 \ --num_train_epochs 3.0 \ --max_samples 100000 \ --per_device_train_batch_size 2 \ --gradient_accumulation_steps 8 \ --lr_scheduler_type cosine \ --max_grad_norm 1.0 \ --logging_steps 5 \ --save_steps 100 \ --warmup_steps 0 \ --optim adamw_torch \ --packing False \ --report_to none \ --output_dir saves/GLM-4-9B-Chat/lora/train_2024-08-07-11-28-01 \ --bf16 True \ --plot_loss True \ --ddp_timeout 180000000 \ --include_num_input_tokens_seen True \ --lora_rank 8 \ --lora_alpha 16 \ --lora_dropout 0 \ --use_dora True \ --lora_target all
Expected behavior
No response
Others
No response