dpo训练后导出模型

[hecongqing]

Reminder

• [X] I have read the README and searched the existing issues.

Reproduction

!/bin/bash

CUDA_VISIBLE_DEVICES=0 python ../../src/train_bash.py \ --stage dpo \ --do_train \ --model_name_or_path /mnt/data/legalexp/LLM_exp/MiniCPM/MiniCPM-2B-sft-bf16 \ --adapter_name_or_path /mnt/data/legalexp/LLM_exp/MiniCPM/minicpm_finetune_baseline_v2/output/CJOLoRA/checkpoint-5000 \ --create_new_adapter \ --dataset cjo \ --dataset_dir /mnt/data/legalexp/LLM_exp/MiniCPM/minicpm_finetune_baseline_v2/data/CJOChatML_DPO/ \ --template cpm \ --finetuning_type lora \ --lora_target q_proj,v_proj \ --output_dir ../../saves/LLaMA2-7B/lora/dpo \ --overwrite_cache \ --overwrite_output_dir \ --cutoff_len 2048 \ --preprocessing_num_workers 16 \ --per_device_train_batch_size 1 \ --per_device_eval_batch_size 1 \ --gradient_accumulation_steps 8 \ --lr_scheduler_type cosine \ --logging_steps 10 \ --warmup_steps 20 \ --save_steps 100 \ --eval_steps 100 \ --evaluation_strategy steps \ --load_best_model_at_end \ --learning_rate 1e-5 \ --num_train_epochs 1.0 \ --val_size 0.1 \ --dpo_ftx 1.0 \ --plot_loss \ --max_samples 100 \ --fp16

!/bin/bash

CUDA_VISIBLE_DEVICES=0 python ../../src/train_bash.py \ --stage dpo \ --do_train \ --model_name_or_path /mnt/data/legalexp/LLM_exp/MiniCPM/MiniCPM-2B-sft-bf16 \ --adapter_name_or_path /mnt/data/legalexp/LLM_exp/MiniCPM/minicpm_finetune_baseline_v2/output/CJOLoRA/checkpoint-5000 \ --create_new_adapter \ --dataset cjo \ --dataset_dir /mnt/data/legalexp/LLM_exp/MiniCPM/minicpm_finetune_baseline_v2/data/CJOChatML_DPO/ \ --template cpm \ --finetuning_type lora \ --lora_target q_proj,v_proj \ --output_dir ../../saves/LLaMA2-7B/lora/dpo \ --overwrite_cache \ --overwrite_output_dir \ --cutoff_len 2048 \ --preprocessing_num_workers 16 \ --per_device_train_batch_size 1 \ --per_device_eval_batch_size 1 \ --gradient_accumulation_steps 8 \ --lr_scheduler_type cosine \ --logging_steps 10 \ --warmup_steps 20 \ --save_steps 100 \ --eval_steps 100 \ --evaluation_strategy steps \ --load_best_model_at_end \ --learning_rate 1e-5 \ --num_train_epochs 1.0 \ --val_size 0.1 \ --dpo_ftx 1.0 \ --plot_loss \ --max_samples 100 \ --fp16

Expected behavior

No response

System Info

No response

Others

minicpm 先经过sft微调后，再通过dpo微调，最后导出模型进行infer的时候

是合并最初minicpm 模型，还是 sft后的模型

!/bin/bash

DO NOT use quantized model or quantization_bit when merging lora weights

CUDA_VISIBLE_DEVICES=0 python ../../src/export_model.py \ --model_name_or_path /mnt/data/legalexp/LLM_exp/MiniCPM/MiniCPM-2B-sft-bf16 \ --adapter_name_or_path ../../saves/LLaMA2-7B/lora/dpo \ --template default \ --finetuning_type lora \ --export_dir ../../saves/minicpm_dpo \ --export_size 2 \ --export_legacy_format False

[hiyouga]

sft 的