Closed qkrtnskfk23 closed 11 months ago
Hi! Given a script for full parameter fine-tuning, only three simple steps are needed to change it to PEFT:
--llama_type
argument to llama_peft or llama_adapter based on your needs**.json
configuration file and pass it (together with the original llama configuration, e.g. 7B.json
) to the --llama_config
argument. An example is hereThe following bash script is an example for Norm+Bias+Lora tuning on sharegpt
#!/bin/bash
pretrained_path=$1
pretrained_type=meta_ori
llama_config="$2 configs/model/finetune/sg/llamaPeft_normBiasLora.json"
tokenizer_path="$3"
data_config=configs/data/finetune/sg/dialog_sharegpt.yaml
data_parallel=sdp
model_parallel=1
exp_name=finetune/sg/dialog_sharegpt_normBiasLora
echo "exp name: $exp_name"
mkdir -p output/"$exp_name"
torchrun --master_port=1112 --nproc_per_node=8 main_finetune.py \
--output_dir output/"$exp_name" --epochs 4 --warmup_epochs 0.5 \
--batch_size 4 --accum_iter 8 --num_workers 4 \
--max_words 2048 \
--lr 0.00003 --min_lr 0.000005 --clip_grad 2 --weight_decay 0.02 \
--data_parallel "$data_parallel" --model_parallel_size "$model_parallel" --checkpointing \
--llama_type llama_peft --llama_config $llama_config --tokenizer_path "$tokenizer_path" \
--no_visual \
--pretrained_path "$pretrained_path" --pretrained_type="$pretrained_type" \
--data_config $data_config --dialog \
2>&1 | tee -a output/"$exp_name"/output.log
echo "exp name: $exp_name"
Hi.
I wonder that where i can use the language-only parameter-efficient fine-tuning for multi-turn.
The examples are only for single turn, so i am curious of it.
Thanks.