Open Naozumi520 opened 2 hours ago
My config:
### model
model_name_or_path: ../NovaLLM_7b_20241013_serverChatHistoryMsgPT-s1-epoch=1.0
### method
stage: sft
do_train: true
finetuning_type: full
use_unsloth: true
use_galore: true
galore_layerwise: true
galore_target: all
galore_rank: 128
galore_scale: 2.0
### dataset
dataset: somedataset
template: qwen
cutoff_len: 1024
overwrite_cache: true
preprocessing_num_workers: 16
### output
output_dir: ../output
logging_steps: 10
save_steps: 500
plot_loss: true
overwrite_output_dir: true
### train
per_device_train_batch_size: 1
learning_rate: 1.0e-4
num_train_epochs: 2.0
lr_scheduler_type: cosine
warmup_ratio: 0.1
pure_bf16: true
bf16: true
ddp_timeout: 180000000
### eval
val_size: 0.15
per_device_eval_batch_size: 1
eval_strategy: steps
eval_steps: 250
report_to: wandb
run_name: 7b
RuntimeError: Unsloth: The tokenizer
../NovaLLM_7b_20241013_serverChatHistoryMsgPT-s1-epoch=1.0
does not have a {% if add_generation_prompt %} for generation purposes. Please file a bug report immediately - thanks!Finetune using LLaMA-Factory, both default and qwen template did not work.
the chat_template in tokenzier_config.json: