search

hiyouga / LLaMA-Factory

Unify Efficient Fine-Tuning of 100+ LLMs
Apache License 2.0
25.26k stars 3.13k forks source link

stop word of template of qwen #4444

Closed HackGiter closed 4 days ago

HackGiter commented 4 days ago

Currently, stop word of template of qwen is <|im_end|>. I think it should be <|endoftext|>, right? Normal behavior of pretraining data processing should be every example is seperated by <|endoftext|> instead of <|im_end|>

hiyouga commented 4 days ago

Do not specify the template argument during pretraining. It will use <|endoftext|> as the eos token