尝试将qwen1.5 14b llamafy后unsloth训练报错

[Data2Me]

Reminder

• [X] I have read the README and searched the existing issues.

Reproduction

报错信息： File "/mnt/cpfs/guanyutao/new-LLaMa-Factory/LLaMA-Factory/src/train.py", line 5, in main run_exp() File "/mnt/cpfs/guanyutao/new-LLaMa-Factory/LLaMA-Factory/src/llmtuner/train/tuner.py", line 33, in run_exp run_sft(model_args, data_args, training_args, finetuning_args, generating_args, callbacks) File "/mnt/cpfs/guanyutao/new-LLaMa-Factory/LLaMA-Factory/src/llmtuner/train/sft/workflow.py", line 34, in run_sft model = load_model(tokenizer, model_args, finetuning_args, training_args.do_train) File "/mnt/cpfs/guanyutao/new-LLaMa-Factory/LLaMA-Factory/src/llmtuner/model/loader.py", line 117, in load_model model = load_unsloth_pretrained_model(config, model_args) File "/mnt/cpfs/guanyutao/new-LLaMa-Factory/LLaMA-Factory/src/llmtuner/model/utils/unsloth.py", line 43, in load_unsloth_pretrainedmodel model, = FastLanguageModel.from_pretrained(**unsloth_kwargs) File "/opt/conda/lib/python3.10/site-packages/unsloth/models/loader.py", line 142, in from_pretrained model, tokenizer = dispatch_model.from_pretrained( File "/opt/conda/lib/python3.10/site-packages/unsloth/models/llama.py", line 1121, in from_pretrained model, tokenizer = patch_tokenizer(model, tokenizer) File "/opt/conda/lib/python3.10/site-packages/unsloth/models/_utils.py", line 185, in patch_tokenizer raise RuntimeError( RuntimeError: Unsloth: Tokenizer's pad_token cannot be = eos_token, and we couldn't find a replacement of either <|reserved... or <|placeholder... replacement of either <|reserved... or <|placeholder... 采用https://github.com/Minami-su/character_AI_open/blob/main/llamafy_qwen_v2.py进行llamafy，这是llamafy后的模型文件：

Expected behavior

No response

System Info

No response

Others

No response

[hiyouga]

unsloth 最新版已经支持了 qwen1.5 https://github.com/unslothai/unsloth/pull/447

[Data2Me]

unsloth 最新版已经支持了 qwen1.5 unslothai/unsloth#447

更新最新的unsloth，在本仓库直接微调qwen1.5就可以了吗？

[hiyouga]

是的

[camposs1979]

大佬@hiyouga能否指点一下，FSDP + Qlora的情况下，unsloth是不是不支持Qwen1.5。我尝试用最新代码运行过，结果报错： RuntimeError: Unsloth: Unfortunately Qwen2 type models do not support RoPE scaling! unsloth是用的最新的版本。