search

unslothai / unsloth

Finetune Llama 3, Mistral, Phi & Gemma LLMs 2-5x faster with 80% less memory
https://unsloth.ai
Apache License 2.0
12.29k stars 796 forks source link

qlora taining on qwen1.5-15b-chat #459

Open wsp317 opened 1 month ago

wsp317 commented 1 month ago

训练qwen1.5-14b-chat，遇到下面的报错，transformers==4.38.2

RuntimeError( "Unsloth: Tokenizer's pad_token cannot be = eos_token, and we couldn't find a\n"\ "replacement of either <|reserved... or <|placeholder..." )

danielhanchen commented 1 month ago

Oh that is an issue - the pad_token must be not the same as the eos_token, otherwise the finetune will be incorrect. I'll see if I can extend the tokenizer itself

wsp317 commented 1 month ago

I change the pad_token from <|endoftext|> to <|im_end|> in qwen's tokenizer_config.json file, and the training seems work.

danielhanchen commented 1 month ago

@wsp317 I fixed it just then! Sorry on the delay! I

If you're on a local machine, please update Unsloth via

pip uninstall unsloth -y
pip install --upgrade --force-reinstall --no-cache-dir git+https://github.com/unslothai/unsloth.git

Colab and Kaggle is fine (just restart it)