Fix an issue with philschmid/gemma-tokenizer-chatml tokenizer in sft

huggingface / alignment-handbook

Robust recipes to align language models with human and AI preferences

https://huggingface.co/HuggingFaceH4

Apache License 2.0

4.18k stars 354 forks source link

Fix an issue with philschmid/gemma-tokenizer-chatml tokenizer in sft #146

Closed kykim0 closed 2 months ago

kykim0 commented 3 months ago

The setup_chat_format() logic should not be applied for the philschmid/gemma-tokenizer-chatml tokenizer, otherwise gemma models are trained w/o proper bos, eos tokens.

HuggingFaceDocBuilderDev commented 3 months ago

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.