Data Gets Tokenized Before Special Tokens Are Added

[hammoudhasan]

Please check that this issue hasn't been reported before.

• [X] I searched previous Bug Reports didn't find any similar reports.

Expected Behavior

When one defines special tokens or added tokens they should be added to the tokenizer configuration figure before running the preprocessing tokenization step.

Current behaviour

Currently data is being tokenized without the specified new special tokens (i.e replaced by spaces where as in the defined chat template should appear).

Steps to reproduce

• Get tiny-llama as a base model and as your tokenizer. (pretrained not instruction finetuned)
• Try to add chatml template tokens.
• Check tokenized data.

Config yaml

No response

Possible solution

No response

Which Operating Systems are you using?

• [X] Linux
• [ ] macOS
• [ ] Windows

Python Version

3.10

axolotl branch-commit

main

Acknowledgements

• [X] My issue title is concise, descriptive, and in title casing.
• [X] I have searched the existing issues to make sure this bug has not been reported yet.
• [X] I am using the latest version of axolotl.
• [X] I have provided enough information for the maintainers to reproduce and diagnose the issue.

[winglian]

@hammoudhasan did you try adding?

tokens:
  - "<|im_start|>"
  - "<|im_end|>"

[hammoudhasan]

I don't think I had those in my config. Let me test and get back to you on that (: Thank you @winglian

[hammoudhasan]

As you mentioned those tokens were missing from my end ! Adding those worked.