Also, this seems to affect token ids and/or token id mapping for the BOS and EOS tokens. For Yi, these should be <|startoftext|> and <|endoftext|> not <s> and </s>.
Current behaviour
(Mentioned above)
Steps to reproduce
You can fine-tune Yi 34B for 1 step, with no token changes in the yaml and inspect the generated tokenizer files. You will see that tokenizer.json is not generated. You can then load the tokenizer from the output dir and run the tests above.
Config yaml
I had the following in my yaml, but it should not and does not have any influence:
Please check that this issue hasn't been reported before.
Expected Behavior
I have trained Yi 34B which uses LlamaTokenizer, but has different
pre_processor
post_processor
anddecoder
. It also uses atypical BOS and EOS tokens.You can see the differece in resulting tokenization here:
First token of each sequence is tokenized differently, because Yi should not be adding virtual space at the start, unlike Llama 2.
You can see the config diff here: https://huggingface.co/01-ai/Yi-34B-200K/raw/main/tokenizer.json https://huggingface.co/meta-llama/Llama-2-7b-hf/raw/main/tokenizer.json
Also, this seems to affect token ids and/or token id mapping for the BOS and EOS tokens. For Yi, these should be
<|startoftext|>
and<|endoftext|>
not<s>
and</s>
.Current behaviour
(Mentioned above)
Steps to reproduce
You can fine-tune Yi 34B for 1 step, with no token changes in the yaml and inspect the generated tokenizer files. You will see that
tokenizer.json
is not generated. You can then load the tokenizer from the output dir and run the tests above.Config yaml
I had the following in my yaml, but it should not and does not have any influence:
Possible solution
No response
Which Operating Systems are you using?
Python Version
N/A
axolotl branch-commit
Don't have the env anymore, but should be still reproducible
Acknowledgements