<reponame> vs <repo_name>

QwenLM / Qwen2.5-Coder

Qwen2.5-Coder is the code version of Qwen2.5, the large language model series developed by Qwen team, Alibaba Cloud.

482 stars 30 forks source link

<reponame> vs <repo_name> #57

Closed steampunque closed 4 months ago

steampunque commented 4 months ago

\ as given in the example tokenizes as plaintext as follows: SPECIAL=1 MODE=tokenize lm "\" 35403,1648,3487,1873,35393

while appears to be the actual special token: SPECIAL=1 MODE=tokenize lm "" 92299

The full set of special tokens seems OK with this change. and are missing in the tokenizer_config.json for both base and chat, but both base and chat return: SPECIAL=1 MODE=tokenize lm "" 92295,92296,92297,92298,92299,92300

I am using llama.cpp ggufs for this test.

cyente commented 4 months ago

that is really something wrong, when i write the examples. the special token is <repo_name> not <reponame>. I will revise the examples right now. Thanks!