Qwen1&Qwen1.5&Qwen2 tokenizer fix

alibaba / Pai-Megatron-Patch

The official repo of Pai-Megatron-Patch for LLM & VLM large scale training developed by Alibaba Cloud.

Apache License 2.0

674 stars 94 forks source link

Qwen1&Qwen1.5&Qwen2 tokenizer fix #266

Closed Carrie-Yi closed 3 months ago

Carrie-Yi commented 3 months ago

fix Qwen1&Qwen1.5&Qwen2 tokenize code @jerryli1981

CLAassistant commented 3 months ago

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}