search

microsoft / Megatron-DeepSpeed

Ongoing research training transformer language models at scale, including: BERT & GPT-2
Other
1.9k stars 346 forks source link

Support Llama2Tokenizer #375

Open jinyouzhi opened 7 months ago

jinyouzhi commented 7 months ago

This PR hope to provide more options in Tokenizer and is primarily based on https://github.com/NVIDIA/Megatron-LM/commit/4dc071520485577bbbe7f64c1e49e2df46931dc8 for reference.