作者您好，请问chatglm-6b的Tokenizer是用什么算法训练的？ - Githubissues

THUDM / ChatGLM-6B

ChatGLM-6B: An Open Bilingual Dialogue Language Model | 开源双语对话语言模型

Apache License 2.0

39.96k stars 5.15k forks source link

作者您好，请问chatglm-6b的Tokenizer是用什么算法训练的？ #1443

Closed imxtx closed 5 months ago

imxtx commented 6 months ago

Is there an existing issue for this?

[X] I have searched the existing issues

Current Behavior

non

Expected Behavior

No response

Steps To Reproduce

non

Environment

- OS:
- Python:
- Transformers:
- PyTorch:
- CUDA Support (`python -c "import torch; print(torch.cuda.is_available())"`) :

Anything else?

作者您好，请问chatglm-6b的Tokenizer是用什么算法训练的？是BPE吗？请问有具体的流程和代码吗？