训练代码中的eos_token_id设置疑问

OpenMOSS / MOSS

An open-source tool-augmented conversational language model from Fudan University

https://txsun1997.github.io/blogs/moss.html

Apache License 2.0

11.95k stars 1.15k forks source link

Open xdnjust opened 1 year ago

xdnjust commented 1 year ago

老师还，感谢开源工作！有个问题， finetune_moss.py文件中第178行将tokenizer.eos_token_id重新设置的目的是什么，我没看懂后面的解释