Closed TylunasLi closed 6 months ago
SentencePiece
HuggingFace Tokenizers
PretrainedTokenizerFast
HuggingFace Transformers
set_special_tokens_llm_model
支持 InternLM-7B 模型推理
stop_token_id
测试过Tokenizer的模型:
SentencePiece
HuggingFace Tokenizers
PretrainedTokenizerFast
没存储json路径,目前只支持从本地文件转换。HuggingFace Transformers
set_special_tokens_llm_model
方式在转换时配置,支持 InternLM-7B 模型推理
stop_token_id
。测试过Tokenizer的模型: