Open DuFangY opened 10 months ago
alright I solved this just now, by reinstall 4.27.1 transformers, problem here in my environment can be solved.
alright I solved this just now, by reinstall 4.27.1 transformers, problem here in my environment can be solved. I solved the problem by following your method, thank you!
4.27.1
thank you,solved
move line "self.sp_tokenizer = SPTokenizer(vocab_file, num_image_tokens=num_image_tokens) " before "super().init(" at "init" of class ChatGLMTokenizer can solve this issue.
check if you are using ChatGLM1 but no ChatGLM2
4.27.1 not run on my computer.
Exception has occurred: RuntimeError
Internal: D:\a\sentencepiece\sentencepiece\src\sentencepiece_processor.cc(1102) [model_proto->ParseFromArray(serialized.data(), serialized.size())]
File "C:\Users\Administrator.cache\huggingface\modules\transformers_modules\visualglm-6b-model-gitee\tokenization_chatglm.py", line 22, in init
self.sp.Load(model_path)
File "C:\Users\Administrator.cache\huggingface\modules\transformers_modules\visualglm-6b-model-gitee\tokenization_chatglm.py", line 64, in init
self.text_tokenizer = TextTokenizer(vocab_file)
File "C:\Users\Administrator.cache\huggingface\modules\transformers_modules\visualglm-6b-model-gitee\tokenization_chatglm.py", line 221, in init
self.sp_tokenizer = SPTokenizer(vocab_file, num_image_tokens=num_image_tokens)
File "D:\1MyFiles\code\python\visualglm-6b-main\2test.py", line 4, in
+1, is that solved?