RuntimeError: Internal: src/sentencepiece_processor.cc(1101) [model_proto->ParseFromArray(serialized.data(), serialized.size())]

[wangfh5]

系统: WSL2

1. 准备了一个空虚拟环境, 选用传统索引方式, 安装依赖requirements-sy.txt;
2. 运行run_data_processing.sh, 无问题;
3. 运行run_GLM6B.sh, 提示没有安装torch, 于是安装;
4. 运行run_GLM6B.sh, 报错RuntimeError: Internal: src/sentencepiece_processor.cc(1101) [model_proto->ParseFromArray(serialized.data(), serialized.size())]

完整的报错输出如下. 这个错误我一直遇到, 不管是用传统索引还是用语义分割.

glm模型地址 model/chatglm-6b
rwkv模型地址 model/RWKV-4-Raven-7B-v7-ChnEng-20230404-ctx2048.pth
rwkv模型参数 cuda fp16
日志记录 True
知识库类型 s
LLM模型类型 glm6b
chunk_size 400
chunk_count 3
<frozen importlib._bootstrap>:1049: ImportWarning: _ImportRedirect.find_spec() not found; falling back to find_module()
serving on 0.0.0.0:17860 view at http://127.0.0.1:17860
<frozen importlib._bootstrap>:1049: ImportWarning: _ImportRedirect.find_spec() not found; falling back to find_module()
<frozen importlib._bootstrap>:1049: ImportWarning: _ImportRedirect.find_spec() not found; falling back to find_module()
<frozen importlib._bootstrap>:1049: ImportWarning: _ImportRedirect.find_spec() not found; falling back to find_module()
<frozen importlib._bootstrap>:1049: ImportWarning: _ImportRedirect.find_spec() not found; falling back to find_module()
<frozen importlib._bootstrap>:1049: ImportWarning: _ImportRedirect.find_spec() not found; falling back to find_module()
/home/wangfh5/anaconda3/envs/wenda/lib/python3.11/site-packages/jieba/analyse/tfidf.py:47: ResourceWarning: unclosed file <_io.BufferedReader name='/home/wangfh5/anaconda3/envs/wenda/lib/python3.11/site-packages/jieba/analyse/idf.txt'>
  content = open(new_idf_path, 'rb').read().decode('utf-8')
ResourceWarning: Enable tracemalloc to get the object allocation traceback
知识库加载完成
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Exception in thread Thread-1 (load_model):
Traceback (most recent call last):
  File "/home/wangfh5/anaconda3/envs/wenda/lib/python3.11/threading.py", line 1038, in _bootstrap_inner
    self.run()
  File "/home/wangfh5/anaconda3/envs/wenda/lib/python3.11/threading.py", line 975, in run
    self._target(*self._args, **self._kwargs)
  File "/home/wangfh5/AIGC_projects/wenda/wenda.py", line 117, in load_model
    LLM.load_model()
  File "/home/wangfh5/AIGC_projects/wenda/plugins/llm_glm6b.py", line 25, in load_model
    tokenizer = AutoTokenizer.from_pretrained(settings.glm_path, local_files_only=True, trust_remote_code=True)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/wangfh5/anaconda3/envs/wenda/lib/python3.11/site-packages/transformers/models/auto/tokenization_auto.py", line 658, in from_pretrained
    return tokenizer_class.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/wangfh5/anaconda3/envs/wenda/lib/python3.11/site-packages/transformers/tokenization_utils_base.py", line 1804, in from_pretrained
    return cls._from_pretrained(
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/wangfh5/anaconda3/envs/wenda/lib/python3.11/site-packages/transformers/tokenization_utils_base.py", line 1959, in _from_pretrained
    tokenizer = cls(*init_inputs, **init_kwargs)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/wangfh5/.cache/huggingface/modules/transformers_modules/local/tokenization_chatglm.py", line 205, in __init__
    self.sp_tokenizer = SPTokenizer(vocab_file, num_image_tokens=num_image_tokens)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/wangfh5/.cache/huggingface/modules/transformers_modules/local/tokenization_chatglm.py", line 61, in __init__
    self.text_tokenizer = TextTokenizer(vocab_file)
                          ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/wangfh5/.cache/huggingface/modules/transformers_modules/local/tokenization_chatglm.py", line 22, in __init__
    self.sp.Load(model_path)
  File "/home/wangfh5/anaconda3/envs/wenda/lib/python3.11/site-packages/sentencepiece/__init__.py", line 905, in Load
    return self.LoadFromFile(model_file)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/wangfh5/anaconda3/envs/wenda/lib/python3.11/site-packages/sentencepiece/__init__.py", line 310, in LoadFromFile
    return _sentencepiece.SentencePieceProcessor_LoadFromFile(self, arg)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Internal: src/sentencepiece_processor.cc(1101) [model_proto->ParseFromArray(serialized.data(), serialized.size())]

[l15y]

这部分是是官方的代码，建议过去问：https://github.com/THUDM/ChatGLM-6B

[wangfh5]

好的, 不过我本地部署的ChatGLM-6B是能运行的

[wangfh5]

@l15y 有可能是我下载的ChatGLM-6b模型的问题. 不知道大佬方不方便在(后续版本的)懒人包里面也加上完整的chatglm-6b模型作为可选项?

[l15y]

可以直接官方下最新的

[wangfh5]

正确下载模型之后ok了, https://github.com/THUDM/ChatGLM-6B/issues/480