I tried to use the docker image of llama.cpp to convert the following vector models to gguf format, but I got the same error without exception: NotImplementedError: BPE pre-tokenizer was not recognized - update get_vocab_base_pre()
sensenova/piccolo-base-zh
thenlper/gte-base-zh
BAAI/bge-base-zh-v1.5
infgrad/stella-base-zh-v2
I understand that this problem may be encountered by all models that use BPE. Please tell me how to solve this problem. It seems that this bug does not exist in older versions of llama.cpp. Which version numbers of the docker image can I try?
Name and Version
ggerganov/llama.cpp full bb70a8fa2901
What operating system are you seeing the problem on?
What happened?
I tried to use the docker image of llama.cpp to convert the following vector models to gguf format, but I got the same error without exception:
NotImplementedError: BPE pre-tokenizer was not recognized - update get_vocab_base_pre()
I understand that this problem may be encountered by all models that use BPE. Please tell me how to solve this problem. It seems that this bug does not exist in older versions of llama.cpp. Which version numbers of the docker image can I try?
Name and Version
ggerganov/llama.cpp full bb70a8fa2901
What operating system are you seeing the problem on?
Linux
Relevant log output