search

ggerganov / llama.cpp

LLM inference in C/C++
MIT License
64.59k stars 9.25k forks source link

Bug: NotImplementedError: BPE pre-tokenizer was not recognized - update get_vocab_base_pre() #8649

Open wencan opened 1 month ago

wencan commented 1 month ago

What happened?

I tried to use the docker image of llama.cpp to convert the following vector models to gguf format, but I got the same error without exception: NotImplementedError: BPE pre-tokenizer was not recognized - update get_vocab_base_pre()

I understand that this problem may be encountered by all models that use BPE. Please tell me how to solve this problem. It seems that this bug does not exist in older versions of llama.cpp. Which version numbers of the docker image can I try?

Name and Version

ggerganov/llama.cpp full bb70a8fa2901

What operating system are you seeing the problem on?

Linux

Relevant log output

INFO:hf-to-gguf:Loading model: piccolo-base-zh
INFO:gguf.gguf_writer:gguf: This GGUF file is for Little Endian only
INFO:hf-to-gguf:Exporting model...
INFO:hf-to-gguf:gguf: loading model part 'pytorch_model.bin'
INFO:hf-to-gguf:token_embd.weight,               torch.float16 --> F16, shape = {768, 21128}
INFO:hf-to-gguf:position_embd.weight,            torch.float16 --> F32, shape = {768, 512}
INFO:hf-to-gguf:token_types.weight,              torch.float16 --> F32, shape = {768, 2}
INFO:hf-to-gguf:token_embd_norm.weight,          torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:token_embd_norm.bias,            torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.0.attn_q.weight,             torch.float16 --> F16, shape = {768, 768}
INFO:hf-to-gguf:blk.0.attn_q.bias,               torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.0.attn_k.weight,             torch.float16 --> F16, shape = {768, 768}
INFO:hf-to-gguf:blk.0.attn_k.bias,               torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.0.attn_v.weight,             torch.float16 --> F16, shape = {768, 768}
INFO:hf-to-gguf:blk.0.attn_v.bias,               torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.0.attn_output.weight,        torch.float16 --> F16, shape = {768, 768}
INFO:hf-to-gguf:blk.0.attn_output.bias,          torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.0.attn_output_norm.weight,   torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.0.attn_output_norm.bias,     torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.0.ffn_up.weight,             torch.float16 --> F16, shape = {768, 3072}
INFO:hf-to-gguf:blk.0.ffn_up.bias,               torch.float16 --> F32, shape = {3072}
INFO:hf-to-gguf:blk.0.ffn_down.weight,           torch.float16 --> F16, shape = {3072, 768}
INFO:hf-to-gguf:blk.0.ffn_down.bias,             torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.0.layer_output_norm.weight,  torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.0.layer_output_norm.bias,    torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.1.attn_q.weight,             torch.float16 --> F16, shape = {768, 768}
INFO:hf-to-gguf:blk.1.attn_q.bias,               torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.1.attn_k.weight,             torch.float16 --> F16, shape = {768, 768}
INFO:hf-to-gguf:blk.1.attn_k.bias,               torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.1.attn_v.weight,             torch.float16 --> F16, shape = {768, 768}
INFO:hf-to-gguf:blk.1.attn_v.bias,               torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.1.attn_output.weight,        torch.float16 --> F16, shape = {768, 768}
INFO:hf-to-gguf:blk.1.attn_output.bias,          torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.1.attn_output_norm.weight,   torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.1.attn_output_norm.bias,     torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.1.ffn_up.weight,             torch.float16 --> F16, shape = {768, 3072}
INFO:hf-to-gguf:blk.1.ffn_up.bias,               torch.float16 --> F32, shape = {3072}
INFO:hf-to-gguf:blk.1.ffn_down.weight,           torch.float16 --> F16, shape = {3072, 768}
INFO:hf-to-gguf:blk.1.ffn_down.bias,             torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.1.layer_output_norm.weight,  torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.1.layer_output_norm.bias,    torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.2.attn_q.weight,             torch.float16 --> F16, shape = {768, 768}
INFO:hf-to-gguf:blk.2.attn_q.bias,               torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.2.attn_k.weight,             torch.float16 --> F16, shape = {768, 768}
INFO:hf-to-gguf:blk.2.attn_k.bias,               torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.2.attn_v.weight,             torch.float16 --> F16, shape = {768, 768}
INFO:hf-to-gguf:blk.2.attn_v.bias,               torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.2.attn_output.weight,        torch.float16 --> F16, shape = {768, 768}
INFO:hf-to-gguf:blk.2.attn_output.bias,          torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.2.attn_output_norm.weight,   torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.2.attn_output_norm.bias,     torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.2.ffn_up.weight,             torch.float16 --> F16, shape = {768, 3072}
INFO:hf-to-gguf:blk.2.ffn_up.bias,               torch.float16 --> F32, shape = {3072}
INFO:hf-to-gguf:blk.2.ffn_down.weight,           torch.float16 --> F16, shape = {3072, 768}
INFO:hf-to-gguf:blk.2.ffn_down.bias,             torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.2.layer_output_norm.weight,  torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.2.layer_output_norm.bias,    torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.3.attn_q.weight,             torch.float16 --> F16, shape = {768, 768}
INFO:hf-to-gguf:blk.3.attn_q.bias,               torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.3.attn_k.weight,             torch.float16 --> F16, shape = {768, 768}
INFO:hf-to-gguf:blk.3.attn_k.bias,               torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.3.attn_v.weight,             torch.float16 --> F16, shape = {768, 768}
INFO:hf-to-gguf:blk.3.attn_v.bias,               torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.3.attn_output.weight,        torch.float16 --> F16, shape = {768, 768}
INFO:hf-to-gguf:blk.3.attn_output.bias,          torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.3.attn_output_norm.weight,   torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.3.attn_output_norm.bias,     torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.3.ffn_up.weight,             torch.float16 --> F16, shape = {768, 3072}
INFO:hf-to-gguf:blk.3.ffn_up.bias,               torch.float16 --> F32, shape = {3072}
INFO:hf-to-gguf:blk.3.ffn_down.weight,           torch.float16 --> F16, shape = {3072, 768}
INFO:hf-to-gguf:blk.3.ffn_down.bias,             torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.3.layer_output_norm.weight,  torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.3.layer_output_norm.bias,    torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.4.attn_q.weight,             torch.float16 --> F16, shape = {768, 768}
INFO:hf-to-gguf:blk.4.attn_q.bias,               torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.4.attn_k.weight,             torch.float16 --> F16, shape = {768, 768}
INFO:hf-to-gguf:blk.4.attn_k.bias,               torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.4.attn_v.weight,             torch.float16 --> F16, shape = {768, 768}
INFO:hf-to-gguf:blk.4.attn_v.bias,               torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.4.attn_output.weight,        torch.float16 --> F16, shape = {768, 768}
INFO:hf-to-gguf:blk.4.attn_output.bias,          torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.4.attn_output_norm.weight,   torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.4.attn_output_norm.bias,     torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.4.ffn_up.weight,             torch.float16 --> F16, shape = {768, 3072}
INFO:hf-to-gguf:blk.4.ffn_up.bias,               torch.float16 --> F32, shape = {3072}
INFO:hf-to-gguf:blk.4.ffn_down.weight,           torch.float16 --> F16, shape = {3072, 768}
INFO:hf-to-gguf:blk.4.ffn_down.bias,             torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.4.layer_output_norm.weight,  torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.4.layer_output_norm.bias,    torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.5.attn_q.weight,             torch.float16 --> F16, shape = {768, 768}
INFO:hf-to-gguf:blk.5.attn_q.bias,               torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.5.attn_k.weight,             torch.float16 --> F16, shape = {768, 768}
INFO:hf-to-gguf:blk.5.attn_k.bias,               torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.5.attn_v.weight,             torch.float16 --> F16, shape = {768, 768}
INFO:hf-to-gguf:blk.5.attn_v.bias,               torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.5.attn_output.weight,        torch.float16 --> F16, shape = {768, 768}
INFO:hf-to-gguf:blk.5.attn_output.bias,          torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.5.attn_output_norm.weight,   torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.5.attn_output_norm.bias,     torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.5.ffn_up.weight,             torch.float16 --> F16, shape = {768, 3072}
INFO:hf-to-gguf:blk.5.ffn_up.bias,               torch.float16 --> F32, shape = {3072}
INFO:hf-to-gguf:blk.5.ffn_down.weight,           torch.float16 --> F16, shape = {3072, 768}
INFO:hf-to-gguf:blk.5.ffn_down.bias,             torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.5.layer_output_norm.weight,  torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.5.layer_output_norm.bias,    torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.6.attn_q.weight,             torch.float16 --> F16, shape = {768, 768}
INFO:hf-to-gguf:blk.6.attn_q.bias,               torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.6.attn_k.weight,             torch.float16 --> F16, shape = {768, 768}
INFO:hf-to-gguf:blk.6.attn_k.bias,               torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.6.attn_v.weight,             torch.float16 --> F16, shape = {768, 768}
INFO:hf-to-gguf:blk.6.attn_v.bias,               torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.6.attn_output.weight,        torch.float16 --> F16, shape = {768, 768}
INFO:hf-to-gguf:blk.6.attn_output.bias,          torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.6.attn_output_norm.weight,   torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.6.attn_output_norm.bias,     torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.6.ffn_up.weight,             torch.float16 --> F16, shape = {768, 3072}
INFO:hf-to-gguf:blk.6.ffn_up.bias,               torch.float16 --> F32, shape = {3072}
INFO:hf-to-gguf:blk.6.ffn_down.weight,           torch.float16 --> F16, shape = {3072, 768}
INFO:hf-to-gguf:blk.6.ffn_down.bias,             torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.6.layer_output_norm.weight,  torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.6.layer_output_norm.bias,    torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.7.attn_q.weight,             torch.float16 --> F16, shape = {768, 768}
INFO:hf-to-gguf:blk.7.attn_q.bias,               torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.7.attn_k.weight,             torch.float16 --> F16, shape = {768, 768}
INFO:hf-to-gguf:blk.7.attn_k.bias,               torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.7.attn_v.weight,             torch.float16 --> F16, shape = {768, 768}
INFO:hf-to-gguf:blk.7.attn_v.bias,               torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.7.attn_output.weight,        torch.float16 --> F16, shape = {768, 768}
INFO:hf-to-gguf:blk.7.attn_output.bias,          torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.7.attn_output_norm.weight,   torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.7.attn_output_norm.bias,     torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.7.ffn_up.weight,             torch.float16 --> F16, shape = {768, 3072}
INFO:hf-to-gguf:blk.7.ffn_up.bias,               torch.float16 --> F32, shape = {3072}
INFO:hf-to-gguf:blk.7.ffn_down.weight,           torch.float16 --> F16, shape = {3072, 768}
INFO:hf-to-gguf:blk.7.ffn_down.bias,             torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.7.layer_output_norm.weight,  torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.7.layer_output_norm.bias,    torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.8.attn_q.weight,             torch.float16 --> F16, shape = {768, 768}
INFO:hf-to-gguf:blk.8.attn_q.bias,               torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.8.attn_k.weight,             torch.float16 --> F16, shape = {768, 768}
INFO:hf-to-gguf:blk.8.attn_k.bias,               torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.8.attn_v.weight,             torch.float16 --> F16, shape = {768, 768}
INFO:hf-to-gguf:blk.8.attn_v.bias,               torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.8.attn_output.weight,        torch.float16 --> F16, shape = {768, 768}
INFO:hf-to-gguf:blk.8.attn_output.bias,          torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.8.attn_output_norm.weight,   torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.8.attn_output_norm.bias,     torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.8.ffn_up.weight,             torch.float16 --> F16, shape = {768, 3072}
INFO:hf-to-gguf:blk.8.ffn_up.bias,               torch.float16 --> F32, shape = {3072}
INFO:hf-to-gguf:blk.8.ffn_down.weight,           torch.float16 --> F16, shape = {3072, 768}
INFO:hf-to-gguf:blk.8.ffn_down.bias,             torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.8.layer_output_norm.weight,  torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.8.layer_output_norm.bias,    torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.9.attn_q.weight,             torch.float16 --> F16, shape = {768, 768}
INFO:hf-to-gguf:blk.9.attn_q.bias,               torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.9.attn_k.weight,             torch.float16 --> F16, shape = {768, 768}
INFO:hf-to-gguf:blk.9.attn_k.bias,               torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.9.attn_v.weight,             torch.float16 --> F16, shape = {768, 768}
INFO:hf-to-gguf:blk.9.attn_v.bias,               torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.9.attn_output.weight,        torch.float16 --> F16, shape = {768, 768}
INFO:hf-to-gguf:blk.9.attn_output.bias,          torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.9.attn_output_norm.weight,   torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.9.attn_output_norm.bias,     torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.9.ffn_up.weight,             torch.float16 --> F16, shape = {768, 3072}
INFO:hf-to-gguf:blk.9.ffn_up.bias,               torch.float16 --> F32, shape = {3072}
INFO:hf-to-gguf:blk.9.ffn_down.weight,           torch.float16 --> F16, shape = {3072, 768}
INFO:hf-to-gguf:blk.9.ffn_down.bias,             torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.9.layer_output_norm.weight,  torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.9.layer_output_norm.bias,    torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.10.attn_q.weight,            torch.float16 --> F16, shape = {768, 768}
INFO:hf-to-gguf:blk.10.attn_q.bias,              torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.10.attn_k.weight,            torch.float16 --> F16, shape = {768, 768}
INFO:hf-to-gguf:blk.10.attn_k.bias,              torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.10.attn_v.weight,            torch.float16 --> F16, shape = {768, 768}
INFO:hf-to-gguf:blk.10.attn_v.bias,              torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.10.attn_output.weight,       torch.float16 --> F16, shape = {768, 768}
INFO:hf-to-gguf:blk.10.attn_output.bias,         torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.10.attn_output_norm.weight,  torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.10.attn_output_norm.bias,    torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.10.ffn_up.weight,            torch.float16 --> F16, shape = {768, 3072}
INFO:hf-to-gguf:blk.10.ffn_up.bias,              torch.float16 --> F32, shape = {3072}
INFO:hf-to-gguf:blk.10.ffn_down.weight,          torch.float16 --> F16, shape = {3072, 768}
INFO:hf-to-gguf:blk.10.ffn_down.bias,            torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.10.layer_output_norm.weight, torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.10.layer_output_norm.bias,   torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.11.attn_q.weight,            torch.float16 --> F16, shape = {768, 768}
INFO:hf-to-gguf:blk.11.attn_q.bias,              torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.11.attn_k.weight,            torch.float16 --> F16, shape = {768, 768}
INFO:hf-to-gguf:blk.11.attn_k.bias,              torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.11.attn_v.weight,            torch.float16 --> F16, shape = {768, 768}
INFO:hf-to-gguf:blk.11.attn_v.bias,              torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.11.attn_output.weight,       torch.float16 --> F16, shape = {768, 768}
INFO:hf-to-gguf:blk.11.attn_output.bias,         torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.11.attn_output_norm.weight,  torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.11.attn_output_norm.bias,    torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.11.ffn_up.weight,            torch.float16 --> F16, shape = {768, 3072}
INFO:hf-to-gguf:blk.11.ffn_up.bias,              torch.float16 --> F32, shape = {3072}
INFO:hf-to-gguf:blk.11.ffn_down.weight,          torch.float16 --> F16, shape = {3072, 768}
INFO:hf-to-gguf:blk.11.ffn_down.bias,            torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.11.layer_output_norm.weight, torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:blk.11.layer_output_norm.bias,   torch.float16 --> F32, shape = {768}
INFO:hf-to-gguf:Set meta model
INFO:hf-to-gguf:Set model parameters
INFO:hf-to-gguf:gguf: context length = 512
INFO:hf-to-gguf:gguf: embedding length = 768
INFO:hf-to-gguf:gguf: feed forward length = 3072
INFO:hf-to-gguf:gguf: head count = 12
INFO:hf-to-gguf:gguf: layer norm epsilon = 1e-12
INFO:hf-to-gguf:gguf: file type = 1
INFO:hf-to-gguf:Set model tokenizer
WARNING:hf-to-gguf:

WARNING:hf-to-gguf:**************************************************************************************
WARNING:hf-to-gguf:** WARNING: The BPE pre-tokenizer was not recognized!
WARNING:hf-to-gguf:**          There are 2 possible reasons for this:
WARNING:hf-to-gguf:**          - the model has not been added to convert_hf_to_gguf_update.py yet
WARNING:hf-to-gguf:**          - the pre-tokenization config has changed upstream
WARNING:hf-to-gguf:**          Check your model files and convert_hf_to_gguf_update.py and update them accordingly.
WARNING:hf-to-gguf:** ref:     https://github.com/ggerganov/llama.cpp/pull/6920
WARNING:hf-to-gguf:**
WARNING:hf-to-gguf:** chkhsh:  8e62295832751ca1e8f92f2226f403dea30dc5165e448b5bfa05af5340c64ec7
WARNING:hf-to-gguf:**************************************************************************************
WARNING:hf-to-gguf:

Traceback (most recent call last):
  File "/app/./convert_hf_to_gguf.py", line 3688, in <module>
    main()
  File "/app/./convert_hf_to_gguf.py", line 3682, in main
    model_instance.write()
  File "/app/./convert_hf_to_gguf.py", line 401, in write
    self.prepare_metadata(vocab_only=False)
  File "/app/./convert_hf_to_gguf.py", line 394, in prepare_metadata
    self.set_vocab()
  File "/app/./convert_hf_to_gguf.py", line 2414, in set_vocab
    tokens, toktypes, tokpre = self.get_vocab_base()
  File "/app/./convert_hf_to_gguf.py", line 482, in get_vocab_base
    tokpre = self.get_vocab_base_pre(tokenizer)
  File "/app/./convert_hf_to_gguf.py", line 620, in get_vocab_base_pre
    raise NotImplementedError("BPE pre-tokenizer was not recognized - update get_vocab_base_pre()")
NotImplementedError: BPE pre-tokenizer was not recognized - update get_vocab_base_pre()
wencan commented 1 month ago

I compiled the llama.cpp file with version b2636, successfully converted sensenova/piccolo-base-zh to gguf format, and quantized it to Q4_K-M format.

NeuralAIM commented 1 month ago

Same with sentence-transformers/LaBSE

b2636 - working