当运行时出现了tensor 'token_embd.weight' has wrong shape; expected 4096, 32001, got 4096, 32000, 1, 1

[imchangyue]

提交前必须检查以下项目

• [X] 请确保使用的是仓库最新代码（git pull），一些问题已被解决和修复。
• [X] 由于相关依赖频繁更新，请确保按照Wiki中的相关步骤执行
• [X] 我已阅读FAQ章节并且已在Issue中对问题进行了搜索，没有找到相似问题和解决方案
• [X] 第三方插件问题：例如llama.cpp、text-generation-webui、LlamaChat等，同时建议到对应的项目中查找解决方案
• [X] 模型正确性检查：务必检查模型的SHA256.md，模型不对的情况下无法保证效果和正常运行

问题类型

模型量化和部署

基础模型

LLaMA-7B

操作系统

Windows

详细描述问题

D:/llama.cpp $ ./main -m ./models/7B/ggml-model-q4_0.gguf -n 128
Log start
main: build = 2029 (d62520eb)
main: built with cc (GCC) 13.2.0 for x86_64-w64-mingw32
main: seed  = 1706770105
llama_model_loader: loaded meta data with 22 key-value pairs and 291 tensors from ./models/7B/ggml-model-q4_0.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = llama
llama_model_loader: - kv   1:                               general.name str              = LLaMA v2
llama_model_loader: - kv   2:                       llama.context_length u32              = 4096
llama_model_loader: - kv   3:                     llama.embedding_length u32              = 4096
llama_model_loader: - kv   4:                          llama.block_count u32              = 32
llama_model_loader: - kv   5:                  llama.feed_forward_length u32              = 11008
llama_model_loader: - kv   6:                 llama.rope.dimension_count u32              = 128
llama_model_loader: - kv   7:                 llama.attention.head_count u32              = 32
llama_model_loader: - kv   8:              llama.attention.head_count_kv u32              = 32
llama_model_loader: - kv   9:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010
llama_model_loader: - kv  10:                          general.file_type u32              = 2
llama_model_loader: - kv  11:                       tokenizer.ggml.model str              = llama
llama_model_loader: - kv  12:                      tokenizer.ggml.tokens arr[str,32001]   = ["<unk>", "<s>", "</s>", "<0x00>", "<...
llama_model_loader: - kv  13:                      tokenizer.ggml.scores arr[f32,32001]   = [0.000000, 0.000000, 0.000000, 0.0000...
llama_model_loader: - kv  14:                  tokenizer.ggml.token_type arr[i32,32001]   = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ...
llama_model_loader: - kv  15:                tokenizer.ggml.bos_token_id u32              = 1
llama_model_loader: - kv  16:                tokenizer.ggml.eos_token_id u32              = 2
llama_model_loader: - kv  17:            tokenizer.ggml.unknown_token_id u32              = 0
llama_model_loader: - kv  18:            tokenizer.ggml.padding_token_id u32              = 0
llama_model_loader: - kv  19:               tokenizer.ggml.add_bos_token bool             = true
llama_model_loader: - kv  20:               tokenizer.ggml.add_eos_token bool             = false
llama_model_loader: - kv  21:               general.quantization_version u32              = 2
llama_model_loader: - type  f32:   65 tensors
llama_model_loader: - type q4_0:  225 tensors
llama_model_loader: - type q6_K:    1 tensors
llm_load_vocab: special tokens definition check successful ( 260/32001 ).
llm_load_print_meta: format           = GGUF V3 (latest)
llm_load_print_meta: arch             = llama
llm_load_print_meta: vocab type       = SPM
llm_load_print_meta: n_vocab          = 32001
llm_load_print_meta: n_merges         = 0
llm_load_print_meta: n_ctx_train      = 4096
llm_load_print_meta: n_embd           = 4096
llm_load_print_meta: n_head           = 32
llm_load_print_meta: n_head_kv        = 32
llm_load_print_meta: n_layer          = 32
llm_load_print_meta: n_rot            = 128
llm_load_print_meta: n_embd_head_k    = 128
llm_load_print_meta: n_embd_head_v    = 128
llm_load_print_meta: n_gqa            = 1
llm_load_print_meta: n_embd_k_gqa     = 4096
llm_load_print_meta: n_embd_v_gqa     = 4096
llm_load_print_meta: f_norm_eps       = 0.0e+00
llm_load_print_meta: f_norm_rms_eps   = 1.0e-05
llm_load_print_meta: f_clamp_kqv      = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: n_ff             = 11008
llm_load_print_meta: n_expert         = 0
llm_load_print_meta: n_expert_used    = 0
llm_load_print_meta: rope scaling     = linear
llm_load_print_meta: freq_base_train  = 10000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: n_yarn_orig_ctx  = 4096
llm_load_print_meta: rope_finetuned   = unknown
llm_load_print_meta: model type       = 7B
llm_load_print_meta: model ftype      = Q4_0
llm_load_print_meta: model params     = 6.74 B
llm_load_print_meta: model size       = 3.56 GiB (4.54 BPW)
llm_load_print_meta: general.name     = LLaMA v2
llm_load_print_meta: BOS token        = 1 '<s>'
llm_load_print_meta: EOS token        = 2 '</s>'
llm_load_print_meta: UNK token        = 0 '<unk>'
llm_load_print_meta: PAD token        = 0 '<unk>'
llm_load_print_meta: LF token         = 13 '<0x0A>'
llm_load_tensors: ggml ctx size =    0.11 MiB
llama_model_load: error loading model: create_tensor: tensor 'token_embd.weight' has wrong shape; expected  4096, 32001, got  4096, 32000,     1,     1
llama_load_model_from_file: failed to load model
llama_init_from_gpt_params: error: failed to load model './models/7B/ggml-model-q4_0.gguf'
main: error: unable to load model

依赖情况（代码类问题务必提供）

均为最新版本

运行日志或截图

[ymcui]

你的模型从哪儿来的？词表大小很奇怪，报错信息显示是32001，既不是原版英文llama，也不是我们的中文llama。

[imchangyue]

obtain the original LLaMA model weights and place them in ./models

ls ./models 65B 30B 13B 7B tokenizer_checklist.chk tokenizer.model

[Optional] for models using BPE tokenizers

ls ./models 65B 30B 13B 7B vocab.json

install Python dependencies

python3 -m pip install -r requirements.txt

convert the 7B model to ggml FP16 format

python3 convert.py models/7B/

[Optional] for models using BPE tokenizers

python convert.py models/7B/ --vocabtype bpe

quantize the model to 4-bits (using q4_0 method)

./quantize ./models/7B/ggml-model-f16.gguf ./models/7B/ggml-model-q4_0.gguf q4_0

update the gguf filetype to current if older version is unsupported by another application

./quantize ./models/7B/ggml-model-q4_0.gguf ./models/7B/ggml-model-q4_0-v2.gguf COPY

run the inference

./main -m ./models/7B/ggml-model-q4_0.gguf -n 128 我是在运行这一步的时候卡住了 Exception: Vocab size mismatch (model has 32000, but models\7B-chat\tokenizer.model has 32001) 在这一步的时候我改了一个.json文件里面的参数，但是最后运行到这一步卡住了，但是这一次改参数不好使了 因为我下的modelscope里面的7B-chat,链接是这个：https://modelscope.cn/models/modelscope/Llama-2-7b-chat-ms/summary

[ymcui]

我不太清楚modelscope你下载的模型是不是原版llama-2。原版llama-2(或llama-2-chat)的词表大小就是32000，直接转换不会报这个问题，你可以考虑从其他地方下载，比如huggingface。你可以只下载tokenizer、config进行替换，看看能不能解决（前提是你从modelscope下载的权重里词表大小是32000）。

P.S. 你的这个问题和我们的项目无关，我就先关闭了。