Open Ace-RR opened 2 months ago
@nv-guomingz This is a new feature due to changes in the weight format and tokenizer of mistral v0.3
hi @Ace-RR , a quick war is to delete the consolidated.safetensors from you local model directory. Or you can apply below changes on your side.
Hi @Ace-RR we've merged the fixing into internal repo and please fetch the next weekly update for your further work.
This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 15 days."
System Info
on H100 Nvidia
Who can help?
No response
Information
Tasks
examples
folder (such as GLUE/SQuAD, ...)Reproduction
1.git -C /workspace clone https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3
2.python tensorrt_llm/examples/llama/convert_checkpoint.py --model_dir /workspace/Mistral-7B-Instruct-v0.3 --output_dir /workspace/trt_ckpt/mistral3/fp16 --dtype bfloat16
Expected behavior
0.11.0.dev2024060400 Total time of converting checkpoints: xx:xx:xx
actual behavior
[TensorRT-LLM] TensorRT-LLM version: 0.11.0.dev2024060400 0.11.0.dev2024060400 Traceback (most recent call last): File "/app/tensorrt_llm/examples/llama/convert_checkpoint.py", line 439, in
main()
File "/app/tensorrt_llm/examples/llama/convert_checkpoint.py", line 431, in main
convert_and_save_hf(args)
File "/app/tensorrt_llm/examples/llama/convert_checkpoint.py", line 366, in convert_and_save_hf
execute(args.workers, [convert_and_save_rank] * world_size, args)
File "/app/tensorrt_llm/examples/llama/convert_checkpoint.py", line 390, in execute
f(args, rank)
File "/app/tensorrt_llm/examples/llama/convert_checkpoint.py", line 355, in convert_and_save_rank
llama = LLaMAForCausalLM.from_hugging_face(
File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/models/llama/model.py", line 292, in from_hugging_face
weights = load_weights_from_hf_safetensors(hf_model_dir, config)
File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/models/llama/convert.py", line 1577, in load_weights_from_hf_safetensors
weights['transformer.vocab_embedding.weight'] = load(
File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/models/llama/convert.py", line 1555, in load
res = safetensors_ptrs[ptr_idx].get_tensor(key)
safetensors_rust.SafetensorError: File does not contain tensor model.embed_tokens.weight
additional notes
Mistral 7B v0.3 requires transformers 4.42.0.dev0
version of transformers with tensorrtllm_backend: 4.40.2
But the command doesn't work even if transformers is upgrade to 4.42.0.dev0