llama3-70b int8+kv8 convert checkpoint failed on v0.10.0 branch

[NaNAGISaSA]

System Info

• CPU architecture: x86_64
• GPU properties
  • GPU name: NVIDIA A100
  • GPU memory size: 40G
• Libraries
  • TensorRT-LLM branch or tag: v0.10.0
  • Container used: yes, make -C docker release_build on v0.10.0 branch
• NVIDIA driver version: 525.89.02
• OS: Ubuntu 22.04

Who can help?

@Tracin @nv-guomingz

Information

• [X] The official example scripts
• [ ] My own modified scripts

Tasks

• [X] An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• [ ] My own task or dataset (give details below)

Reproduction

model_name=llama3_70b hf_model_dir=/some-path/Meta-Llama-3-70B-Instruct convert_model_dir=/some-path trt_engine_dir=/some-path tp_size=2 # tp_size=4 and tp_size=8 produces the same error

python3 examples/llama/convert_checkpoint.py --model_dir ${hf_model_dir} \ --tp_size ${tp_size} \ --workers ${tp_size} \ --use_weight_only \ --weight_only_precision int8 \ --int8_kv_cache \ --dtype bfloat16 \ --output_dir ${convert_model_dir}/${dtype}/${tp_size}-gpu/

Expected behavior

convert success

actual behavior

[TensorRT-LLM] TensorRT-LLM version: 0.10.0 0.10.0 Loading checkpoint shards: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 30/30 [00:23<00:00, 1.27it/s] You are using the default legacy behaviour of the <class 'transformers.models.llama.tokenization_llama.LlamaTokenizer'>. This is expected, and simply means that the legacy (previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, set legacy=False. This should only be set if you understand what it means, and thoroughly read the reason why this was added as explained in https://github.com/huggingface/transformers/pull/24565 Traceback (most recent call last): File "/workspace/volume/wangchao2/TensorRT-LLM/examples/llama/convert_checkpoint.py", line 464, in main() File "/workspace/volume/wangchao2/TensorRT-LLM/examples/llama/convert_checkpoint.py", line 456, in main convert_and_save_hf(args) File "/workspace/volume/wangchao2/TensorRT-LLM/examples/llama/convert_checkpoint.py", line 360, in convert_and_save_hf LLaMAForCausalLM.quantize(args.model_dir, File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/models/llama/model.py", line 414, in quantize convert.quantize( File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/models/llama/convert.py", line 1387, in quantize act_range, llama_qkv_para, llama_smoother = smooth_quant( File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/models/llama/convert.py", line 1160, in smooth_quant tokenizer = AutoTokenizer.from_pretrained(model_dir, File "/usr/local/lib/python3.10/dist-packages/transformers/models/auto/tokenization_auto.py", line 883, in from_pretrained return tokenizer_class_py.from_pretrained(pretrained_model_name_or_path, *inputs, *kwargs) File "/usr/local/lib/python3.10/dist-packages/transformers/tokenization_utils_base.py", line 2089, in from_pretrained return cls._from_pretrained( File "/usr/local/lib/python3.10/dist-packages/transformers/tokenization_utils_base.py", line 2311, in _from_pretrained tokenizer = cls(init_inputs, **init_kwargs) File "/usr/local/lib/python3.10/dist-packages/transformers/models/llama/tokenization_llama.py", line 169, in init self.sp_model = self.get_spm_processor(kwargs.pop("from_slow", False)) File "/usr/local/lib/python3.10/dist-packages/transformers/models/llama/tokenization_llama.py", line 196, in get_spm_processor tokenizer.Load(self.vocab_file) File "/usr/local/lib/python3.10/dist-packages/sentencepiece/init.py", line 961, in Load return self.LoadFromFile(model_file) File "/usr/local/lib/python3.10/dist-packages/sentencepiece/init.py", line 316, in LoadFromFile return _sentencepiece.SentencePieceProcessor_LoadFromFile(self, arg) TypeError: not a string

additional notes

I also tested llama3-8b, change hf_model_dir to Meta-Llama-3-8B-Instruct, convertion is success:

[TensorRT-LLM] TensorRT-LLM version: 0.10.0 0.10.0 Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:02<00:00, 1.36it/s] Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained. calibrating model: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 512/512 [01:25<00:00, 6.00it/s] Weights loaded. Total time: 00:00:41 Weights loaded. Total time: 00:00:36 Total time of converting checkpoints: 00:03:31

[hijkzzz]

Could you try the latest version TRT_LLM 0.11+ see the tutorial: https://nvidia.github.io/TensorRT-LLM/installation/linux.html

[Yoh-Z]

Could you try the latest version TRT_LLM 0.11+ see the tutorial: https://nvidia.github.io/TensorRT-LLM/installation/linux.html

Which commit corresponds to version 0.11.0