2024-08-05 09:36:22.985 | INFO | llmc.utils.export_trtllm:cvt_trtllm_engine:93 - Start to export trtllm engine...
Loading checkpoint shards: 100%|████████████████████████████████████████████████████████████████| 19/19 [00:09<00:00, 2.08it/s]
[08/05/2024-09:36:34] Some parameters are on the meta device device because they were offloaded to the cpu.
Traceback (most recent call last):
File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/app/llmc/llmc/__main__.py", line 147, in <module>
main(config)
File "/app/llmc/llmc/__main__.py", line 88, in main
cvt_trtllm_engine(
File "/app/llmc/llmc/utils/export_trtllm.py", line 95, in cvt_trtllm_engine
convert_and_save_hf(hf_model, output_dir, cfg)
File "/app/llmc/llmc/utils/export_trtllm.py", line 88, in convert_and_save_hf
convert_and_save_rank(cfg, rank=0)
File "/app/llmc/llmc/utils/export_trtllm.py", line 75, in convert_and_save_rank
llama = LLaMAForCausalLM.from_hugging_face(
File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/models/llama/model.py", line 280, in from_hugging_face
llama = convert.from_hugging_face(
File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/models/llama/convert.py", line 1325, in from_hugging_face
weights = load_weights_from_hf(config=config,
File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/models/llama/convert.py", line 1434, in load_weights_from_hf
weights = convert_hf_llama(
File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/models/llama/convert.py", line 1087, in convert_hf_llama
convert_layer(l)
File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/models/llama/convert.py", line 740, in convert_layer
get_tllm_linear_weight(split_v, tllm_prex + 'attention.qkv.',
File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/models/llama/convert.py", line 457, in get_tllm_linear_weight
v.cpu(), plugin_weight_only_quant_type)
NotImplementedError: Cannot copy out of meta tensor; no data!
config file:
get error: