TensorRT-LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and build TensorRT engines that contain state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT-LLM also contains components to create Python and C++ runtimes that execute those TensorRT engines.
/code/tensorrt_llm# python examples/llama/convert_checkpoint.py --model_dir /code/tensorrt_llm/Mixtral-8x7B-Instruct-v0.1/ --dtype float16 --output_dir /code/tensorrt_llm/examples/Mixtral-8x7B-Instruct-v0.1/ --int8_kv_cache --dataset-cache-dir .
[TensorRT-LLM] TensorRT-LLM version: 0.10.0.dev2024041600
0.10.0.dev2024041600
Loading checkpoint shards: 100%|██████████████████████████████████████████████████████| 19/19 [00:14<00:00, 1.35it/s]
[04/19/2024-05:28:34] Some parameters are on the meta device device because they were offloaded to the cpu.
/usr/local/lib/python3.10/dist-packages/datasets/load.py:1461: FutureWarning: The repository for ccdv/cnn_dailymail contains custom code which must be executed to correctly load the dataset. You can inspect the repository content at https://hf.co/datasets/ccdv/cnn_dailymail
You can avoid this message in future by passing the argument trust_remote_code=True.
Passing trust_remote_code=True will be mandatory to load this dataset from the next major release of datasets.
warnings.warn(
Downloading builder script: 100%|████████████████████████████████████████████████| 9.27k/9.27k [00:00<00:00, 38.4MB/s]
Downloading readme: 100%|████████████████████████████████████████████████████████| 13.9k/13.9k [00:00<00:00, 54.7MB/s]
Traceback (most recent call last):
File "/code/tensorrt_llm/examples/llama/convert_checkpoint.py", line 446, in
main()
File "/code/tensorrt_llm/examples/llama/convert_checkpoint.py", line 438, in main
convert_and_save_hf(args)
File "/code/tensorrt_llm/examples/llama/convert_checkpoint.py", line 345, in convert_and_save_hf
LLaMAForCausalLM.quantize(args.model_dir,
File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/models/llama/model.py", line 363, in quantize
convert.quantize(
File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/models/llama/convert.py", line 1244, in quantize
act_range, llama_qkv_para, llama_smoother = smooth_quant(
File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/models/llama/convert.py", line 1037, in smooth_quant
dataset = load_dataset("ccdv/cnn_dailymail",
File "/usr/local/lib/python3.10/dist-packages/datasets/load.py", line 2582, in load_dataset
builder_instance.download_and_prepare(
File "/usr/local/lib/python3.10/dist-packages/datasets/builder.py", line 1005, in download_and_prepare
self._download_and_prepare(
File "/usr/local/lib/python3.10/dist-packages/datasets/builder.py", line 1767, in _download_and_prepare
super()._download_and_prepare(
File "/usr/local/lib/python3.10/dist-packages/datasets/builder.py", line 1078, in _download_and_prepare
split_generators = self._split_generators(dl_manager, *split_generators_kwargs)
File "/root/.cache/huggingface/modules/datasets_modules/datasets/ccdv--cnn_dailymail/0107f7388b5c6fae455a5661bcd134fc22da53ea75852027040d8d1e997f101f/cnn_dailymail.py", line 255, in _split_generators
dl_paths = dl_manager.download_and_extract(_DL_URLS)
File "/usr/local/lib/python3.10/dist-packages/datasets/download/download_manager.py", line 570, in download_and_extract
return self.extract(self.download(url_or_urls))
File "/usr/local/lib/python3.10/dist-packages/datasets/download/download_manager.py", line 434, in download
downloaded_path_or_paths = map_nested(
File "/usr/local/lib/python3.10/dist-packages/datasets/utils/py_utils.py", line 475, in map_nested
mapped = [
File "/usr/local/lib/python3.10/dist-packages/datasets/utils/py_utils.py", line 476, in
_single_map_nested((function, obj, types, None, True, None))
File "/usr/local/lib/python3.10/dist-packages/datasets/utils/py_utils.py", line 368, in _single_map_nested
return function(data_struct)
File "/usr/local/lib/python3.10/dist-packages/datasets/download/download_manager.py", line 459, in _download
out = cached_path(url_or_filename, download_config=download_config)
File "/usr/local/lib/python3.10/dist-packages/datasets/utils/file_utils.py", line 190, in cached_path
output_path = get_from_cache(
File "/usr/local/lib/python3.10/dist-packages/datasets/utils/file_utils.py", line 589, in get_from_cache
raise ConnectionError(f"Couldn't reach {url} ({repr(head_error)})")
ConnectionError: Couldn't reach https://huggingface.co/datasets/ccdv/cnn_dailymail/resolve/main/cnn_stories.tgz (ConnectionError(MaxRetryError("HTTPSConnectionPool(host='cdn-lfs.huggingface.co', port=443): Max retries exceeded with url: /datasets/ccdv/cnn_dailymail/e8fbc0027e54e0a916abd9c969eb35f708ed1467d7ef4e3b17a56739d65cb200?response-content-disposition=attachment%3B+filename%3DUTF-8%27%27cnn_stories.tgz%3B+filename%3D%22cnn_stories.tgz%22%3B&Expires=1713763720&Policy=eyJTdGF0ZW1lbnQiOlt7IkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcxMzc2MzcyMH19LCJSZXNvdXJjZSI6Imh0dHBzOi8vY2RuLWxmcy5odWdnaW5nZmFjZS5jby9kYXRhc2V0cy9jY2R2L2Nubl9kYWlseW1haWwvZThmYmMwMDI3ZTU0ZTBhOTE2YWJkOWM5NjllYjM1ZjcwOGVkMTQ2N2Q3ZWY0ZTNiMTdhNTY3MzlkNjVjYjIwMD9yZXNwb25zZS1jb250ZW50LWRpc3Bvc2l0aW9uPSoifV19&Signature=pZ3ot5krXIXBDiyZR6c8oaijndL0nxBmrqC7mGxL25Fj8-lX5~h14QDfJTO1eI~tvdkTbzxU7bc7eP0HbQ0cP8FRxcZGVwavW5hIj~h-xyy~5rsywjmsudTDJwNpk103W1aHBPGM03FVXAGNPTYxa~e4fIPdYDRTREFuGCoQlMNLc4PlsRh0bvUKe1w1bsc9hmvOaD36xJoP-1jwHiL16OE1WbaB7qte-rBsUktXeMi1hNS~darFES46Das1vwqrlGl7b5IRyu5OIF6-t~dyMvufCFR4e0yUcNW7e3kuEzURiwOkT91N7evOu8DQkrNdagJgMuS078YvkxQZtgeZjQ__&Key-Pair-Id=KVTP0A1DKRTAX (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7359eb95c520>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution'))")))
It looks an issue of the dataset. We could not bring too many helps. You could try download the datasets manually and prevent downloading in the program.
/code/tensorrt_llm# python examples/llama/convert_checkpoint.py --model_dir /code/tensorrt_llm/Mixtral-8x7B-Instruct-v0.1/ --dtype float16 --output_dir /code/tensorrt_llm/examples/Mixtral-8x7B-Instruct-v0.1/ --int8_kv_cache --dataset-cache-dir . [TensorRT-LLM] TensorRT-LLM version: 0.10.0.dev2024041600 0.10.0.dev2024041600 Loading checkpoint shards: 100%|██████████████████████████████████████████████████████| 19/19 [00:14<00:00, 1.35it/s] [04/19/2024-05:28:34] Some parameters are on the meta device device because they were offloaded to the cpu. /usr/local/lib/python3.10/dist-packages/datasets/load.py:1461: FutureWarning: The repository for ccdv/cnn_dailymail contains custom code which must be executed to correctly load the dataset. You can inspect the repository content at https://hf.co/datasets/ccdv/cnn_dailymail You can avoid this message in future by passing the argument
main()
File "/code/tensorrt_llm/examples/llama/convert_checkpoint.py", line 438, in main
convert_and_save_hf(args)
File "/code/tensorrt_llm/examples/llama/convert_checkpoint.py", line 345, in convert_and_save_hf
LLaMAForCausalLM.quantize(args.model_dir,
File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/models/llama/model.py", line 363, in quantize
convert.quantize(
File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/models/llama/convert.py", line 1244, in quantize
act_range, llama_qkv_para, llama_smoother = smooth_quant(
File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/models/llama/convert.py", line 1037, in smooth_quant
dataset = load_dataset("ccdv/cnn_dailymail",
File "/usr/local/lib/python3.10/dist-packages/datasets/load.py", line 2582, in load_dataset
builder_instance.download_and_prepare(
File "/usr/local/lib/python3.10/dist-packages/datasets/builder.py", line 1005, in download_and_prepare
self._download_and_prepare(
File "/usr/local/lib/python3.10/dist-packages/datasets/builder.py", line 1767, in _download_and_prepare
super()._download_and_prepare(
File "/usr/local/lib/python3.10/dist-packages/datasets/builder.py", line 1078, in _download_and_prepare
split_generators = self._split_generators(dl_manager, *split_generators_kwargs)
File "/root/.cache/huggingface/modules/datasets_modules/datasets/ccdv--cnn_dailymail/0107f7388b5c6fae455a5661bcd134fc22da53ea75852027040d8d1e997f101f/cnn_dailymail.py", line 255, in _split_generators
dl_paths = dl_manager.download_and_extract(_DL_URLS)
File "/usr/local/lib/python3.10/dist-packages/datasets/download/download_manager.py", line 570, in download_and_extract
return self.extract(self.download(url_or_urls))
File "/usr/local/lib/python3.10/dist-packages/datasets/download/download_manager.py", line 434, in download
downloaded_path_or_paths = map_nested(
File "/usr/local/lib/python3.10/dist-packages/datasets/utils/py_utils.py", line 475, in map_nested
mapped = [
File "/usr/local/lib/python3.10/dist-packages/datasets/utils/py_utils.py", line 476, in
_single_map_nested((function, obj, types, None, True, None))
File "/usr/local/lib/python3.10/dist-packages/datasets/utils/py_utils.py", line 368, in _single_map_nested
return function(data_struct)
File "/usr/local/lib/python3.10/dist-packages/datasets/download/download_manager.py", line 459, in _download
out = cached_path(url_or_filename, download_config=download_config)
File "/usr/local/lib/python3.10/dist-packages/datasets/utils/file_utils.py", line 190, in cached_path
output_path = get_from_cache(
File "/usr/local/lib/python3.10/dist-packages/datasets/utils/file_utils.py", line 589, in get_from_cache
raise ConnectionError(f"Couldn't reach {url} ({repr(head_error)})")
ConnectionError: Couldn't reach https://huggingface.co/datasets/ccdv/cnn_dailymail/resolve/main/cnn_stories.tgz (ConnectionError(MaxRetryError("HTTPSConnectionPool(host='cdn-lfs.huggingface.co', port=443): Max retries exceeded with url: /datasets/ccdv/cnn_dailymail/e8fbc0027e54e0a916abd9c969eb35f708ed1467d7ef4e3b17a56739d65cb200?response-content-disposition=attachment%3B+filename %3DUTF-8%27%27cnn_stories.tgz%3B+filename%3D%22cnn_stories.tgz%22%3B&Expires=1713763720&Policy=eyJTdGF0ZW1lbnQiOlt7IkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcxMzc2MzcyMH19LCJSZXNvdXJjZSI6Imh0dHBzOi8vY2RuLWxmcy5odWdnaW5nZmFjZS5jby9kYXRhc2V0cy9jY2R2L2Nubl9kYWlseW1haWwvZThmYmMwMDI3ZTU0ZTBhOTE2YWJkOWM5NjllYjM1ZjcwOGVkMTQ2N2Q3ZWY0ZTNiMTdhNTY3MzlkNjVjYjIwMD9yZXNwb25zZS1jb250ZW50LWRpc3Bvc2l0aW9uPSoifV19&Signature=pZ3ot5krXIXBDiyZR6c8oaijndL0nxBmrqC7mGxL25Fj8-lX5~h14QDfJTO1eI~tvdkTbzxU7bc7eP0HbQ0cP8FRxcZGVwavW5hIj~h-xyy~5rsywjmsudTDJwNpk103W1aHBPGM03FVXAGNPTYxa~e4fIPdYDRTREFuGCoQlMNLc4PlsRh0bvUKe1w1bsc9hmvOaD36xJoP-1jwHiL16OE1WbaB7qte-rBsUktXeMi1hNS~darFES46Das1vwqrlGl7b5IRyu5OIF6-t~dyMvufCFR4e0yUcNW7e3kuEzURiwOkT91N7evOu8DQkrNdagJgMuS078YvkxQZtgeZjQ__&Key-Pair-Id=KVTP0A1DKRTAX (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7359eb95c520>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution'))")))
trust_remote_code=True
. Passingtrust_remote_code=True
will be mandatory to load this dataset from the next major release ofdatasets
. warnings.warn( Downloading builder script: 100%|████████████████████████████████████████████████| 9.27k/9.27k [00:00<00:00, 38.4MB/s] Downloading readme: 100%|████████████████████████████████████████████████████████| 13.9k/13.9k [00:00<00:00, 54.7MB/s] Traceback (most recent call last): File "/code/tensorrt_llm/examples/llama/convert_checkpoint.py", line 446, inIts not able to Download the https://huggingface.co/datasets/ccdv/cnn_dailymail/resolve/main/cnn_stories.tgz file from within the docker.