Issue while convert Mixtral-8x7B to tensorrt format on 3090

[ChandanVerma]

/code/tensorrt_llm# python examples/llama/convert_checkpoint.py --model_dir /code/tensorrt_llm/Mixtral-8x7B-Instruct-v0.1/ --dtype float16 --output_dir /code/tensorrt_llm/examples/Mixtral-8x7B-Instruct-v0.1/ --int8_kv_cache --dataset-cache-dir . [TensorRT-LLM] TensorRT-LLM version: 0.10.0.dev2024041600 0.10.0.dev2024041600 Loading checkpoint shards: 100%|██████████████████████████████████████████████████████| 19/19 [00:14<00:00, 1.35it/s] [04/19/2024-05:28:34] Some parameters are on the meta device device because they were offloaded to the cpu. /usr/local/lib/python3.10/dist-packages/datasets/load.py:1461: FutureWarning: The repository for ccdv/cnn_dailymail contains custom code which must be executed to correctly load the dataset. You can inspect the repository content at https://hf.co/datasets/ccdv/cnn_dailymail You can avoid this message in future by passing the argument trust_remote_code=True. Passing trust_remote_code=True will be mandatory to load this dataset from the next major release of datasets. warnings.warn( Downloading builder script: 100%|████████████████████████████████████████████████| 9.27k/9.27k [00:00<00:00, 38.4MB/s] Downloading readme: 100%|████████████████████████████████████████████████████████| 13.9k/13.9k [00:00<00:00, 54.7MB/s] Traceback (most recent call last): File "/code/tensorrt_llm/examples/llama/convert_checkpoint.py", line 446, in main() File "/code/tensorrt_llm/examples/llama/convert_checkpoint.py", line 438, in main convert_and_save_hf(args) File "/code/tensorrt_llm/examples/llama/convert_checkpoint.py", line 345, in convert_and_save_hf LLaMAForCausalLM.quantize(args.model_dir, File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/models/llama/model.py", line 363, in quantize convert.quantize( File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/models/llama/convert.py", line 1244, in quantize act_range, llama_qkv_para, llama_smoother = smooth_quant( File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/models/llama/convert.py", line 1037, in smooth_quant dataset = load_dataset("ccdv/cnn_dailymail", File "/usr/local/lib/python3.10/dist-packages/datasets/load.py", line 2582, in load_dataset builder_instance.download_and_prepare( File "/usr/local/lib/python3.10/dist-packages/datasets/builder.py", line 1005, in download_and_prepare self._download_and_prepare( File "/usr/local/lib/python3.10/dist-packages/datasets/builder.py", line 1767, in _download_and_prepare super()._download_and_prepare( File "/usr/local/lib/python3.10/dist-packages/datasets/builder.py", line 1078, in _download_and_prepare split_generators = self._split_generators(dl_manager, *split_generators_kwargs) File "/root/.cache/huggingface/modules/datasets_modules/datasets/ccdv--cnn_dailymail/0107f7388b5c6fae455a5661bcd134fc22da53ea75852027040d8d1e997f101f/cnn_dailymail.py", line 255, in _split_generators dl_paths = dl_manager.download_and_extract(_DL_URLS) File "/usr/local/lib/python3.10/dist-packages/datasets/download/download_manager.py", line 570, in download_and_extract return self.extract(self.download(url_or_urls)) File "/usr/local/lib/python3.10/dist-packages/datasets/download/download_manager.py", line 434, in download downloaded_path_or_paths = map_nested( File "/usr/local/lib/python3.10/dist-packages/datasets/utils/py_utils.py", line 475, in map_nested mapped = [ File "/usr/local/lib/python3.10/dist-packages/datasets/utils/py_utils.py", line 476, in _single_map_nested((function, obj, types, None, True, None)) File "/usr/local/lib/python3.10/dist-packages/datasets/utils/py_utils.py", line 368, in _single_map_nested return function(data_struct) File "/usr/local/lib/python3.10/dist-packages/datasets/download/download_manager.py", line 459, in _download out = cached_path(url_or_filename, download_config=download_config) File "/usr/local/lib/python3.10/dist-packages/datasets/utils/file_utils.py", line 190, in cached_path output_path = get_from_cache( File "/usr/local/lib/python3.10/dist-packages/datasets/utils/file_utils.py", line 589, in get_from_cache raise ConnectionError(f"Couldn't reach {url} ({repr(head_error)})") ConnectionError: Couldn't reach https://huggingface.co/datasets/ccdv/cnn_dailymail/resolve/main/cnn_stories.tgz (ConnectionError(MaxRetryError("HTTPSConnectionPool(host='cdn-lfs.huggingface.co', port=443): Max retries exceeded with url: /datasets/ccdv/cnn_dailymail/e8fbc0027e54e0a916abd9c969eb35f708ed1467d7ef4e3b17a56739d65cb200?response-content-disposition=attachment%3B+filename%3DUTF-8%27%27cnn_stories.tgz%3B+filename%3D%22cnn_stories.tgz%22%3B&Expires=1713763720&Policy=eyJTdGF0ZW1lbnQiOlt7IkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcxMzc2MzcyMH19LCJSZXNvdXJjZSI6Imh0dHBzOi8vY2RuLWxmcy5odWdnaW5nZmFjZS5jby9kYXRhc2V0cy9jY2R2L2Nubl9kYWlseW1haWwvZThmYmMwMDI3ZTU0ZTBhOTE2YWJkOWM5NjllYjM1ZjcwOGVkMTQ2N2Q3ZWY0ZTNiMTdhNTY3MzlkNjVjYjIwMD9yZXNwb25zZS1jb250ZW50LWRpc3Bvc2l0aW9uPSoifV19&Signature=pZ3ot5krXIXBDiyZR6c8oaijndL0nxBmrqC7mGxL25Fj8-lX5~h14QDfJTO1eI~tvdkTbzxU7bc7eP0HbQ0cP8FRxcZGVwavW5hIj~h-xyy~5rsywjmsudTDJwNpk103W1aHBPGM03FVXAGNPTYxa~e4fIPdYDRTREFuGCoQlMNLc4PlsRh0bvUKe1w1bsc9hmvOaD36xJoP-1jwHiL16OE1WbaB7qte-rBsUktXeMi1hNS~darFES46Das1vwqrlGl7b5IRyu5OIF6-t~dyMvufCFR4e0yUcNW7e3kuEzURiwOkT91N7evOu8DQkrNdagJgMuS078YvkxQZtgeZjQ__&Key-Pair-Id=KVTP0A1DKRTAX (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7359eb95c520>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution'))")))

Its not able to Download the https://huggingface.co/datasets/ccdv/cnn_dailymail/resolve/main/cnn_stories.tgz file from within the docker.

[byshiue]

It looks an issue of the dataset. We could not bring too many helps. You could try download the datasets manually and prevent downloading in the program.