I have run main.py,
--tasks
piqa,arc_easy,arc_challenge,boolq,hellaswag,winogrande
Due to the inability of the dataset to be loaded, I can only download the dataset from Hugging and load it locally. Then I found that there was no boolq file in im_eval.tasks.
What I was thinking was to find the. py file containing the dataset in im_eval. tasks and change the variable: DATASET-PATH to an absolute path. But it does not include the boolq file. Dear author, how do I need to change the settings for locally loading other tasks for Selected Tasks: ['winogrande ',' boolq ',' hellaswag ',' arc_challenge ',' piqa ',' arc_easy '].
Selected Tasks: ['winogrande', 'boolq', 'hellaswag', 'arc_challenge', 'piqa', 'arc_easy']
Downloading data: 100%|██████████| 2.06M/2.06M [00:02<00:00, 978kB/s]
Downloading data: 100%|██████████| 118k/118k [00:00<00:00, 145kB/s]
Downloading data: 100%|██████████| 85.9k/85.9k [00:00<00:00, 144kB/s]
Generating train split: 100%|██████████| 40398/40398 [00:00<00:00, 378782.41 examples/s]
Generating test split: 100%|██████████| 1767/1767 [00:00<00:00, 269885.84 examples/s]
Generating validation split: 100%|██████████| 1267/1267 [00:00<00:00, 234611.42 examples/s]
Downloading data: 100%|██████████| 3.85M/3.85M [00:01<00:00, 2.43MB/s]
Downloading data: 100%|██████████| 1.31M/1.31M [00:00<00:00, 2.21MB/s]
Downloading data: 100%|██████████| 1.31M/1.31M [00:00<00:00, 2.04MB/s]
Generating train split: 100%|██████████| 9427/9427 [00:00<00:00, 121299.71 examples/s]
Generating validation split: 100%|██████████| 3270/3270 [00:00<00:00, 136654.95 examples/s]
Generating test split: 100%|██████████| 3245/3245 [00:00<00:00, 133984.19 examples/s]
/home/sam/anaconda3/envs/omniquant/lib/python3.10/site-packages/datasets/load.py:1478: FutureWarning: The repository for hellaswag contains custom code which must be executed to correctly load the dataset. You can inspect the repository content at https://hf.co/datasets/hellaswag
You can avoid this message in future by passing the argument trust_remote_code=True.
Passing trust_remote_code=True will be mandatory to load this dataset from the next major release of datasets.
warnings.warn(
Downloading builder script: 100%|██████████| 4.36k/4.36k [00:00<00:00, 15.3MB/s]
Downloading metadata: 100%|██████████| 2.53k/2.53k [00:00<00:00, 8.71MB/s]
Downloading readme: 100%|██████████| 6.84k/6.84k [00:00<00:00, 12.7MB/s]
Downloading data: 47.5MB [00:03, 12.6MB/s]
Traceback (most recent call last):
File "/home/sam/anaconda3/envs/omniquant/lib/python3.10/site-packages/urllib3/connectionpool.py", line 467, in _make_request
self._validate_conn(conn)
File "/home/sam/anaconda3/envs/omniquant/lib/python3.10/site-packages/urllib3/connectionpool.py", line 1099, in _validate_conn
conn.connect()
File "/home/sam/anaconda3/envs/omniquant/lib/python3.10/site-packages/urllib3/connection.py", line 653, in connect
sock_and_verified = _ssl_wrap_socket_and_match_hostname(
File "/home/sam/anaconda3/envs/omniquant/lib/python3.10/site-packages/urllib3/connection.py", line 806, in _ssl_wrap_socket_and_match_hostname
ssl_sock = ssl_wrapsocket(
File "/home/sam/anaconda3/envs/omniquant/lib/python3.10/site-packages/urllib3/util/ssl.py", line 465, in ssl_wrap_socket
ssl_sock = _ssl_wrap_socket_impl(sock, context, tls_in_tls, serverhostname)
File "/home/sam/anaconda3/envs/omniquant/lib/python3.10/site-packages/urllib3/util/ssl.py", line 509, in _ssl_wrap_socket_impl
return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
File "/home/sam/anaconda3/envs/omniquant/lib/python3.10/ssl.py", line 513, in wrap_socket
return self.sslsocket_class._create(
File "/home/sam/anaconda3/envs/omniquant/lib/python3.10/ssl.py", line 1104, in _create
self.do_handshake()
File "/home/sam/anaconda3/envs/omniquant/lib/python3.10/ssl.py", line 1375, in do_handshake
self._sslobj.do_handshake()
ssl.SSLEOFError: [SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/sam/anaconda3/envs/omniquant/lib/python3.10/site-packages/urllib3/connectionpool.py", line 793, in urlopen
response = self._make_request(
File "/home/sam/anaconda3/envs/omniquant/lib/python3.10/site-packages/urllib3/connectionpool.py", line 491, in _make_request
raise new_e
urllib3.exceptions.SSLError: [SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/sam/anaconda3/envs/omniquant/lib/python3.10/site-packages/requests/adapters.py", line 486, in send
resp = conn.urlopen(
File "/home/sam/anaconda3/envs/omniquant/lib/python3.10/site-packages/urllib3/connectionpool.py", line 847, in urlopen
retries = retries.increment(
File "/home/sam/anaconda3/envs/omniquant/lib/python3.10/site-packages/urllib3/util/retry.py", line 515, in increment
raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type]
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='raw.githubusercontent.com', port=443): Max retries exceeded with url: /rowanz/hellaswag/master/data/hellaswag_test.jsonl (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)')))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/sam/Doctorproject/OmniQuant-main/main.py", line 384, in
main()
File "/home/sam/Doctorproject/OmniQuant-main/main.py", line 379, in main
evaluate(lm, args,logger)
File "/home/sam/anaconda3/envs/omniquant/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, *kwargs)
File "/home/sam/Doctorproject/OmniQuant-main/main.py", line 148, in evaluate
t_results = evaluator.simple_evaluate(
File "/home/sam/Doctorproject/OmniQuant-main/lm_eval/utils.py", line 160, in _wrapper
return fn(args, kwargs)
File "/home/sam/Doctorproject/OmniQuant-main/lm_eval/evaluator.py", line 66, in simple_evaluate
task_dict = lm_eval.tasks.get_task_dict(task_names)
File "/home/sam/Doctorproject/OmniQuant-main/lm_eval/tasks/init.py", line 342, in get_task_dict
task_name_dict = {
File "/home/sam/Doctorproject/OmniQuant-main/lm_eval/tasks/init.py", line 343, in
task_name: get_task(task_name)()
File "/home/sam/Doctorproject/OmniQuant-main/lm_eval/base.py", line 412, in init
self.download(data_dir, cache_dir, download_mode)
File "/home/sam/Doctorproject/OmniQuant-main/lm_eval/base.py", line 441, in download
self.dataset = datasets.load_dataset(
File "/home/sam/anaconda3/envs/omniquant/lib/python3.10/site-packages/datasets/load.py", line 2598, in load_dataset
builder_instance.download_and_prepare(
File "/home/sam/anaconda3/envs/omniquant/lib/python3.10/site-packages/datasets/builder.py", line 1021, in download_and_prepare
self._download_and_prepare(
File "/home/sam/anaconda3/envs/omniquant/lib/python3.10/site-packages/datasets/builder.py", line 1783, in _download_and_prepare
super()._download_and_prepare(
File "/home/sam/anaconda3/envs/omniquant/lib/python3.10/site-packages/datasets/builder.py", line 1094, in _download_and_prepare
split_generators = self._split_generators(dl_manager, split_generators_kwargs)
File "/home/sam/.cache/huggingface/modules/datasets_modules/datasets/hellaswag/512a66dd8b1b1643ab4a48aa4f150d04c91680da6a4096498a5e5f799623d5ae/hellaswag.py", line 72, in _split_generators
dl_dir = dl_manager.download_and_extract(urls_to_download)
File "/home/sam/anaconda3/envs/omniquant/lib/python3.10/site-packages/datasets/download/download_manager.py", line 570, in download_and_extract
return self.extract(self.download(url_or_urls))
File "/home/sam/anaconda3/envs/omniquant/lib/python3.10/site-packages/datasets/download/download_manager.py", line 434, in download
downloaded_path_or_paths = map_nested(
File "/home/sam/anaconda3/envs/omniquant/lib/python3.10/site-packages/datasets/utils/py_utils.py", line 477, in map_nested
mapped = [
File "/home/sam/anaconda3/envs/omniquant/lib/python3.10/site-packages/datasets/utils/py_utils.py", line 478, in
_single_map_nested((function, obj, types, None, True, None))
File "/home/sam/anaconda3/envs/omniquant/lib/python3.10/site-packages/datasets/utils/py_utils.py", line 370, in _single_map_nested
return function(data_struct)
File "/home/sam/anaconda3/envs/omniquant/lib/python3.10/site-packages/datasets/download/download_manager.py", line 459, in _download
out = cached_path(url_or_filename, download_config=download_config)
File "/home/sam/anaconda3/envs/omniquant/lib/python3.10/site-packages/datasets/utils/file_utils.py", line 190, in cached_path
output_path = get_from_cache(
File "/home/sam/anaconda3/envs/omniquant/lib/python3.10/site-packages/datasets/utils/file_utils.py", line 634, in get_from_cache
http_get(
File "/home/sam/anaconda3/envs/omniquant/lib/python3.10/site-packages/datasets/utils/file_utils.py", line 382, in http_get
response = _request_with_retry(
File "/home/sam/anaconda3/envs/omniquant/lib/python3.10/site-packages/datasets/utils/file_utils.py", line 308, in _request_with_retry
raise err
File "/home/sam/anaconda3/envs/omniquant/lib/python3.10/site-packages/datasets/utils/file_utils.py", line 304, in _request_with_retry
response = requests.request(method=method.upper(), url=url, timeout=timeout, params)
File "/home/sam/anaconda3/envs/omniquant/lib/python3.10/site-packages/requests/api.py", line 59, in request
return session.request(method=method, url=url, kwargs)
File "/home/sam/anaconda3/envs/omniquant/lib/python3.10/site-packages/requests/sessions.py", line 589, in request
resp = self.send(prep, send_kwargs)
File "/home/sam/anaconda3/envs/omniquant/lib/python3.10/site-packages/requests/sessions.py", line 703, in send
r = adapter.send(request, kwargs)
File "/home/sam/anaconda3/envs/omniquant/lib/python3.10/site-packages/requests/adapters.py", line 517, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='raw.githubusercontent.com', port=443): Max retries exceeded with url: /rowanz/hellaswag/master/data/hellaswag_test.jsonl (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)')))
hsb1995
commented
6 months ago
I have run main.py, --tasks piqa,arc_easy,arc_challenge,boolq,hellaswag,winogrande
Due to the inability of the dataset to be loaded, I can only download the dataset from Hugging and load it locally. Then I found that there was no boolq file in im_eval.tasks.
What I was thinking was to find the. py file containing the dataset in im_eval. tasks and change the variable: DATASET-PATH to an absolute path. But it does not include the boolq file. Dear author, how do I need to change the settings for locally loading other tasks for Selected Tasks: ['winogrande ',' boolq ',' hellaswag ',' arc_challenge ',' piqa ',' arc_easy '].
Selected Tasks: ['winogrande', 'boolq', 'hellaswag', 'arc_challenge', 'piqa', 'arc_easy'] Downloading data: 100%|██████████| 2.06M/2.06M [00:02<00:00, 978kB/s] Downloading data: 100%|██████████| 118k/118k [00:00<00:00, 145kB/s] Downloading data: 100%|██████████| 85.9k/85.9k [00:00<00:00, 144kB/s] Generating train split: 100%|██████████| 40398/40398 [00:00<00:00, 378782.41 examples/s] Generating test split: 100%|██████████| 1767/1767 [00:00<00:00, 269885.84 examples/s] Generating validation split: 100%|██████████| 1267/1267 [00:00<00:00, 234611.42 examples/s] Downloading data: 100%|██████████| 3.85M/3.85M [00:01<00:00, 2.43MB/s] Downloading data: 100%|██████████| 1.31M/1.31M [00:00<00:00, 2.21MB/s] Downloading data: 100%|██████████| 1.31M/1.31M [00:00<00:00, 2.04MB/s] Generating train split: 100%|██████████| 9427/9427 [00:00<00:00, 121299.71 examples/s] Generating validation split: 100%|██████████| 3270/3270 [00:00<00:00, 136654.95 examples/s] Generating test split: 100%|██████████| 3245/3245 [00:00<00:00, 133984.19 examples/s] /home/sam/anaconda3/envs/omniquant/lib/python3.10/site-packages/datasets/load.py:1478: FutureWarning: The repository for hellaswag contains custom code which must be executed to correctly load the dataset. You can inspect the repository content at https://hf.co/datasets/hellaswag You can avoid this message in future by passing the argument
trust_remote_code=True. Passingtrust_remote_code=Truewill be mandatory to load this dataset from the next major release ofdatasets. warnings.warn( Downloading builder script: 100%|██████████| 4.36k/4.36k [00:00<00:00, 15.3MB/s] Downloading metadata: 100%|██████████| 2.53k/2.53k [00:00<00:00, 8.71MB/s] Downloading readme: 100%|██████████| 6.84k/6.84k [00:00<00:00, 12.7MB/s] Downloading data: 47.5MB [00:03, 12.6MB/s] Traceback (most recent call last): File "/home/sam/anaconda3/envs/omniquant/lib/python3.10/site-packages/urllib3/connectionpool.py", line 467, in _make_request self._validate_conn(conn) File "/home/sam/anaconda3/envs/omniquant/lib/python3.10/site-packages/urllib3/connectionpool.py", line 1099, in _validate_conn conn.connect() File "/home/sam/anaconda3/envs/omniquant/lib/python3.10/site-packages/urllib3/connection.py", line 653, in connect sock_and_verified = _ssl_wrap_socket_and_match_hostname( File "/home/sam/anaconda3/envs/omniquant/lib/python3.10/site-packages/urllib3/connection.py", line 806, in _ssl_wrap_socket_and_match_hostname ssl_sock = ssl_wrapsocket( File "/home/sam/anaconda3/envs/omniquant/lib/python3.10/site-packages/urllib3/util/ssl.py", line 465, in ssl_wrap_socket ssl_sock = _ssl_wrap_socket_impl(sock, context, tls_in_tls, serverhostname) File "/home/sam/anaconda3/envs/omniquant/lib/python3.10/site-packages/urllib3/util/ssl.py", line 509, in _ssl_wrap_socket_impl return ssl_context.wrap_socket(sock, server_hostname=server_hostname) File "/home/sam/anaconda3/envs/omniquant/lib/python3.10/ssl.py", line 513, in wrap_socket return self.sslsocket_class._create( File "/home/sam/anaconda3/envs/omniquant/lib/python3.10/ssl.py", line 1104, in _create self.do_handshake() File "/home/sam/anaconda3/envs/omniquant/lib/python3.10/ssl.py", line 1375, in do_handshake self._sslobj.do_handshake() ssl.SSLEOFError: [SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "/home/sam/anaconda3/envs/omniquant/lib/python3.10/site-packages/urllib3/connectionpool.py", line 793, in urlopen response = self._make_request( File "/home/sam/anaconda3/envs/omniquant/lib/python3.10/site-packages/urllib3/connectionpool.py", line 491, in _make_request raise new_e urllib3.exceptions.SSLError: [SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)
The above exception was the direct cause of the following exception:
Traceback (most recent call last): File "/home/sam/anaconda3/envs/omniquant/lib/python3.10/site-packages/requests/adapters.py", line 486, in send resp = conn.urlopen( File "/home/sam/anaconda3/envs/omniquant/lib/python3.10/site-packages/urllib3/connectionpool.py", line 847, in urlopen retries = retries.increment( File "/home/sam/anaconda3/envs/omniquant/lib/python3.10/site-packages/urllib3/util/retry.py", line 515, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='raw.githubusercontent.com', port=443): Max retries exceeded with url: /rowanz/hellaswag/master/data/hellaswag_test.jsonl (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)')))
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "/home/sam/Doctorproject/OmniQuant-main/main.py", line 384, in
main()
File "/home/sam/Doctorproject/OmniQuant-main/main.py", line 379, in main
evaluate(lm, args,logger)
File "/home/sam/anaconda3/envs/omniquant/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, *kwargs)
File "/home/sam/Doctorproject/OmniQuant-main/main.py", line 148, in evaluate
t_results = evaluator.simple_evaluate(
File "/home/sam/Doctorproject/OmniQuant-main/lm_eval/utils.py", line 160, in _wrapper
return fn(args, kwargs)
File "/home/sam/Doctorproject/OmniQuant-main/lm_eval/evaluator.py", line 66, in simple_evaluate
task_dict = lm_eval.tasks.get_task_dict(task_names)
File "/home/sam/Doctorproject/OmniQuant-main/lm_eval/tasks/init.py", line 342, in get_task_dict
task_name_dict = {
File "/home/sam/Doctorproject/OmniQuant-main/lm_eval/tasks/init.py", line 343, in
task_name: get_task(task_name)()
File "/home/sam/Doctorproject/OmniQuant-main/lm_eval/base.py", line 412, in init
self.download(data_dir, cache_dir, download_mode)
File "/home/sam/Doctorproject/OmniQuant-main/lm_eval/base.py", line 441, in download
self.dataset = datasets.load_dataset(
File "/home/sam/anaconda3/envs/omniquant/lib/python3.10/site-packages/datasets/load.py", line 2598, in load_dataset
builder_instance.download_and_prepare(
File "/home/sam/anaconda3/envs/omniquant/lib/python3.10/site-packages/datasets/builder.py", line 1021, in download_and_prepare
self._download_and_prepare(
File "/home/sam/anaconda3/envs/omniquant/lib/python3.10/site-packages/datasets/builder.py", line 1783, in _download_and_prepare
super()._download_and_prepare(
File "/home/sam/anaconda3/envs/omniquant/lib/python3.10/site-packages/datasets/builder.py", line 1094, in _download_and_prepare
split_generators = self._split_generators(dl_manager, split_generators_kwargs)
File "/home/sam/.cache/huggingface/modules/datasets_modules/datasets/hellaswag/512a66dd8b1b1643ab4a48aa4f150d04c91680da6a4096498a5e5f799623d5ae/hellaswag.py", line 72, in _split_generators
dl_dir = dl_manager.download_and_extract(urls_to_download)
File "/home/sam/anaconda3/envs/omniquant/lib/python3.10/site-packages/datasets/download/download_manager.py", line 570, in download_and_extract
return self.extract(self.download(url_or_urls))
File "/home/sam/anaconda3/envs/omniquant/lib/python3.10/site-packages/datasets/download/download_manager.py", line 434, in download
downloaded_path_or_paths = map_nested(
File "/home/sam/anaconda3/envs/omniquant/lib/python3.10/site-packages/datasets/utils/py_utils.py", line 477, in map_nested
mapped = [
File "/home/sam/anaconda3/envs/omniquant/lib/python3.10/site-packages/datasets/utils/py_utils.py", line 478, in
_single_map_nested((function, obj, types, None, True, None))
File "/home/sam/anaconda3/envs/omniquant/lib/python3.10/site-packages/datasets/utils/py_utils.py", line 370, in _single_map_nested
return function(data_struct)
File "/home/sam/anaconda3/envs/omniquant/lib/python3.10/site-packages/datasets/download/download_manager.py", line 459, in _download
out = cached_path(url_or_filename, download_config=download_config)
File "/home/sam/anaconda3/envs/omniquant/lib/python3.10/site-packages/datasets/utils/file_utils.py", line 190, in cached_path
output_path = get_from_cache(
File "/home/sam/anaconda3/envs/omniquant/lib/python3.10/site-packages/datasets/utils/file_utils.py", line 634, in get_from_cache
http_get(
File "/home/sam/anaconda3/envs/omniquant/lib/python3.10/site-packages/datasets/utils/file_utils.py", line 382, in http_get
response = _request_with_retry(
File "/home/sam/anaconda3/envs/omniquant/lib/python3.10/site-packages/datasets/utils/file_utils.py", line 308, in _request_with_retry
raise err
File "/home/sam/anaconda3/envs/omniquant/lib/python3.10/site-packages/datasets/utils/file_utils.py", line 304, in _request_with_retry
response = requests.request(method=method.upper(), url=url, timeout=timeout, params)
File "/home/sam/anaconda3/envs/omniquant/lib/python3.10/site-packages/requests/api.py", line 59, in request
return session.request(method=method, url=url, kwargs)
File "/home/sam/anaconda3/envs/omniquant/lib/python3.10/site-packages/requests/sessions.py", line 589, in request
resp = self.send(prep, send_kwargs)
File "/home/sam/anaconda3/envs/omniquant/lib/python3.10/site-packages/requests/sessions.py", line 703, in send
r = adapter.send(request, kwargs)
File "/home/sam/anaconda3/envs/omniquant/lib/python3.10/site-packages/requests/adapters.py", line 517, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='raw.githubusercontent.com', port=443): Max retries exceeded with url: /rowanz/hellaswag/master/data/hellaswag_test.jsonl (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)')))