Closed hhy150 closed 2 months ago
我们团队成员两天前在新机器上顺利复现了该模型,所以应该不是模型地址的原因,可以尝试排查一下网络?
你好哇,我直接搜索也是报这个错呢,我看前面获取数据貌似也是可以的。
然后我的整个报错信息是这样的:
2024-06-13 22:32:41,955 - modelscope - INFO - PyTorch version 1.13.1 Found.
2024-06-13 22:32:41,955 - modelscope - INFO - Loading ast index from /data2/hhy/.cache/modelscope/ast_indexer
2024-06-13 22:32:41,955 - modelscope - INFO - No valid ast index found from /data2/hhy/.cache/modelscope/ast_indexer, generating ast index from prebuilt!
2024-06-13 22:32:42,040 - modelscope - INFO - Loading done! Current index file version is 1.7.1, with md5 22229ed48df4c136e5165c03566fd2b7 and a total number of 861 components indexed
2024-06-13 22:32:44,536 - modelscope - WARNING - The reference has been Deprecated in modelscope v1.4.0+, please use from modelscope.msdatasets.dataset_cls.custom_datasets import TorchCustomDataset
2024-06-13 22:32:44,747 - INFO - adaseq.data.dataset_manager - Will use a custom loading script: /data2/hhy/MNER/adaseq/adaseq/data/dataset_builders/named_entity_recognition_dataset_builder.py
Downloading data: 525kB [00:00, 7.11MB/s]
Generating train split: 4000 examples [00:00, 7265.71 examples/s]
Generating valid split: 1000 examples [00:00, 13259.06 examples/s]
Generating test split: 3257 examples [00:00, 13454.56 examples/s]
2024-06-13 22:32:47,306 - INFO - adaseq.data.datasetmanager - First sample in train set: {'id': '0', 'tokens': ['RT', '@JayKenMinaj', '', ':', 'Me', 'outside', 'of', 'where', 'George', 'Zimmerman', 'got', 'shot', 'at', '.', 'You', 'know', 'God', 'is', 'so', 'good', '.', 'http://t.co/Z3neVBQ7vF'], 'spans': [{'start': 8, 'end': 10, 'type': 'PER'}], 'mask': [True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True]}
Counting labels by count_span_labels: 100%|████████████████████████████████████| 4000/4000 [00:00<00:00, 10238.49 examples/s]
Counting labels by count_span_labels: 100%|████████████████████████████████████| 1000/1000 [00:00<00:00, 10096.76 examples/s]
2024-06-13 22:32:47,798 - INFO - adaseq.data.preprocessors.sequence_labeling_preprocessor - label_to_id: {'O': 0, 'B-LOC': 1, 'I-LOC': 2, 'E-LOC': 3, 'S-LOC': 4, 'B-ORG': 5, 'I-ORG': 6, 'E-ORG': 7, 'S-ORG': 8, 'B-OTHER': 9, 'I-OTHER': 10, 'E-OTHER': 11, 'S-OTHER': 12, 'B-PER': 13, 'I-PER': 14, 'E-PER': 15, 'S-PER': 16}
2024-06-13 22:33:27,979 - ERROR - modelscope - Authentication token does not exist, failed to access model xlm-roberta-large which may not exist or may be private. Please login first.
2024-06-13 22:33:27,979 - ERROR - modelscope - Response details: {'Code': 10010205001, 'Message': '获取模型信息失败,信息:record not found', 'RequestId': '5cf2843a-ad38-4792-b054-4815881ce4d8', 'Success': False}
Traceback (most recent call last): File "/data2/hhy/anaconda3/envs/mner/lib/python3.8/site-packages/urllib3/connection.py", line 198, in _new_conn sock = connection.create_connection( File "/data2/hhy/anaconda3/envs/mner/lib/python3.8/site-packages/urllib3/util/connection.py", line 85, in create_connection raise err File "/data2/hhy/anaconda3/envs/mner/lib/python3.8/site-packages/urllib3/util/connection.py", line 73, in create_connection sock.connect(sa) socket.timeout: timed out
The above exception was the direct cause of the following exception:
Traceback (most recent call last): File "/data2/hhy/anaconda3/envs/mner/lib/python3.8/site-packages/urllib3/connectionpool.py", line 793, in urlopen response = self._make_request( File "/data2/hhy/anaconda3/envs/mner/lib/python3.8/site-packages/urllib3/connectionpool.py", line 491, in _make_request raise new_e File "/data2/hhy/anaconda3/envs/mner/lib/python3.8/site-packages/urllib3/connectionpool.py", line 467, in _make_request self._validate_conn(conn) File "/data2/hhy/anaconda3/envs/mner/lib/python3.8/site-packages/urllib3/connectionpool.py", line 1099, in _validate_conn conn.connect() File "/data2/hhy/anaconda3/envs/mner/lib/python3.8/site-packages/urllib3/connection.py", line 616, in connect self.sock = sock = self._new_conn() File "/data2/hhy/anaconda3/envs/mner/lib/python3.8/site-packages/urllib3/connection.py", line 207, in _new_conn raise ConnectTimeoutError( urllib3.exceptions.ConnectTimeoutError: (<urllib3.connection.HTTPSConnection object at 0x7f5b1a8a8af0>, 'Connection to huggingface.co timed out. (connect timeout=10)')
The above exception was the direct cause of the following exception:
Traceback (most recent call last): File "/data2/hhy/anaconda3/envs/mner/lib/python3.8/site-packages/requests/adapters.py", line 589, in send resp = conn.urlopen( File "/data2/hhy/anaconda3/envs/mner/lib/python3.8/site-packages/urllib3/connectionpool.py", line 847, in urlopen retries = retries.increment( File "/data2/hhy/anaconda3/envs/mner/lib/python3.8/site-packages/urllib3/util/retry.py", line 515, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='huggingface.co', port=443): Max retries exceeded with url: /xlm-roberta-large/resolve/main/config.json (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7f5b1a8a8af0>, 'Connection to huggingface.co timed out. (connect timeout=10)'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "/data2/hhy/anaconda3/envs/mner/lib/python3.8/site-packages/huggingface_hub/file_download.py", line 1722, in _get_metadata_or_catch_error metadata = get_hf_file_metadata(url=url, proxies=proxies, timeout=etag_timeout, headers=headers) File "/data2/hhy/anaconda3/envs/mner/lib/python3.8/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn return fn(args, kwargs) File "/data2/hhy/anaconda3/envs/mner/lib/python3.8/site-packages/huggingface_hub/file_download.py", line 1645, in get_hf_file_metadata r = _request_wrapper( File "/data2/hhy/anaconda3/envs/mner/lib/python3.8/site-packages/huggingface_hub/file_download.py", line 372, in _request_wrapper response = _request_wrapper( File "/data2/hhy/anaconda3/envs/mner/lib/python3.8/site-packages/huggingface_hub/file_download.py", line 395, in _request_wrapper response = get_session().request(method=method, url=url, params) File "/data2/hhy/anaconda3/envs/mner/lib/python3.8/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, send_kwargs) File "/data2/hhy/anaconda3/envs/mner/lib/python3.8/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, kwargs) File "/data2/hhy/anaconda3/envs/mner/lib/python3.8/site-packages/huggingface_hub/utils/_http.py", line 66, in send return super().send(request, args, **kwargs) File "/data2/hhy/anaconda3/envs/mner/lib/python3.8/site-packages/requests/adapters.py", line 610, in send raise ConnectTimeout(e, request=request) requests.exceptions.ConnectTimeout: (MaxRetryError("HTTPSConnectionPool(host='huggingface.co', port=443): Max retries exceeded with url: /xlm-roberta-large/resolve/main/config.json (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7f5b1a8a8af0>, 'Connection to huggingface.co timed out. (connect timeout=10)'))"), '(Request ID: 95b7d315-4d56-4dcc-9d8f-a9968d366891)')
The above exception was the direct cause of the following exception:
Traceback (most recent call last): File "/data2/hhy/anaconda3/envs/mner/lib/python3.8/site-packages/transformers/utils/hub.py", line 417, in cached_file resolved_file = hf_hub_download( File "/data2/hhy/anaconda3/envs/mner/lib/python3.8/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn return fn(*args, **kwargs) File "/data2/hhy/anaconda3/envs/mner/lib/python3.8/site-packages/huggingface_hub/file_download.py", line 1221, in hf_hub_download return _hf_hub_download_to_cache_dir( File "/data2/hhy/anaconda3/envs/mner/lib/python3.8/site-packages/huggingface_hub/file_download.py", line 1325, in _hf_hub_download_to_cache_dir _raise_on_head_call_error(head_call_error, force_download, local_files_only) File "/data2/hhy/anaconda3/envs/mner/lib/python3.8/site-packages/huggingface_hub/file_download.py", line 1826, in _raise_on_head_call_error raise LocalEntryNotFoundError( huggingface_hub.utils._errors.LocalEntryNotFoundError: An error happened while trying to locate the file on the Hub and we cannot find the requested files in the local cache. Please check your connection and try again or make sure your Internet connection is on.
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "/data2/hhy/MNER/adaseq/adaseq/data/tokenizer.py", line 19, in build_tokenizer return AutoTokenizer.from_pretrained(model_name_or_path, use_fast=use_fast, kwargs) File "/data2/hhy/anaconda3/envs/mner/lib/python3.8/site-packages/transformers/models/auto/tokenization_auto.py", line 658, in from_pretrained config = AutoConfig.from_pretrained( File "/data2/hhy/anaconda3/envs/mner/lib/python3.8/site-packages/transformers/models/auto/configuration_auto.py", line 944, in from_pretrained config_dict, unused_kwargs = PretrainedConfig.get_config_dict(pretrained_model_name_or_path, kwargs) File "/data2/hhy/anaconda3/envs/mner/lib/python3.8/site-packages/transformers/configuration_utils.py", line 574, in get_config_dict config_dict, kwargs = cls._get_config_dict(pretrained_model_name_or_path, **kwargs) File "/data2/hhy/anaconda3/envs/mner/lib/python3.8/site-packages/transformers/configuration_utils.py", line 629, in _get_config_dict resolved_config_file = cached_file( File "/data2/hhy/anaconda3/envs/mner/lib/python3.8/site-packages/transformers/utils/hub.py", line 452, in cached_file raise EnvironmentError( OSError: We couldn't connect to 'https://huggingface.co' to load this file, couldn't find it in the cached files and it looks like xlm-roberta-large is not the path to a directory containing a file named config.json. Checkout your internet connection or see how to run the library in offline mode at 'https://huggingface.co/docs/transformers/installation#offline-mode'.
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "/data2/hhy/anaconda3/envs/mner/lib/python3.8/site-packages/modelscope/utils/registry.py", line 212, in build_from_cfg return obj_cls(args) File "/data2/hhy/MNER/adaseq/adaseq/data/preprocessors/sequence_labeling_preprocessor.py", line 38, in init super().init(model_dir, label_to_id=label_to_id, return_offsets=True, kwargs) File "/data2/hhy/MNER/adaseq/adaseq/data/preprocessors/nlp_preprocessor.py", line 71, in init self.tokenizer = build_tokenizer( File "/data2/hhy/MNER/adaseq/adaseq/data/tokenizer.py", line 22, in build_tokenizer model_name_or_path = get_or_download_model_dir(model_name_or_path) File "/data2/hhy/MNER/adaseq/adaseq/utils/hub_utils.py", line 14, in get_or_download_model_dir model_cache_dir = snapshot_download( File "/data2/hhy/anaconda3/envs/mner/lib/python3.8/site-packages/modelscope/hub/snapshot_download.py", line 96, in snapshot_download revision = _api.get_valid_revision( File "/data2/hhy/anaconda3/envs/mner/lib/python3.8/site-packages/modelscope/hub/api.py", line 464, in get_valid_revision revisions = self.list_model_revisions( File "/data2/hhy/anaconda3/envs/mner/lib/python3.8/site-packages/modelscope/hub/api.py", line 433, in list_model_revisions handle_http_response(r, logger, cookies, model_id) File "/data2/hhy/anaconda3/envs/mner/lib/python3.8/site-packages/modelscope/hub/errors.py", line 90, in handle_http_response raise error File "/data2/hhy/anaconda3/envs/mner/lib/python3.8/site-packages/modelscope/hub/errors.py", line 82, in handle_http_response response.raise_for_status() File "/data2/hhy/anaconda3/envs/mner/lib/python3.8/site-packages/requests/models.py", line 1024, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http://www.modelscope.cn/api/v1/models/xlm-roberta-large/revisions?EndTime=1688313600
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/data2/hhy/anaconda3/envs/mner/lib/python3.8/runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/data2/hhy/anaconda3/envs/mner/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/data2/hhy/MNER/adaseq/scripts/train.py", line 38, in
是否可能是因为连不上huggingface所导致的?
刚刚查到一个解决方案:https://developer.aliyun.com/ask/626669,解决了这个问题,2024-06-13 22:33:27,979 - ERROR - modelscope - Authentication token does not exist, failed to access model xlm-roberta-large which may not exist or may be private. Please login first. 之后确实出现了huggingface的问题,然后看你之前说的加上了环境变量就可以了。感谢感谢!! export HF_ENDPOINT=https://hf-mirror.com/
模型地址现在貌似找不到了?不知道有没有其他下载地址,下载到本地,然后到哪里改下代码呢?