ebyrne
commented
2 years ago The connection pool errors are unfortunately just noise from boto. (We're working on that.). But the real issue here is that your poxy appears setup incorrectly. That's not specific to Ego4d. Let us know once you resolve that if you are able to download the dataset.
cyfedu-dlut
python -m ego4d.cli.cli --output_directory="~/ego4d_data" --datasets full_scale annotations --metadata --benchmarks EM datasets: ['full_scale', 'annotations']
Filtering by benchmarks: ['em'] Retrieving object metadata from S3... 100%|█████████████████████████████████████████████████████████████████████████████████████████████████| 3254/3254 [00:03<00:00, 1044.25object/s] Checking if latest file versions are already downloaded... 0%| | 0/3254 [00:00<?, ?file/s]already_downloaded: no version entry for existing file: /GPUFS/nsccgz_ywang_ywj/ego4d_data/v1/annotations/narration.json 0%| | 1/3254 [00:00<05:41, 9.54file/s]already_downloaded: no version entry for existing file: /GPUFS/nsccgz_ywang_ywj/ego4d_data/v1/annotations/fho_sta_train.json already_downloaded: no version entry for existing file: /GPUFS/nsccgz_ywang_ywj/ego4d_data/v1/annotations/fho_sta_val.json already_downloaded: no version entry for existing file: /GPUFS/nsccgz_ywang_ywj/ego4d_data/v1/annotations/av_val.json already_downloaded: no version entry for existing file: /GPUFS/nsccgz_ywang_ywj/ego4d_data/v1/annotations/vq_train.json already_downloaded: no version entry for existing file: /GPUFS/nsccgz_ywang_ywj/ego4d_data/v1/annotations/fho_lta_train.json already_downloaded: no version entry for existing file: /GPUFS/nsccgz_ywang_ywj/ego4d_data/v1/annotations/fho_scod_train.json already_downloaded: no version entry for existing file: /GPUFS/nsccgz_ywang_ywj/ego4d_data/v1/annotations/moments_val.json already_downloaded: no version entry for existing file: /GPUFS/nsccgz_ywang_ywj/ego4d_data/v1/annotations/av_train.json already_downloaded: no version entry for existing file: /GPUFS/nsccgz_ywang_ywj/ego4d_data/v1/annotations/nlq_train.json already_downloaded: no version entry for existing file: /GPUFS/nsccgz_ywang_ywj/ego4d_data/v1/annotations/vq_val.json already_downloaded: no version entry for existing file: /GPUFS/nsccgz_ywang_ywj/ego4d_data/v1/annotations/narration_noun_taxonomy.csv already_downloaded: no version entry for existing file: /GPUFS/nsccgz_ywang_ywj/ego4d_data/v1/annotations/nlq_val.json 0%| | 2/3254 [00:00<22:04, 2.46file/s]already_downloaded: no version entry for existing file: /GPUFS/nsccgz_ywang_ywj/ego4d_data/v1/annotations/moments_train.json already_downloaded: no version entry for existing file: /GPUFS/nsccgz_ywang_ywj/ego4d_data/v1/annotations/fho_scod_val.json already_downloaded: no version entry for existing file: /GPUFS/nsccgz_ywang_ywj/ego4d_data/v1/annotations/fho_hands_val.json WARNING:urllib3.connectionpool:Connection pool is full, discarding connection: ego4d-consortium-sharing.s3.us-west-1.amazonaws.com. Connection pool size: 10 WARNING:urllib3.connectionpool:Connection pool is full, discarding connection: ego4d-consortium-sharing.s3.us-west-1.amazonaws.com. Connection pool size: 10 already_downloaded: no version entry for existing file: /GPUFS/nsccgz_ywang_ywj/ego4d_data/v1/annotations/fho_lta_val.json already_downloaded: no version entry for existing file: /GPUFS/nsccgz_ywang_ywj/ego4d_data/v1/annotations/fho_lta_taxonomy.json WARNING:urllib3.connectionpool:Connection pool is full, discarding connection: ego4d-consortium-sharing.s3.us-west-1.amazonaws.com. Connection pool size: 10 WARNING:urllib3.connectionpool:Connection pool is full, discarding connection: ego4d-consortium-sharing.s3.us-west-1.amazonaws.com. Connection pool size: 10 already_downloaded: no version entry for existing file: /GPUFS/nsccgz_ywang_ywj/ego4d_data/v1/annotations/narration_verb_taxonomy.csv 0%| | 3/3254 [00:00<17:09, 3.16file/s]WARNING:urllib3.connectionpool:Connection pool is full, discarding connection: ego4d-consortium-sharing.s3.us-west-1.amazonaws.com. Connection pool size: 10 already_downloaded: no version entry for existing file: /GPUFS/nsccgz_ywang_ywj/ego4d_data/v1/annotations/fho_hands_train.json 1%|█▎ | 43/3254 [00:32<40:27, 1.32file/s] WARNING:urllib3.connectionpool:Connection pool is full, discarding connection: ego4d-unict.s3.eu-central-1.amazonaws.com. Connection pool size: 10 WARNING:urllib3.connectionpool:Connection pool is full, discarding connection: ego4d-unict.s3.eu-central-1.amazonaws.com. Connection pool size: 10 WARNING:urllib3.connectionpool:Connection pool is full, discarding connection: ego4d-unict.s3.eu-central-1.amazonaws.com. Connection pool size: 10 WARNING:urllib3.connectionpool:Connection pool is full, discarding connection: ego4d-unict.s3.eu-central-1.amazonaws.com. Connection pool size: 10 WARNING:urllib3.connectionpool:Connection pool is full, discarding connection: ego4d-unict.s3.eu-central-1.amazonaws.com. Connection pool size: 10 Traceback (most recent call last): File "/GPUFS/nsccgz_ywang_ywj/anaconda3/envs/cjd_ego4d/lib/python3.8/site-packages/urllib3-1.26.8-py3.8.egg/urllib3/connectionpool.py", line 700, in urlopen self._prepare_proxy(conn) File "/GPUFS/nsccgz_ywang_ywj/anaconda3/envs/cjd_ego4d/lib/python3.8/site-packages/urllib3-1.26.8-py3.8.egg/urllib3/connectionpool.py", line 994, in _prepare_proxy conn.connect() File "/GPUFS/nsccgz_ywang_ywj/anaconda3/envs/cjd_ego4d/lib/python3.8/site-packages/urllib3-1.26.8-py3.8.egg/urllib3/connection.py", line 371, in connect self._tunnel() File "/GPUFS/nsccgz_ywang_ywj/anaconda3/envs/cjd_ego4d/lib/python3.8/http/client.py", line 905, in _tunnel raise OSError("Tunnel connection failed: %d %s" % (code, OSError: Tunnel connection failed: 503 Service Unavailable
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "/GPUFS/nsccgz_ywang_ywj/anaconda3/envs/cjd_ego4d/lib/python3.8/site-packages/botocore/httpsession.py", line 405, in send urllib_response = conn.urlopen( File "/GPUFS/nsccgz_ywang_ywj/anaconda3/envs/cjd_ego4d/lib/python3.8/site-packages/urllib3-1.26.8-py3.8.egg/urllib3/connectionpool.py", line 785, in urlopen retries = retries.increment( File "/GPUFS/nsccgz_ywang_ywj/anaconda3/envs/cjd_ego4d/lib/python3.8/site-packages/urllib3-1.26.8-py3.8.egg/urllib3/util/retry.py", line 525, in increment raise six.reraise(type(error), error, _stacktrace) File "/GPUFS/nsccgz_ywang_ywj/anaconda3/envs/cjd_ego4d/lib/python3.8/site-packages/urllib3-1.26.8-py3.8.egg/urllib3/packages/six.py", line 769, in reraise raise value.with_traceback(tb) File "/GPUFS/nsccgz_ywang_ywj/anaconda3/envs/cjd_ego4d/lib/python3.8/site-packages/urllib3-1.26.8-py3.8.egg/urllib3/connectionpool.py", line 700, in urlopen self._prepare_proxy(conn) File "/GPUFS/nsccgz_ywang_ywj/anaconda3/envs/cjd_ego4d/lib/python3.8/site-packages/urllib3-1.26.8-py3.8.egg/urllib3/connectionpool.py", line 994, in _prepare_proxy conn.connect() File "/GPUFS/nsccgz_ywang_ywj/anaconda3/envs/cjd_ego4d/lib/python3.8/site-packages/urllib3-1.26.8-py3.8.egg/urllib3/connection.py", line 371, in connect self._tunnel() File "/GPUFS/nsccgz_ywang_ywj/anaconda3/envs/cjd_ego4d/lib/python3.8/http/client.py", line 905, in _tunnel raise OSError("Tunnel connection failed: %d %s" % (code, urllib3.exceptions.ProxyError: ('Cannot connect to proxy.', OSError('Tunnel connection failed: 503 Service Unavailable'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "/GPUFS/nsccgz_ywang_ywj/anaconda3/envs/cjd_ego4d/lib/python3.8/runpy.py", line 194, in _run_module_as_main return _run_code(code, main_globals, None, File "/GPUFS/nsccgz_ywang_ywj/anaconda3/envs/cjd_ego4d/lib/python3.8/runpy.py", line 87, in _run_code exec(code, run_globals) File "/GPUFS/nsccgz_ywang_ywj/anaconda3/envs/cjd_ego4d/lib/python3.8/site-packages/ego4d-1.0-py3.8.egg/ego4d/cli/cli.py", line 220, in
File "/GPUFS/nsccgz_ywang_ywj/anaconda3/envs/cjd_ego4d/lib/python3.8/site-packages/ego4d-1.0-py3.8.egg/ego4d/cli/cli.py", line 136, in main
File "/GPUFS/nsccgz_ywang_ywj/anaconda3/envs/cjd_ego4d/lib/python3.8/site-packages/ego4d-1.0-py3.8.egg/ego4d/cli/download.py", line 241, in filter_already_downloaded
File "/GPUFS/nsccgz_ywang_ywj/anaconda3/envs/cjd_ego4d/lib/python3.8/site-packages/tqdm-4.63.0-py3.8.egg/tqdm/std.py", line 1195, in iter
for obj in iterable:
File "/GPUFS/nsccgz_ywang_ywj/anaconda3/envs/cjd_ego4d/lib/python3.8/concurrent/futures/_base.py", line 619, in result_iterator
yield fs.pop().result()
File "/GPUFS/nsccgz_ywang_ywj/anaconda3/envs/cjd_ego4d/lib/python3.8/concurrent/futures/_base.py", line 444, in result
return self.get_result()
File "/GPUFS/nsccgz_ywang_ywj/anaconda3/envs/cjd_ego4d/lib/python3.8/concurrent/futures/_base.py", line 389, in get_result
raise self._exception
File "/GPUFS/nsccgz_ywang_ywj/anaconda3/envs/cjd_ego4d/lib/python3.8/concurrent/futures/thread.py", line 57, in run
result = self.fn(*self.args, *self.kwargs)
File "/GPUFS/nsccgz_ywang_ywj/anaconda3/envs/cjd_ego4d/lib/python3.8/site-packages/ego4d-1.0-py3.8.egg/ego4d/cli/download.py", line 244, in
File "/GPUFS/nsccgz_ywang_ywj/anaconda3/envs/cjd_ego4d/lib/python3.8/site-packages/ego4d-1.0-py3.8.egg/ego4d/cli/download.py", line 193, in already_downloaded
File "/GPUFS/nsccgz_ywang_ywj/anaconda3/envs/cjd_ego4d/lib/python3.8/site-packages/ego4d-1.0-py3.8.egg/ego4d/cli/download.py", line 65, in exists
File "/GPUFS/nsccgz_ywang_ywj/.local/lib/python3.8/site-packages/boto3/resources/factory.py", line 505, in do_action
response = action(self, args, **kwargs)
File "/GPUFS/nsccgz_ywang_ywj/.local/lib/python3.8/site-packages/boto3/resources/action.py", line 83, in call__
response = getattr(parent.meta.client, operation_name)(*args, params)
File "/GPUFS/nsccgz_ywang_ywj/anaconda3/envs/cjd_ego4d/lib/python3.8/site-packages/botocore/client.py", line 395, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/GPUFS/nsccgz_ywang_ywj/anaconda3/envs/cjd_ego4d/lib/python3.8/site-packages/botocore/client.py", line 711, in _make_api_call
http, parsed_response = self._make_request(
File "/GPUFS/nsccgz_ywang_ywj/anaconda3/envs/cjd_ego4d/lib/python3.8/site-packages/botocore/client.py", line 731, in _make_request
return self._endpoint.make_request(operation_model, request_dict)
File "/GPUFS/nsccgz_ywang_ywj/anaconda3/envs/cjd_ego4d/lib/python3.8/site-packages/botocore/endpoint.py", line 107, in make_request
return self._send_request(request_dict, operation_model)
File "/GPUFS/nsccgz_ywang_ywj/anaconda3/envs/cjd_ego4d/lib/python3.8/site-packages/botocore/endpoint.py", line 183, in _send_request
while self._needs_retry(attempts, operation_model, request_dict,
File "/GPUFS/nsccgz_ywang_ywj/anaconda3/envs/cjd_ego4d/lib/python3.8/site-packages/botocore/endpoint.py", line 305, in _needs_retry
responses = self._event_emitter.emit(
File "/GPUFS/nsccgz_ywang_ywj/anaconda3/envs/cjd_ego4d/lib/python3.8/site-packages/botocore/hooks.py", line 357, in emit
return self._emitter.emit(aliased_event_name, kwargs)
File "/GPUFS/nsccgz_ywang_ywj/anaconda3/envs/cjd_ego4d/lib/python3.8/site-packages/botocore/hooks.py", line 228, in emit
return self._emit(event_name, kwargs)
File "/GPUFS/nsccgz_ywang_ywj/anaconda3/envs/cjd_ego4d/lib/python3.8/site-packages/botocore/hooks.py", line 211, in _emit
response = handler(kwargs)
File "/GPUFS/nsccgz_ywang_ywj/anaconda3/envs/cjd_ego4d/lib/python3.8/site-packages/botocore/utils.py", line 1484, in redirect_from_error
new_region = self.get_bucket_region(bucket, response)
File "/GPUFS/nsccgz_ywang_ywj/anaconda3/envs/cjd_ego4d/lib/python3.8/site-packages/botocore/utils.py", line 1541, in get_bucket_region
response = self._client.head_bucket(Bucket=bucket)
File "/GPUFS/nsccgz_ywang_ywj/anaconda3/envs/cjd_ego4d/lib/python3.8/site-packages/botocore/client.py", line 395, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/GPUFS/nsccgz_ywang_ywj/anaconda3/envs/cjd_ego4d/lib/python3.8/site-packages/botocore/client.py", line 711, in _make_api_call
http, parsed_response = self._make_request(
File "/GPUFS/nsccgz_ywang_ywj/anaconda3/envs/cjd_ego4d/lib/python3.8/site-packages/botocore/client.py", line 731, in _make_request
return self._endpoint.make_request(operation_model, request_dict)
File "/GPUFS/nsccgz_ywang_ywj/anaconda3/envs/cjd_ego4d/lib/python3.8/site-packages/botocore/endpoint.py", line 107, in make_request
return self._send_request(request_dict, operation_model)
File "/GPUFS/nsccgz_ywang_ywj/anaconda3/envs/cjd_ego4d/lib/python3.8/site-packages/botocore/endpoint.py", line 183, in _send_request
while self._needs_retry(attempts, operation_model, request_dict,
File "/GPUFS/nsccgz_ywang_ywj/anaconda3/envs/cjd_ego4d/lib/python3.8/site-packages/botocore/endpoint.py", line 305, in _needs_retry
responses = self._event_emitter.emit(
File "/GPUFS/nsccgz_ywang_ywj/anaconda3/envs/cjd_ego4d/lib/python3.8/site-packages/botocore/hooks.py", line 357, in emit
return self._emitter.emit(aliased_event_name, kwargs)
File "/GPUFS/nsccgz_ywang_ywj/anaconda3/envs/cjd_ego4d/lib/python3.8/site-packages/botocore/hooks.py", line 228, in emit
return self._emit(event_name, kwargs)
File "/GPUFS/nsccgz_ywang_ywj/anaconda3/envs/cjd_ego4d/lib/python3.8/site-packages/botocore/hooks.py", line 211, in _emit
response = handler(**kwargs)
File "/GPUFS/nsccgz_ywang_ywj/anaconda3/envs/cjd_ego4d/lib/python3.8/site-packages/botocore/retryhandler.py", line 192, in call
if self._checker(**checker_kwargs):
File "/GPUFS/nsccgz_ywang_ywj/anaconda3/envs/cjd_ego4d/lib/python3.8/site-packages/botocore/retryhandler.py", line 265, in call
should_retry = self._should_retry(attempt_number, response,
File "/GPUFS/nsccgz_ywang_ywj/anaconda3/envs/cjd_ego4d/lib/python3.8/site-packages/botocore/retryhandler.py", line 292, in _should_retry
return self._checker(attempt_number, response, caught_exception)
File "/GPUFS/nsccgz_ywang_ywj/anaconda3/envs/cjd_ego4d/lib/python3.8/site-packages/botocore/retryhandler.py", line 331, in call__
checker_response = checker(attempt_number, response,
File "/GPUFS/nsccgz_ywang_ywj/anaconda3/envs/cjd_ego4d/lib/python3.8/site-packages/botocore/retryhandler.py", line 231, in call
return self._check_caught_exception(
File "/GPUFS/nsccgz_ywang_ywj/anaconda3/envs/cjd_ego4d/lib/python3.8/site-packages/botocore/retryhandler.py", line 374, in _check_caught_exception
raise caught_exception
File "/GPUFS/nsccgz_ywang_ywj/anaconda3/envs/cjd_ego4d/lib/python3.8/site-packages/botocore/endpoint.py", line 249, in _do_get_response
http_response = self._send(request)
File "/GPUFS/nsccgz_ywang_ywj/anaconda3/envs/cjd_ego4d/lib/python3.8/site-packages/botocore/endpoint.py", line 321, in _send
return self.http_session.send(request)
File "/GPUFS/nsccgz_ywang_ywj/anaconda3/envs/cjd_ego4d/lib/python3.8/site-packages/botocore/httpsession.py", line 436, in send
raise ProxyConnectionError(proxy_url=mask_proxy_url(proxy_url), error=e)
botocore.exceptions.ProxyConnectionError: Failed to connect to proxy URL: "http://10.20.18.21:3128"