search

facebookresearch / Ego4d

Ego4d dataset repository. Download the dataset, visualize, extract features & example usage of the dataset
https://ego4d-data.org/docs/
MIT License
359 stars 49 forks source link

[EndpointConnectionError] when downloading #224

Closed aopolin-lv closed 1 year ago

aopolin-lv commented 1 year ago

Hello, when I excute ego4d --output_directory="./" --datasets full_scale annotations command, the program is terminated by such error.

Datasets to download: {'annotations', 'full_scale'}
Download Path: v2
Ego4D Metadata: ego4d.json
Checking requested datasets and versions...
Created download directory for version 'v2' of dataset: 'annotations' at: v2/annotations
Created download directory for version 'v2' of dataset: 'full_scale' at: v2/full_scale
Retrieving object metadata from S3...
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 9640/9640 [00:07<00:00, 1272.76object/s]
Checking if latest file versions are already downloaded...
  0%|▌                                                                                                                                                                                                                                                                                             | 17/9640 [00:01<10:28, 15.31file/s]
WARNING:urllib3.connectionpool:Connection pool is full, discarding connection: ego4d-consortium-sharing.s3.us-west-1.amazonaws.com. Connection pool size: 10
WARNING:urllib3.connectionpool:Connection pool is full, discarding connection: ego4d-consortium-sharing.s3.us-west-1.amazonaws.com. Connection pool size: 10
WARNING:urllib3.connectionpool:Connection pool is full, discarding connection: ego4d-consortium-sharing.s3.us-west-1.amazonaws.com. Connection pool size: 10
WARNING:urllib3.connectionpool:Connection pool is full, discarding connection: ego4d-consortium-sharing.s3.us-west-1.amazonaws.com. Connection pool size: 10
WARNING:urllib3.connectionpool:Connection pool is full, discarding connection: ego4d-consortium-sharing.s3.us-west-1.amazonaws.com. Connection pool size: 10
  0%|▉                                                                                                                                                                                                                                                                                          | 30/9640 [04:07<21:59:22,  8.24s/file]
WARNING:urllib3.connectionpool:Connection pool is full, discarding connection: ego4d-unict.s3.eu-central-1.amazonaws.com. Connection pool size: 10
WARNING:urllib3.connectionpool:Connection pool is full, discarding connection: ego4d-unict.s3.eu-central-1.amazonaws.com. Connection pool size: 10
WARNING:urllib3.connectionpool:Connection pool is full, discarding connection: ego4d-unict.s3.eu-central-1.amazonaws.com. Connection pool size: 10
WARNING:urllib3.connectionpool:Connection pool is full, discarding connection: ego4d-unict.s3.eu-central-1.amazonaws.com. Connection pool size: 10
WARNING:urllib3.connectionpool:Connection pool is full, discarding connection: ego4d-unict.s3.eu-central-1.amazonaws.com. Connection pool size: 10
Traceback (most recent call last):
  File "/root/miniconda3/envs/ego/lib/python3.9/site-packages/urllib3/connection.py", line 174, in _new_conn
    conn = connection.create_connection(
  File "/root/miniconda3/envs/ego/lib/python3.9/site-packages/urllib3/util/connection.py", line 95, in create_connection
    raise err
  File "/root/miniconda3/envs/ego/lib/python3.9/site-packages/urllib3/util/connection.py", line 85, in create_connection
    sock.connect(sa)
OSError: [Errno 101] Network is unreachable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/miniconda3/envs/ego/lib/python3.9/site-packages/botocore/httpsession.py", line 465, in send
    urllib_response = conn.urlopen(
  File "/root/miniconda3/envs/ego/lib/python3.9/site-packages/urllib3/connectionpool.py", line 798, in urlopen
    retries = retries.increment(
  File "/root/miniconda3/envs/ego/lib/python3.9/site-packages/urllib3/util/retry.py", line 525, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/root/miniconda3/envs/ego/lib/python3.9/site-packages/urllib3/packages/six.py", line 770, in reraise
    raise value
  File "/root/miniconda3/envs/ego/lib/python3.9/site-packages/urllib3/connectionpool.py", line 714, in urlopen
    httplib_response = self._make_request(
  File "/root/miniconda3/envs/ego/lib/python3.9/site-packages/urllib3/connectionpool.py", line 403, in _make_request
    self._validate_conn(conn)
  0%|▉                                                                                                                                                                                                                                                                                          | 30/9640 [04:07<21:59:22,  8.24[30/51]
WARNING:urllib3.connectionpool:Connection pool is full, discarding connection: ego4d-unict.s3.eu-central-1.amazonaws.com. Connection pool size: 10                                                                                                                                                                                     
WARNING:urllib3.connectionpool:Connection pool is full, discarding connection: ego4d-unict.s3.eu-central-1.amazonaws.com. Connection pool size: 10                                                                                                                                                                                     
WARNING:urllib3.connectionpool:Connection pool is full, discarding connection: ego4d-unict.s3.eu-central-1.amazonaws.com. Connection pool size: 10                                                                                                                                                                                     
WARNING:urllib3.connectionpool:Connection pool is full, discarding connection: ego4d-unict.s3.eu-central-1.amazonaws.com. Connection pool size: 10                                                                                                                                                                                     
WARNING:urllib3.connectionpool:Connection pool is full, discarding connection: ego4d-unict.s3.eu-central-1.amazonaws.com. Connection pool size: 10                                                                                                                                                                                     
Traceback (most recent call last):                                                                                                                                                                                                                                                                                                     
  File "/root/miniconda3/envs/ego/lib/python3.9/site-packages/urllib3/connection.py", line 174, in _new_conn                                                                                                                                                                                                                           
    conn = connection.create_connection(                                                                                                                                                                                                                                                                                               
  File "/root/miniconda3/envs/ego/lib/python3.9/site-packages/urllib3/util/connection.py", line 95, in create_connection                                                                                                                                                                                                               
    raise err                                                                                                                                                                                                                                                                                                                          
  File "/root/miniconda3/envs/ego/lib/python3.9/site-packages/urllib3/util/connection.py", line 85, in create_connection                                                                                                                                                                                                               
    sock.connect(sa)                                                                                                                                                                                                                                                                                                                   
OSError: [Errno 101] Network is unreachable                                                                                                                                                                                                                                                                                            

During handling of the above exception, another exception occurred:                                                                                                                                                                                                                                                                    

Traceback (most recent call last):                                                                                                                                                                                                                                                                                                     
  File "/root/miniconda3/envs/ego/lib/python3.9/site-packages/botocore/httpsession.py", line 465, in send                                                                                                                                                                                                                              
    urllib_response = conn.urlopen(                                                                                                                                                                                                                                                                                                    
  File "/root/miniconda3/envs/ego/lib/python3.9/site-packages/urllib3/connectionpool.py", line 798, in urlopen                                                                                                                                                                                                                         
    retries = retries.increment(                                                                                                                                                                                                                                                                                                       
  File "/root/miniconda3/envs/ego/lib/python3.9/site-packages/urllib3/util/retry.py", line 525, in increment                                                                                                                                                                                                                           
    raise six.reraise(type(error), error, _stacktrace)                                                                                                                                                                                                                                                                                 
  File "/root/miniconda3/envs/ego/lib/python3.9/site-packages/urllib3/packages/six.py", line 770, in reraise                                                                                                                                                                                                                           
    raise value                                                                                                                                                                                                                                                                                                                        
  File "/root/miniconda3/envs/ego/lib/python3.9/site-packages/urllib3/connectionpool.py", line 714, in urlopen                                                                                                                                                                                                                         
    httplib_response = self._make_request(                                                                                                                                                                                                                                                                                             
  File "/root/miniconda3/envs/ego/lib/python3.9/site-packages/urllib3/connectionpool.py", line 403, in _make_request                                                                                                                                                                                                                   
    self._validate_conn(conn)                                                                                                                                                                                                                                                                                                          
  File "/root/miniconda3/envs/ego/lib/python3.9/site-packages/urllib3/connectionpool.py", line 1053, in _validate_conn                                                                                                                                                                                                                 
    conn.connect()
  File "/root/miniconda3/envs/ego/lib/python3.9/site-packages/urllib3/connection.py", line 363, in connect
    self.sock = conn = self._new_conn()
  File "/root/miniconda3/envs/ego/lib/python3.9/site-packages/urllib3/connection.py", line 186, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <botocore.awsrequest.AWSHTTPSConnection object at 0x7fe6180d6790>: Failed to establish a new connection: [Errno 101] Network is unreachable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/miniconda3/envs/ego/bin/ego4d", line 8, in <module>
    sys.exit(main())
  File "/root/miniconda3/envs/ego/lib/python3.9/site-packages/ego4d/cli/cli.py", line 251, in main
    main_cfg(config)
  File "/root/miniconda3/envs/ego/lib/python3.9/site-packages/ego4d/cli/cli.py", line 154, in main_cfg
    active_downloads = filter_already_downloaded(
  File "/root/miniconda3/envs/ego/lib/python3.9/site-packages/ego4d/cli/download.py", line 247, in filter_already_downloaded
    to_download = list(
  File "/root/miniconda3/envs/ego/lib/python3.9/site-packages/tqdm/std.py", line 1178, in __iter__
    for obj in iterable:
  File "/root/miniconda3/envs/ego/lib/python3.9/concurrent/futures/_base.py", line 609, in result_iterator
    yield fs.pop().result()
  File "/root/miniconda3/envs/ego/lib/python3.9/concurrent/futures/_base.py", line 446, in result
    return self.__get_result()
  File "/root/miniconda3/envs/ego/lib/python3.9/concurrent/futures/_base.py", line 391, in __get_result
    raise self._exception
  File "/root/miniconda3/envs/ego/lib/python3.9/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/root/miniconda3/envs/ego/lib/python3.9/site-packages/ego4d/cli/download.py", line 250, in <lambda>
    lambda x: x.s3_object and not already_downloaded(x) and x.s3_exists,
  File "/root/miniconda3/envs/ego/lib/python3.9/site-packages/ego4d/cli/download.py", line 198, in already_downloaded
    skip_s3_checks or download.exists()
  File "/root/miniconda3/envs/ego/lib/python3.9/site-packages/ego4d/cli/download.py", line 68, in exists
    self.s3_object.load()
  File "/root/miniconda3/envs/ego/lib/python3.9/site-packages/boto3/resources/factory.py", line 564, in do_action
    response = action(self, *args, **kwargs)
  File "/root/miniconda3/envs/ego/lib/python3.9/site-packages/boto3/resources/action.py", line 88, in __call__
    response = getattr(parent.meta.client, operation_name)(*args, **params)
  File "/root/miniconda3/envs/ego/lib/python3.9/site-packages/botocore/client.py", line 535, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/root/miniconda3/envs/ego/lib/python3.9/site-packages/botocore/client.py", line 963, in _make_api_call
    http, parsed_response = self._make_request(
  File "/root/miniconda3/envs/ego/lib/python3.9/site-packages/botocore/client.py", line 986, in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
  File "/root/miniconda3/envs/ego/lib/python3.9/site-packages/botocore/endpoint.py", line 119, in make_request
    return self._send_request(request_dict, operation_model)
  File "/root/miniconda3/envs/ego/lib/python3.9/site-packages/botocore/endpoint.py", line 202, in _send_request
    while self._needs_retry(
  File "/root/miniconda3/envs/ego/lib/python3.9/site-packages/botocore/endpoint.py", line 354, in _needs_retry
    responses = self._event_emitter.emit(
  File "/root/miniconda3/envs/ego/lib/python3.9/site-packages/botocore/hooks.py", line 412, in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
  File "/root/miniconda3/envs/ego/lib/python3.9/site-packages/botocore/hooks.py", line 256, in emit
    return self._emit(event_name, kwargs)
  File "/root/miniconda3/envs/ego/lib/python3.9/site-packages/botocore/hooks.py", line 239, in _emit
    response = handler(**kwargs)
  File "/root/miniconda3/envs/ego/lib/python3.9/site-packages/botocore/retryhandler.py", line 207, in __call__
    if self._checker(**checker_kwargs):
  File "/root/miniconda3/envs/ego/lib/python3.9/site-packages/botocore/retryhandler.py", line 284, in __call__
    should_retry = self._should_retry(
  File "/root/miniconda3/envs/ego/lib/python3.9/site-packages/botocore/retryhandler.py", line 320, in _should_retry
    return self._checker(attempt_number, response, caught_exception)
  File "/root/miniconda3/envs/ego/lib/python3.9/site-packages/botocore/retryhandler.py", line 363, in __call__
    checker_response = checker(
  File "/root/miniconda3/envs/ego/lib/python3.9/site-packages/botocore/retryhandler.py", line 247, in __call__
    return self._check_caught_exception(
  File "/root/miniconda3/envs/ego/lib/python3.9/site-packages/botocore/retryhandler.py", line 416, in _check_caught_exception
    raise caught_exception
  File "/root/miniconda3/envs/ego/lib/python3.9/site-packages/botocore/endpoint.py", line 281, in _do_get_response
    http_response = self._send(request)
  File "/root/miniconda3/envs/ego/lib/python3.9/site-packages/botocore/endpoint.py", line 377, in _send
    return self.http_session.send(request)
  File "/root/miniconda3/envs/ego/lib/python3.9/site-packages/botocore/httpsession.py", line 494, in send
    raise EndpointConnectionError(endpoint_url=request.url, error=e)
botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL: "https://ego4d-unict.s3.eu-central-1.amazonaws.com/public/v1/full_scale/3e08beb0-9108-4e77-b2ae-80f91ceac474"

How can I do to deal with this error?

huaihjn commented 1 year ago

I have same question, do you slove it?

aopolin-lv commented 1 year ago

I have same question, do you slove it?

I have used VPN on the server and tried to connect many times. Finally, it succeeded.

huaihjn commented 1 year ago

I have same question, do you slove it?

I have used VPN on the server and tried to connect many times. Finally, it succeeded.

thank you

Enchantedll commented 9 months ago

I have used VPN, but also failed. Can you explain in detial how you solved this problem,for example, did you configure aws region? thank you.