search

facebookresearch / Ego4d

Ego4d dataset repository. Download the dataset, visualize, extract features & example usage of the dataset
https://ego4d-data.org/docs/
MIT License
341 stars 47 forks source link

can access v1, but not v2 and v2_1 #343

Closed giuliano-97 closed 1 month ago

giuliano-97 commented 1 month ago

I configured my AWS credentials and installed the Ego4D CLI as per the documentation. This works:

ego4d --list-datasets --version v1
Downloading datasets.csv..

Available Ego4D datasets:
   annotations              The full set of annotations for the majority of benchmarks.
   full_scale               The full scale version of all videos.  (Provide `benchmarks` or `video_uids` filters to reduce the 5TB download size.)
   clips                    Clips available for benchmark training tasks.  (Provide `benchmarks` or `video_uids` filters to reduce the download size.) 
   video_540ss              The downscaled version of all videos - rescaled to 540px on the short side.  (Provide `benchmarks` or `video_uids` filters to reduce the 5TB download size.) 
   annotations_540ss        The annotations corresponding to the downscaled `video_540ss` videos - primarily differing only in spatial annotations (e.g. bounding boxes). 
   3d                       Annotations for the 3D VQ benchmark. 
   3d_scans                 3D location scans for the 3D VQ benchmark. 
   3d_scan_keypoints        3D location scan keypoints for the 3D VQ benchmark. 
   imu                      IMU data for the subset of videos available 
   slowfast8x8_r101_k400    Precomputed [action features](https://ego4d-data.org/docs/data/features/) for the Slowfast 8x8 (R101) model 
   omnivore_video_swinl     Precomputed [action features](https://ego4d-data.org/docs/data/features/) for the Omnivore Video model 
   omnivore_image_swinl     Precomputed [action features](https://ego4d-data.org/docs/data/features/) for the Omnivore Image model 
   fut_loc                  Images and annotations for the future locomotion benchmark. 
   av_models                Model checkpoints for the AV/Social benchmark. 
   lta_models               Model checkpoints for the Long Term Anticipation benchmark. 
   moments_models           Model checkpoints for the Moments benchmark. 
   nlq_models               Model checkpoints for the NLQ benchmark. 
   sta_models               Model checkpoints for the Short Term Anticipation benchmark. 
   vq2d_models              Model checkpoints for the 2D VQ benchmark.

But passing --version v2 or --version v2_1 results in this error:

$ ego4d --list-datasets --version 2_1
Downloading datasets.csv..
ERROR:root:Exception retrieving Ego4D datasets: An error occurred (403) when calling the HeadObject operation: Forbidden
Traceback (most recent call last):
  File "/home/giuliano/.virtualenvs/ego4d/lib/python3.10/site-packages/ego4d/cli/manifest.py", line 237, in print_datasets
    p = download_datasets(version, tmppath, s3)
  File "/home/giuliano/.virtualenvs/ego4d/lib/python3.10/site-packages/ego4d/cli/manifest.py", line 228, in download_datasets
    _datasets_object(version, s3).download_file(str(download_path))
  File "/home/giuliano/.virtualenvs/ego4d/lib/python3.10/site-packages/boto3/s3/inject.py", line 361, in object_download_file
    return self.meta.client.download_file(
  File "/home/giuliano/.virtualenvs/ego4d/lib/python3.10/site-packages/boto3/s3/inject.py", line 192, in download_file
    return transfer.download_file(
  File "/home/giuliano/.virtualenvs/ego4d/lib/python3.10/site-packages/boto3/s3/transfer.py", line 406, in download_file
    future.result()
  File "/home/giuliano/.virtualenvs/ego4d/lib/python3.10/site-packages/s3transfer/futures.py", line 103, in result
    return self._coordinator.result()
  File "/home/giuliano/.virtualenvs/ego4d/lib/python3.10/site-packages/s3transfer/futures.py", line 266, in result
    raise self._exception
  File "/home/giuliano/.virtualenvs/ego4d/lib/python3.10/site-packages/s3transfer/tasks.py", line 269, in _main
    self._submit(transfer_future=transfer_future, **kwargs)
  File "/home/giuliano/.virtualenvs/ego4d/lib/python3.10/site-packages/s3transfer/download.py", line 354, in _submit
    response = client.head_object(
  File "/home/giuliano/.virtualenvs/ego4d/lib/python3.10/site-packages/botocore/client.py", line 565, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/home/giuliano/.virtualenvs/ego4d/lib/python3.10/site-packages/botocore/client.py", line 1017, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden
giuliano-97 commented 1 month ago

I've tried with the latest main, still get the same error:

ego4d --list-datasets --version v2_1
Downloading datasets.csv..
ERROR:root:Exception retrieving Ego4D datasets: An error occurred (403) when calling the HeadObject operation: Forbidden
Traceback (most recent call last):
  File "/home/giuliano/dev/ego4d/Ego4d/ego4d/cli/manifest.py", line 237, in print_datasets
    p = download_datasets(version, tmppath, s3)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/giuliano/dev/ego4d/Ego4d/ego4d/cli/manifest.py", line 228, in download_datasets
    _datasets_object(version, s3).download_file(str(download_path))
  File "/home/giuliano/miniforge3/envs/ego4d/lib/python3.11/site-packages/boto3-1.34.150-py3.11.egg/boto3/s3/inject.py", line 361, in object_download_file
    return self.meta.client.download_file(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/giuliano/miniforge3/envs/ego4d/lib/python3.11/site-packages/boto3-1.34.150-py3.11.egg/boto3/s3/inject.py", line 192, in download_file
    return transfer.download_file(
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/giuliano/miniforge3/envs/ego4d/lib/python3.11/site-packages/boto3-1.34.150-py3.11.egg/boto3/s3/transfer.py", line 406, in download_file
    future.result()
  File "/home/giuliano/miniforge3/envs/ego4d/lib/python3.11/site-packages/s3transfer-0.10.2-py3.11.egg/s3transfer/futures.py", line 103, in result
    return self._coordinator.result()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/giuliano/miniforge3/envs/ego4d/lib/python3.11/site-packages/s3transfer-0.10.2-py3.11.egg/s3transfer/futures.py", line 266, in result
    raise self._exception
  File "/home/giuliano/miniforge3/envs/ego4d/lib/python3.11/site-packages/s3transfer-0.10.2-py3.11.egg/s3transfer/tasks.py", line 269, in _main
    self._submit(transfer_future=transfer_future, **kwargs)
  File "/home/giuliano/miniforge3/envs/ego4d/lib/python3.11/site-packages/s3transfer-0.10.2-py3.11.egg/s3transfer/download.py", line 354, in _submit
    response = client.head_object(
               ^^^^^^^^^^^^^^^^^^^
  File "/home/giuliano/miniforge3/envs/ego4d/lib/python3.11/site-packages/botocore-1.34.150-py3.11.egg/botocore/client.py", line 565, in _api_call
    return self._make_api_call(operation_name, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/giuliano/miniforge3/envs/ego4d/lib/python3.11/site-packages/botocore-1.34.150-py3.11.egg/botocore/client.py", line 1017, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden

@miguelmartin75 any ideas?

giuliano-97 commented 1 month ago

Update While fetching the dataset list doesn't work with --version v2, downloading seems to work:

ego4d --output_directory="~/dev/ego4d/data" --datasets full_scale annotations --video_uids 8301a3fc-aac5-466d-bb02-d0ac7a81ccc6
Datasets to download: {'annotations', 'full_scale'}
Download Path: /home/giuliano/dev/ego4d/data/v2
Downloading Ego4D metadata json..
Ego4D Metadata: /home/giuliano/dev/ego4d/data/ego4d.json
Checking requested datasets and versions...
Created download directory for version 'v2_1' of dataset: 'annotations' at: /home/giuliano/dev/ego4d/data/v2/annotations
Only downloading a subset of the video files because the 'video_uids' flag has been set on the command line or in the config file. A total of 1 video files will be downloaded.

Created download directory for version 'v2_1' of dataset: 'full_scale' at: /home/giuliano/dev/ego4d/data/v2/full_scale
Only downloading a subset of the video files because the 'video_uids' flag has been set on the command line or in the config file. A total of 1 video files will be downloaded.

Retrieving object metadata from S3...
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 34/34 [00:00<00:00, 3135.93object/s]
Checking if latest file versions are already downloaded...
  6%|██████████▏                                                                                                                                                                  | 2/34 [00:00<00:11,  2.75file/s]WARNING:urllib3.connectionpool:Connection pool is full, discarding connection: ego4d-consortium-sharing.s3.us-west-1.amazonaws.com. Connection pool size: 10
WARNING:urllib3.connectionpool:Connection pool is full, discarding connection: ego4d-consortium-sharing.s3.us-west-1.amazonaws.com. Connection pool size: 10
WARNING:urllib3.connectionpool:Connection pool is full, discarding connection: ego4d-consortium-sharing.s3.us-west-1.amazonaws.com. Connection pool size: 10
WARNING:urllib3.connectionpool:Connection pool is full, discarding connection: ego4d-consortium-sharing.s3.us-west-1.amazonaws.com. Connection pool size: 10
WARNING:urllib3.connectionpool:Connection pool is full, discarding connection: ego4d-consortium-sharing.s3.us-west-1.amazonaws.com. Connection pool size: 10
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 34/34 [00:21<00:00,  1.56file/s]
No existing videos to filter.
Expected size of downloaded files is 6.1137 GB. Do you want to start the download? ([y]/n) y
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 6.11G/6.11G [10:29<00:00, 1.94MiB/s]Checking file integrity...
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 6.11G/6.11G [10:29<00:00, 10.4MiB/s]
miguelmartin75 commented 1 month ago

Yeah, I did not update the datasets.csv for V2.1 or V2

It is now available for V2.1

Thank you for the flag.

giuliano-97 commented 1 month ago

thanks, it works now :)