search

apple / ARKitScenes

This repo accompanies the research paper, ARKitScenes - A Diverse Real-World Dataset for 3D Indoor Scene Understanding Using Mobile RGB-D Data and contains the data, scripts to visualize and process assets, and training code described in our paper.
Other
662 stars 58 forks source link

Optimize download script #29

Closed Gofinge closed 2 years ago

Gofinge commented 2 years ago

An optimized download script with the following change log:

  1. Add validation for assets in raw_dataset_assets, which should be a subset of default raw_dataset_assets;
  2. Fix a bug that argparse.ArgumentError takes two arguments, but only one (error message) is provided. Use assert to replace argparse.ArgumentError;
  3. Modify function download_file, download the target file to a temp file and rename to filename only download the data successfully. As a result, the download_file will skip the downloading task if the target file exists, which makes the script can resume downloading after an accident occurs without redownloading.
  4. Add function unzip_file with similar characteristic as download_file meanwhile quite the unzip process.
  5. Optimize downloading and unzipping zip files in download_data.

The new download script should be much more friendly for downloading accidents, especially since ARKitScenes is a large dataset. Hope my work is helpful, and please further verify my code.

Attach with an example bash output when resuming downloading process after interruption:

python download_data.py 3dod --split Training --video_id_csv threedod/3dod_train_val_splits.csv --download_dir ~/Downloads/ARKitScene
WARNING: skipping download of existing file: /Users/gofinge/Downloads/ARKitScene/3dod/metadata.csv
WARNING: skipping download and unzip of target zip file: /Users/gofinge/Downloads/ARKitScene/3dod/Training/41254925.zip
WARNING: skipping download and unzip of target zip file: /Users/gofinge/Downloads/ARKitScene/3dod/Training/41254934.zip
WARNING: skipping download and unzip of target zip file: /Users/gofinge/Downloads/ARKitScene/3dod/Training/41254940.zip
WARNING: skipping download and unzip of target zip file: /Users/gofinge/Downloads/ARKitScene/3dod/Training/41254954.zip
WARNING: skipping download and unzip of target zip file: /Users/gofinge/Downloads/ARKitScene/3dod/Training/41254956.zip
WARNING: skipping download and unzip of target zip file: /Users/gofinge/Downloads/ARKitScene/3dod/Training/45260872.zip
WARNING: skipping download and unzip of target zip file: /Users/gofinge/Downloads/ARKitScene/3dod/Training/45260873.zip
WARNING: skipping download and unzip of target zip file: /Users/gofinge/Downloads/ARKitScene/3dod/Training/45260877.zip
WARNING: skipping download and unzip of target zip file: /Users/gofinge/Downloads/ARKitScene/3dod/Training/47333462.zip
WARNING: skipping download and unzip of target zip file: /Users/gofinge/Downloads/ARKitScene/3dod/Training/47333463.zip
Downloading file /Users/gofinge/Downloads/ARKitScene/3dod/Training/45260889.zip
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 57.6M  100 57.6M    0     0  6725k      0  0:00:08  0:00:08 --:--:-- 13.9M
Unzipping zip file /Users/gofinge/Downloads/ARKitScene/3dod/Training/45260889.zip
Downloading file /Users/gofinge/Downloads/ARKitScene/3dod/Training/45260890.zip
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 50.9M  100 50.9M    0     0  5433k      0  0:00:09  0:00:09 --:--:-- 8925k
Unzipping zip file /Users/gofinge/Downloads/ARKitScene/3dod/Training/45260890.zip
Downloading file /Users/gofinge/Downloads/ARKitScene/3dod/Training/45260891.zip
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0 44.5M    0 12766    0     0  10611      0  1:13:20  0:00:01  1:13:19 10611
arik1089 commented 2 years ago

Pull request was merged to main. Thanks again for your contribution.