SPARC-FAIR-Codeathon / sparc-me

A python tool to explore, enhance, and expand SPARC datasets and their descriptions

Apache License 2.0

7 stars 6 forks source link

Ability to download entire dataset (>5GB) using multi-threading #83

Closed LinkunGao closed 1 year ago

LinkunGao commented 2 years ago

Description:

add download a whole dataset, and add an example for this in the examples folder. example_for_download_whole_dataset.py.

Initially using a single thread to complete this task will cost several minutes. Now, I redesign a multi-thread architecture to achieve this task only cost almost 26 seconds!

Related issue(s):

37

Test Environment:

Specify the testing enviroment. Mac OS python 3.9

LinkunGao commented 2 years ago

validate dataset_id: if the dataset id input by the user does not exist in the SPARC dataset, the function will stop and notify the user by printing "no dataset from SPARC".

chinchien-lin commented 1 year ago

Looking good and tested