search

EBI-Metagenomics / emg-toolkit

MGnify API toolkit
https://www.ebi.ac.uk/metagenomics
Apache License 2.0
21 stars 4 forks source link

Overwrites already downloaded data. #5

Closed casperp closed 4 years ago

casperp commented 5 years ago

My bulk_download stopped because of a HTPP ERROR 500. urllib.error.HTTPError: HTTP Error 500: Internal Server Error

But when i restart the download with the same command it overwrites the already existing data. The download already took 30 minutes and wasn't even half way.

Is it possible that the code first checks if there are already data from a failed download and skips these while downloading ?

Command i used to download : bash mg-toolkit bulk_download -a MGYS00001225 -g taxonomic_annotations

Versions:

mscheremetjew commented 5 years ago

@casperp Thanks for reporting this. This feature is already on our todo list. It makes absolutely sense to get this in.

adnbps2 commented 4 years ago

Any updates? I have the same problem: the download won't finish, but it restarts whenever I try to resume the download. This makes it impossible to download the full dataset using mg-toolkit.

mberacochea commented 4 years ago

Thank you @adnbps2 for contacting us. Unfortunately we haven't implemented this yet.

Could you please let me know what were you trying to download?.

mberacochea commented 4 years ago

There is a new version available that will help with this problem. The new version doesn't re-download the same files, it's safe to retry. We have also added a retry mechanism within the toolkit.

The version is 0.7.0 and it's available in pypi.