search

jleuschn / dival

Deep Inversion Validation Library
MIT License
75 stars 13 forks source link

HTTP ERROR： 429 Too Many Requests occurs when downloading train data #34

Closed gaopinghai closed 3 years ago

gaopinghai commented 3 years ago

I have successfully download the test and validation data. But when trying to download the train data， HTTP errors: 429 Too Many Requests always comes out after some time(usually 3~4GB data is done). What should I do?

Connecting to zenodo.org (zenodo.org)|137.138.76.77|:443... connected.
HTTP request sent, awaiting response... 429 Too Many Requests
2021-06-10 16:42:09 ERROR 429: Too Many Requests.

No URLs found in https://zenodo.org/api/files/cfd986de-367d-4a04-be96-b9ea84cd3690/ground_truth_train.zip.
jleuschn commented 3 years ago

Hi, it seems that the server does not accept that many requests in short time. I would guess that increasing the chunk size would help, this could be changed here. It would be great if you could test if using chunk_size=1048576 (1MiB) helps. Then I would change the default argument for the future.

Of course, a workaround would be to download the data manually from zenodo.

gaopinghai commented 3 years ago

Hi, it seems that the server does not accept that many requests in short time. I would guess that increasing the chunk size would help, this could be changed here. It would be great if you could test if using chunk_size=1048576 (1MiB) helps. Then I would change the default argument for the future.

Of course, a workaround would be to download the data manually from zenodo.

Thank you, Sir. I will try it out.

Also, I have tried to manually download the train data. The server also returns with server error. Especially when using wget, I get the same error. I used wget and google default downloader, all failed. So I turned to dival for help.

jleuschn commented 3 years ago

Maybe the access is effectively limited in some countries, we also had a previous user report about troubles to access zenodo.

You can also try this seafile link. Note that every single hdf5 file is zipped separately there. The bash script download.sh from this link downloads and checks everything automatically (in the current working directory). It does so by using seafile_urls.txt (containing single-file download links and filenames) and md5sums.txt.

Please let me know if you experience any troubles with this.

gaopinghai commented 3 years ago

Maybe the access is effectively limited in some countries, we also had a previous user report about troubles to access zenodo.

You can also try this seafile link. Note that every single hdf5 file is zipped separately there. The bash script download.sh from this link downloads and checks everything automatically (in the current working directory). It does so by using seafile_urls.txt (containing single-file download links and filenames) and md5sums.txt.

Please let me know if you experience any troubles with this.

I do think its beacuse of the network environment! And I have solved this by using third party software which can firstly download this data to its cloud disk. Then I successfully download it. The seafile link you provided can also help. Thanks!