search

brry / rdwd

download climate data from DWD (German Weather Service)
https://bookdown.org/brry/rdwd
72 stars 13 forks source link

use polite scraping #19

Open brry opened 4 years ago

brry commented 4 years ago

Consider using polite for downloading files. Maybe this is already conceptually fine with no re-downloads and CURL handle in https://github.com/brry/rdwd/blob/master/R/indexFTP.R#L121

brry commented 4 years ago

indexFTP might be fine, but dataDWD uses download.file with no checks. When calling dataDWD with, say, 500 urls, once banned, it will still keep trying the rest of the urls. Definitely not polite.