unable to download historical data

[psteinb]

Apparently, the dwd has updated some filenames for historical data. I wanted to download the historical monthly numbers for weather stations in Saxony, but received errors for 31 out of 53 stations. Here is an example for Geringswalde-Altgeringswalde:

> library(rdwd)
> link <- selectDWD(id=131, res="monthly", var="kl", per="historical")
> file <- dataDWD(link, read=FALSE)
dataDWD -> dirDWD: creating directory '/home/steinbac/DWDdata'
dataDWD -> newFilename: creating 1 file:
'/home/steinbac/DWDdata/monthly_kl_historical_monatswerte_KL_00131_20041101_20181231_hist.zip'
Warning message:
1 Download has failed (out of 1).
download.file error:
cannot open URL 'ftp://opendata.dwd.de/climate_environment/CDC/observations_germany/climate/monthly/kl/historical/monatswerte_KL_00131_20041101_20181231_hist.zip'

When checking the URL and comparing to CDC folder, I observe that monatswerte_KL_00131_20041101_20181231_hist.zip is now monatswerte_KL_00131_20041101_20191231_hist.zip at this link. So a year's worth of data was added.

> sessionInfo()
R version 3.6.3 (2020-02-29)
Platform: x86_64-redhat-linux-gnu (64-bit)
Running under: Fedora 31 (Workstation Edition)

Matrix products: default
BLAS/LAPACK: /usr/lib64/R/lib/libRblas.so

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
 [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
 [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
 [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
 [9] LC_ADDRESS=C               LC_TELEPHONE=C            
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] rdwd_1.3.1

loaded via a namespace (and not attached):
[1] compiler_3.6.3        parallel_3.6.3        tools_3.6.3          
[4] berryFunctions_1.18.2 abind_1.4-5           pbapply_1.4-2

[brry]

I'v already updated the development version that you can get with rdwd::updateRdwd()

It should be on CRAN some time in the near future as well, but I'm still working on some other updates as well.

[psteinb]

just ran rdwd::updateRdwd() and upgraded to rdwd_1.3.14, the issue prevails. Any help would be appreciated.

[psteinb]

ok, had to reload rdwd after the update. that did the trick. this appears to work now. thanks!

[brry]

Oh good comment. I'll see if I can add instructions to reload accordingly

[psteinb]

mmh, still found a bad URL:

cannot open URL 'ftp://opendata.dwd.de/climate_environment/CDC/observations_germany/climate/monthly/kl/historical/monatswerte_KL_00840_19950301_20191231_hist.zip'

The start date was changed on the server, see https://opendata.dwd.de/climate_environment/CDC/observations_germany/climate/monthly/kl/historical/monatswerte_KL_00840_19891201_20191231_hist.zip more than 5 years more data!

[brry]

Wow, they really are expanding a lot. It's kind of a constant battle to keep up to date. For the moment, you can use current=TRUE, see https://bookdown.org/brry/rdwd/fileindex.html I try not to query their servers too often, so it might be some time untill the next update of the fileIndex...

[psteinb]

fair enough, current=T helped with those stations that didn't work out at first. Thanks for the quick response!