Error when downloading grid data

[MxNl]

Hello, I am struggling to download data from "ftp://opendata.dwd.de/climate_environment/CDC/grids_germany".

Am I doing something fundamentally wrong? Thanks for help!

Indexing

radfiles <- indexFTP("daily/Project_TRY/air_temperature_mean/", 
                     base = gridbase, sleep = 1)

Remove PDF files

radfiles <- radfiles[-(1:2)]

If I would not write the full URL, I get an error message, that dataDWD() wants a URL beginning with ftp://...

radfiles <- paste('ftp://opendata.dwd.de/climate_environment/CDC/grids_germany/', 
                  radfiles, 
                  sep = '')

Pick smaller subset for speed

radfiles <- radfiles[1:3]

data <- dataDWD(radfiles, dir = './DWDdata/Data/', sleep = 5)

And this ist the outpu I am getting:

dataDWD -> dirDWD: adding to directory 'C:/Users/Massimo/Documents/DWDdata/Data'
dataDWD -> newFilename: creating 3 files:
C:/Users/Massimo/Documents/DWDdata/Data/ftp:__opendata.dwd.de_climate_environment_CDC_grids_germany_daily_Project_TRY_air_temperature_mean__TT_199501_daymean.nc.gz
C:/Users/Massimo/Documents/DWDdata/Data/ftp:__opendata.dwd.de_climate_environment_CDC_grids_germany_daily_Project_TRY_air_temperature_mean__TT_199502_daymean.nc.gz
  (and 1 more)
  |++++++++++++++++++++++++++++++++++++++++++++++++++| 100% elapsed = 11s
Error: in dataDWD -> readDWD -> checkFile :  The 3 files  C:/Users/Massimo/Documents/DWDdata/Data/ftp:__opendata.dwd.de_climate_environment_CDC_grids_germany_daily_Project_TRY_air_temperature_mean__TT_199501_daymean.nc.gz, C:/Users/Massimo/Documents/DWDdata/Data/ftp:__opendata.dwd.de_climate_environment_CDC_grids_germany_daily_Project_TRY_air_temperature_mean__TT_199502_daymean.nc.gz (and 1 others)
  do not exist. Current getwd: C:/Users/Massimo/Documents/DWDdata/Data

[brry]

I'm only mobile this weekend, so I can't test, but I think the : in the filename might be the problem. You can skip the paste step and use joinbf=TRUE, base=gridbase in dataDWD.

Beware that reading netcdf files is not yet supported. I have a draft on my computer that I can push to github on Monday if you want.

[MxNl]

Hi, thanks for the fast reply!!! When I follow your hints, meaning skipping the paste part and using joinbf=TRUE, base=gridbase in dataDWD i get the following messages:

dataDWD -> dirDWD: adding to directory 'C:/Users/Massimo/Documents/DWDdata/Data'
dataDWD: 3 files already existing and not downloaded again:  daily_Project_TRY_air_temperature_mean__TT_199501_daymean.nc.gz, daily_Project_TRY_air_temperature_mean__TT_199502_daymean.nc.gz  (and 1 more)
Now downloading 0 files...
  |++++++++++++++++++++++++++++++++++++++++++++++++++| 100% elapsed = 00s
Reading 3 files...
  |                                                  | 0 % ~calculating  Error in readDWD.data(file[i], fread = fread[i], varnames = varnames[i],  : 
  There should be a single 'produkt*' file, but there are 0 in
  C:/Users/Massimo/Documents/DWDdata/Data/daily_Project_TRY_air_temperature_mean__TT_199501_daymean.nc.gz
  Consider re-downloading (with force=TRUE).
In addition: Warning message:
In unzip(file, exdir = exdir) : error 1 in extracting from zip file

[brry]

This is exactly as I expected. readDWD is now calling the default readDWD.data which cannot handle netcdf files. I'll rework readDWD.nc so that I can push it online. I'll try to get that done today or tomorrow...

[brry]

The initial version is out. Install it with remotes::install_github("brry/rdwd").
Then see the examples at ?readDWD.nc

Let me know if it doesn't work somewhere. Testing has not yet been very extensive.

[MxNl]

Hey thanks again for your quick response. Our Proxy at work is blocking the connection to the cdc server. I have to test it this evening at home. I'll let you know.

[MxNl]

So I tried under Windows10 and Linux Mint19 and still facing some problems:

Under Windows: Doing remotes::install_github("brry/rdwd") gives me

Running `R CMD build`...
* checking for file 'C:\Users\Massimo\AppData\Local\Temp\RtmpOMKgF9\remotes268c6bf52fe5\brry-rdwd-820f991/DESCRIPTION' ... OK
* preparing 'rdwd':
* checking DESCRIPTION meta-information ... OK
* cleaning src
* checking for LF line-endings in source and make files and shell scripts
* checking for empty or unneeded directories
* looking to see if a 'data/datalist' file should be added
* building 'rdwd_1.1.20.tar.gz'
Installing package into ‘C:/Users/Massimo/Documents/R/win-library/3.5’
(as ‘lib’ is unspecified)
* installing *source* package 'rdwd' ...
** libs

*** arch - i386
Fehler in system(cmd) : (konvertiert von Warnung) 'make' not found
* removing 'C:/Users/Massimo/Documents/R/win-library/3.5/rdwd'
In R CMD INSTALL
Error: Failed to install 'rdwd' from GitHub:
  (converted from warning) installation of package ‘C:/Users/Massimo/AppData/Local/Temp/RtmpOMKgF9/file268cb21369b/rdwd_1.1.20.tar.gz’ had non-zero exit status
In addition: Warning message:
In missing_devel_warning(pkgdir) :
  Package rdwd has compiled code, but no suitable compiler(s) were found. Installation will likely fail.
  Install Rtools (https://cran.r-project.org/bin/windows/Rtools/).Then use the pkgbuild package, or make sure that Rtools in the PATH.```

Under Linux I am struggling to install ncdf4, tried all solutions found on stackoverflow or github.
Trying tomorrow again

[brry]

on Windows, you need to have Rtools installed directly at C:/Rtools (Compiler paths may not have spaces, as there would be with C:/Program Files/R/Rtools/).

On Linux, did you try the following? sudo apt install netcdf-dev in the shell and then install.packages('ncdf4') in R