search

RETURN-project / makeDataCube

Data management
Apache License 2.0
0 stars 0 forks source link

wvp-global.tar.gz not downloaded #27

Closed PabRod closed 3 years ago

PabRod commented 3 years ago

dllWVP is failing to download wvp-global.tar.gz. Any clue about what can be happening, @wandadk?

See output below:

Registered S3 method overwritten by 'quantmod':
  method            from
  as.zoo.data.frame zoo 
Warning messages:
1: replacing previous import ‘lubridate::intersect’ by ‘raster::intersect’ when loading ‘makeDataCube’ 
2: replacing previous import ‘lubridate::origin’ by ‘raster::origin’ when loading ‘makeDataCube’ 
3: replacing previous import ‘lubridate::union’ by ‘raster::union’ when loading ‘makeDataCube’ 
4: replacing previous import ‘lubridate::setdiff’ by ‘rgeos::setdiff’ when loading ‘makeDataCube’ 
5: replacing previous import ‘rgdal::project’ by ‘terra::project’ when loading ‘makeDataCube’ 
6: replacing previous import ‘terra::time<-’ by ‘zoo::time<-’ when loading ‘makeDataCube’ 
trying URL 'http://hs.pangaea.de/sat/MODIS/Frantz-Stellmes_2018/wvp-global.tar.gz'
Content type 'application/octet-stream' length 793299291 bytes (756.5 MB)
=======
downloaded 108.7 MB

wvp-global.tar.gz
 not downloaded at /home/pablo/Desktop/makeDataCube/data/misc/wvp
Warning messages:
1: In download.file("http://hs.pangaea.de/sat/MODIS/Frantz-Stellmes_2018/wvp-global.tar.gz",  :
  downloaded length 113950081 != reported length 793299291
2: In download.file("http://hs.pangaea.de/sat/MODIS/Frantz-Stellmes_2018/wvp-global.tar.gz",  :
  URL 'https://hs.pangaea.de/sat/MODIS/Frantz-Stellmes_2018/wvp-global.tar.gz': Timeout of 60 seconds was reached
3: In system(paste0("tar -xvzf ", file.path(wvpfolder, "wvp-global.tar.gz"),  :
  running command 'tar -xvzf /home/pablo/Desktop/makeDataCube/data/misc/wvp/wvp-global.tar.gz -C /home/pablo/Desktop/makeDataCube/data/misc/wvp 2>/dev/null' had status 2
wandadk commented 3 years ago

It looks like the download of the precompiled wvp dataset (https://doi.pangaea.de/10.1594/PANGAEA.893109) is failing due to a timeout issue. I found a fix for this bug using the package curl. Shall I commit this to directly to the main branch?

PabRod commented 3 years ago

It sounds like a small patch for something that is already failing, so pushing it to master is not a bad idea.

Had it been a bigger thing, I would have suggested creating a branch named 27-download or something like that.

PabRod commented 3 years ago

For some reason, downloading it with curl, directly from the console, causes an incomplete download.

I am considering using a system call with something like wget http://hs.pangaea.de/sat/MODIS/Frantz-Stellmes_2018/wvp-global.tar.gz. Seems to work better.

PabRod commented 3 years ago

Nevermind. curl works fine when called from R.

Sorry for the noise.

PabRod commented 3 years ago

I streamlined the code in dllWVP a bit (https://github.com/RETURN-project/makeDataCube/commit/b5c082721e05fb8dd9933fb3743264332595ca0d).

Now there is a clearer separation between downloading and unpacking, and the downloaded .tar.gz is kept by default.

wandadk commented 3 years ago

Nice, looks great!