search

fdetsch / MODIS

Download and processing framework for MODIS imagery. The package provides automated access to the global online data archives LP DAAC, LAADS and NSIDC as well as processing capabilities such as file conversion, mosaicking, subsetting and time series.
Other
58 stars 27 forks source link

Corrupted .hdf files with runGdal #77

Closed bcosentino closed 4 years ago

bcosentino commented 4 years ago

The following code to download MOD13Q1 data has worked for me previously but started to return a corrupted file warning in December 2019:

e.tile <- extent(-91.8, -89.1, -1.5, 0.7)
ndvi.cur.yr.raw <- runGdal("MOD13Q1", begin="2019-12-01", extent=e.tile, SDSstring="100000000010", job="/ndvi_16day")

The result:

########################
outProj          =  asIn  (if applicable, derived from Raster*/Spatial*/sf* object)
pixelSize        =  asIn  (if applicable, derived from Raster* object)
resamplingType   =  near 
Output Directory =  Z:/MODIS/gvs/processed//ndvi_16day 
########################
Local structure is up-to-date. Using offline information!
MOD13Q1.A2019337.h08v09.006.2019354133608.hdf is corrupted, trying to re-download it!

Getting file from: LAADS 
############################
trying URL 'https://ladsweb.modaps.eosdis.nasa.gov/archive/allData/6/MOD13Q1/2019/337/MOD13Q1.A2019337.h08v09.006.2019354133608.hdf'
Content type 'text/html; charset=utf-8' length unknown
downloaded 12 KB

Downloaded by the first try!

MOD13Q1.A2019337.h08v09.006.2019354133608.hdf is corrupted, trying to re-download it!

The download is attempted many times with the same error. I'm running this on Windows 10. I updated to the latest package versions, but the problem persists. My full session info is below. Any idea what's going wrong?

R version 3.6.2 (2019-12-12)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows >= 8 x64 (build 9200)

Matrix products: default

locale:
[1] LC_COLLATE=English_United States.1252  LC_CTYPE=English_United States.1252    LC_MONETARY=English_United States.1252
[4] LC_NUMERIC=C                           LC_TIME=English_United States.1252    

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] ncdf4_1.17    MODIS_1.1.6   mapdata_2.3.0 maps_3.3.0    stringr_1.4.0 rgeos_0.5-2   raster_3.0-7  rgdal_1.4-8   sp_1.3-2     

loaded via a namespace (and not attached):
 [1] remotes_2.1.0      sf_0.8-0           lattice_0.20-38    testthat_2.3.1     usethis_1.5.1      htmltools_0.4.0    rlang_0.4.2       
 [8] pkgbuild_1.0.6     e1071_1.7-3        nloptr_1.2.1       later_1.0.0        DBI_1.1.0          foreign_0.8-72     glue_1.3.1        
[15] withr_2.1.2        sessioninfo_1.1.1  mapedit_0.5.0      devtools_2.2.1     codetools_0.2-16   memoise_1.1.0      callr_3.4.0       
[22] fastmap_1.0.1      httpuv_1.5.2       ps_1.3.0           maptools_0.9-9     class_7.3-15       parallel_3.6.2     curl_4.3          
[29] fansi_0.4.0        Rcpp_1.0.3         KernSmooth_2.23-16 xtable_1.8-4       classInt_0.4-2     backports_1.1.5    promises_1.1.0    
[36] desc_1.2.0         pkgload_1.0.2      mime_0.8           fs_1.3.1           ptw_1.9-15         digest_0.6.23      stringi_1.4.3     
[43] processx_3.4.1     shiny_1.4.0        grid_3.6.2         rprojroot_1.3-2    cli_2.0.0          tools_3.6.2        bitops_1.0-6      
[50] magrittr_1.5       crayon_1.3.4       ellipsis_0.3.0     prettyunits_1.0.2  assertthat_0.2.1   rstudioapi_0.10    R6_2.4.1          
[57] units_0.6-5        compiler_3.6.2

MODIS settings:

STORAGE:
_______________
localArcPath : Z:/MODIS/gvs/raw/ 
outDirPath   : Z:/MODIS/gvs/processed/ 

DOWNLOAD:
_______________
MODISserverOrder : LAADS, LPDAAC 
dlmethod         : auto 
stubbornness     : high 
wait             : 0.5 
quiet            : FALSE 

PROCESSING:
_______________
GDAL           : GDAL 2.3.2, released 2018/09/21 
MRT            : Version 4.1 (March 2011) 
pixelSize      : asIn 
outProj        : asIn 
resamplingType : NN 
dataFormat     : GTiff 
cellchunk      : 1
jeangolay commented 4 years ago

I have had the same (or a closely related) problem (since December 2019, as well). I have temporarily fixed it by explicitly setting: MODISserverOrder = "LPDAAC" in MODISoptions.

bcosentino commented 4 years ago

Thanks. I changed the server to LPDAAC and can now download files again. I wonder if the issue with LAADS is related to a new requirement for user authentication beginning December 5, 2020.

fdetsch commented 4 years ago

Good catch. I'll have a closer look at it. In the meantime, just stick with MODISserverOrder = "LPDAAC".

fdetsch commented 4 years ago

Solved with c4becf4ff529cedc7f71eabfd9ab8dbfb900a1c0. LAADS download is back online and MODISserverOrder = c("LPDAAC", "LAADS") the suggested (hence default) setting. LP DAAC is considerably faster when it comes to file retrieval.