search

ropensci / MODIStsp

An "R" package for automatic download and preprocessing of MODIS Land Products Time Series
https://docs.ropensci.org/MODIStsp
GNU General Public License v3.0
155 stars 51 forks source link

Projection MCD43C4 #204

Closed jvanpassel closed 4 years ago

jvanpassel commented 4 years ago

Hello,

I have used MODIStsp version 1.3.9 successfully before to download MCD43C4 images. Right now I am working on a new computer and I downloaded the newest MODIStsp version 1.4.0, but I am unable to download these same images.

The difference seems to be with the native output projection in the GUI. In the new version this just says '4008', while in the older version it said '+init=epsg:4008 +proj=longlat +ellps=clrk66 +no_defs'. When I try to load the extent from the same spatial file, I now get the following error message: 'Error in st_crs.character(out_proj_crs) : invalid crs: 4008'

Whether I manually change the projection or not and then start to download the images, I always get the following error message: [Wed Jun 03 14:01:22 2020] MODIStsp --> Starting processing [Wed Jun 03 14:01:22 2020] Accessing http server at: https://e4ftl01.cr.usgs.gov/MOTA/MCD43C4.006/ [Wed Jun 03 14:01:26 2020] Retrieving list of available Combined Files for Year 2012 [Wed Jun 03 14:01:27 2020] HDF File: MCD43C4.A2012001.006.2016194224247.hdf already exists on your system. Skipping download! [Wed Jun 03 14:01:27 2020] [Wed Jun 03 14:01:27 2020] 1 files for date: 2012.01.01 were successfully downloaded! [Wed Jun 03 14:01:27 2020] Processing Terra Nad_Refl_b1_Red files for date: 2012_01_01 Error in gdal_utils("translate", src_dataset, dst_dataset, opts) : gdal_utils translate: an error occured In addition: Warning message: In CPL_gdaltranslate(source, destination, options, oo) : GDAL Error 3: Free disk space available is 52379979776 bytes, whereas 1027057032002 are at least necessary. You can disable this check by defining the CHECK_DISK_FREE_SPACE configuration option to FALSE.

I already tried to download the older version of MODIStsp on this computer, but I'm not able to for some reason. Would you know how to solve my problem?

lbusett commented 4 years ago

Hi,

this is reasonably related to recent updates to rgdal / and raster packages that I will have to deal with....

Could you please share your session info ?

Lorenzo

jvanpassel commented 4 years ago

Hi,

Thanks for the quick reply, this is my session info:

R version 3.6.0 (2019-04-26) Platform: x86_64-w64-mingw32/x64 (64-bit) Running under: Windows 10 x64 (build 17763)

Matrix products: default

locale: [1] LC_COLLATE=Dutch_Belgium.1252 LC_CTYPE=Dutch_Belgium.1252 LC_MONETARY=Dutch_Belgium.1252 [4] LC_NUMERIC=C LC_TIME=Dutch_Belgium.1252

attached base packages: [1] stats graphics grDevices utils datasets methods base

other attached packages: [1] gWidgetsRGtk2_0.0-86.1 cairoDevice_2.28 gWidgets_0.0-54.2 RGtk2_2.20.36 MODIStsp_1.4.0

loaded via a namespace (and not attached): [1] gdalUtilities_1.1.0 zoo_1.8-8 sf_0.9-3 lattice_0.20-38 mapview_2.7.8
[6] colorspace_1.4-1 htmltools_0.4.0 stats4_3.6.0 viridisLite_0.3.0 base64enc_0.1-3
[11] rlang_0.4.6 e1071_1.7-3 later_1.0.0 DBI_1.1.0 sp_1.4-2
[16] lifecycle_0.2.0 stringr_1.4.0 munsell_0.5.0 mapedit_0.6.0 raster_3.1-5
[21] htmlwidgets_1.5.1 codetools_0.2-16 fastmap_1.0.1 httpuv_1.5.3.1 crosstalk_1.1.0.1
[26] curl_4.3 parallel_3.6.0 class_7.3-15 leafem_0.1.1 xts_0.12-0
[31] Rcpp_1.0.4.6 KernSmooth_2.23-15 xtable_1.8-4 promises_1.1.0 scales_1.1.1
[36] classInt_0.4-3 satellite_1.0.2 leaflet_2.0.3 webshot_0.5.2 jsonlite_1.6.1
[41] mime_0.9 png_0.1-7 digest_0.6.25 stringi_1.4.6 shiny_1.4.0.2
[46] grid_3.6.0 rgdal_1.5-8 tools_3.6.0 bitops_1.0-6 magrittr_1.5
[51] data.table_1.12.8 xml2_1.3.2 httr_1.4.1 R6_2.4.1 units_0.6-6
[56] compiler_3.6.0

lbusett commented 4 years ago

Ok..,, I may have an idea about what's happening. Would you be able to share the spatial file you are using and your processing options (you can the options to a JSON file from the GUI)

Lorenzo

jvanpassel commented 4 years ago

How am I supposed to share these files? When I try to add them, I get a notification that this file type is not supported here.

jvanpassel commented 4 years ago

I converted the JSON file to a txt file so I could share them, is this okay? The modistspv140_options file is the one that doesn't work, while the modistspv139_options file did work on another computer.

modistspv140_options.txt

modistspv139_options.txt

I don't know how to share a shapefile here, so here is some information on it:

testarea<-readOGR(dsn = 'I:/PhD',layer = 'testarea') #small testarea within Brazil and Bolivia for faster running of script OGR data source with driver: ESRI Shapefile Source: "I:\PhD", layer: "testarea" with 1 features It has 1 fields Integer64 fields read as strings: id crs(testarea) CRS arguments: +proj=longlat +datum=WGS84 +no_defs Warning message: In proj4string(x) : CRS object has comment, which is lost in output

lbusett commented 4 years ago

thanks. To share the shapefile, you could try compressing it to zip (all 3/5 files) and attach it.

jvanpassel commented 4 years ago

Aha thank you, here is my shapefile:

testarea.zip

jvanpassel commented 4 years ago

Hi,

Would you have any idea how to solve my problem?

Johanna

lbusett commented 4 years ago

@jvanpassel

sorry - I had very little time, these days. I'll try to have a look as soon as i can.

Lorenzo

lbusett commented 4 years ago

Ok, I can replicate this: looking into it now.

lbusett commented 4 years ago

It appears that gdalbuildvrt does not recognize correctly anymore the "corners" of the HDF files in 4008 projection. For example, this: 

gdalinfo HDF4_EOS:EOS_GRID:"D:/RTemp/RtmpaKR6yR/MODIStsp/HDFs/MCD43C4.A2016001.006.2016196180021.hdf":MCD_CMG_BRDF_0.05Deg:Perc
ent_Snow

gives: 

Corner Coordinates:
Upper Left  (  -0.0500000,   0.0250000) (  0d 3' 0.00"W,  0d 1'30.00"N)
Lower Left  (  -0.0500000,  -0.0250000) (  0d 3' 0.00"W,  0d 1'30.00"S)
Upper Right (   0.0500000,   0.0250000) (  0d 3' 0.00"E,  0d 1'30.00"N)
Lower Right (   0.0500000,  -0.0250000) (  0d 3' 0.00"E,  0d 1'30.00"S)
Center      (   0.0000000,   0.0000000) (  0d 0' 0.01"E,  0d 0' 0.01"N)

, that is completely wrong.

Would you be able to test this also on your machine? If that is the case, I will have to find a way to reassign the corner coordinates before processing.

Lorenzo

jvanpassel commented 4 years ago

Hi,

I used this code in R instead as it didn't work in the command prompt:

gdalinfo('I:/PhD/MODIS/MCD43C4.A2016002.006.2016196180012.hdf')

And this is what I got back (I deleted most lines to make it more clear): [1] "Driver: HDF4/Hierarchical Data Format Release 4"
[2] "Files: I:/PhD/MODIS/MCD43C4.A2016002.006.2016196180012.hdf"
[3] "Size is 512, 512"
[4] "Coordinate System is `'"
[84] "Corner Coordinates:"
[85] "Upper Left ( 0.0, 0.0)"
[86] "Lower Left ( 0.0, 512.0)"
[87] "Upper Right ( 512.0, 0.0)"
[88] "Lower Right ( 512.0, 512.0)"
[89] "Center ( 256.0, 256.0)"

lbusett commented 4 years ago

@jvanpassel

to see what I am needing, you should run the command on an SDS. Something like.

gdalinfo HDF4_EOS:EOS_GRID:'I:/PhD/MODIS/MCD43C4.A2016002.006.2016196180012.hdf':MCD_CMG_BRDF_0.05Deg:Percent_Snow

should work.

jvanpassel commented 4 years ago

Hi, What exactly is an SDS? I tried running this command on different platforms, but I always got error messages.

lbusett commented 4 years ago

an SDS is one of the subdatasets of an HDF file. can you share your error message?

Lorenzo

jvanpassel commented 4 years ago

Thanks, I was able to specify a SDS in R and then I got the following message:

[81] "Corner Coordinates:"
[82] "Upper Left ( -0.0500000, 0.0250000) ( 0d 3' 0.00\"W, 0d 1'30.00\"N)"
[83] "Lower Left ( -0.0500000, -0.0250000) ( 0d 3' 0.00\"W, 0d 1'30.00\"S)"
[84] "Upper Right ( 0.0500000, 0.0250000) ( 0d 3' 0.00\"E, 0d 1'30.00\"N)"
[85] "Lower Right ( 0.0500000, -0.0250000) ( 0d 3' 0.00\"E, 0d 1'30.00\"S)"
[86] "Center ( 0.0000000, 0.0000000) ( 0d 0' 0.01\"E, 0d 0' 0.01\"N)"

lbusett commented 4 years ago

Yes, same problem, apparently....

Can you tell me which GDAL version you have installed?

Lorenzo

jvanpassel commented 4 years ago

My R rgdal version is 1.5-10 and it uses GDAL version 3.0.4.

lbusett commented 4 years ago

OK. same here... It's probably a regressione related to changes in GDAL.

I'll see if I can manage a quick fix.

Lorenzo

przell commented 4 years ago

Hi Lorenzo, we are also having trouble with the Projections. On Ubuntu 18.04, gdal 3.0.4, proj 7.0.0 and MODIStsp 1.4.0. We are using the non-gui processing. Currently the parameter output_proj4 = "32632" and output_proj4 = "+proj=utm +zone=32 +datum=WGS84 +units=m +no_defs +type=crs" in the json file fail with the error: "Internal error in out_res_sel, outproj_str or full_ext. Aborting!"

Having a quick glimpse at your functions I have noted the following:

I can't provide the used json now since I don't have acces to the VM of the colleague who has encountered this problem. But could try to get hold of it, if it helps you.

Sorry this was just a quick review from my side. No extensive testing. Forgive me if I have missed something or am completely wrong. Hopefully this is of help. Best, Peter

lbusett commented 4 years ago

Hi @przell

thanks for signaling this. I do not think that what iyoiu reported is the culprit, though, because "output_proj4" is automatically converted to a crs object already in MODIStps() main function (line 396).

To understand what's happening, I'd need to have a look at the processing parameters. In particular MODIS product, output resolution, output bounding box.

Could you provide me that info?

przell commented 4 years ago

Hi Lorenzo, ok, perfect. Unfortunately, I hadn't checked the main MODIStsp() function 🤦 . I will provide you with the json files and exact errors soon! Thanks for your quick response!

lbusett commented 4 years ago

@jvanpassel

I probably found a solution for you problem - I'll provide a fix soon for you to test.

lbusett commented 4 years ago

@jvanpassel

I pushed a patch to github - ca you try installing from github and see if it solves your issue?

remotes::install_github("ropensci/MODIStsp", ref = "master")

jvanpassel commented 4 years ago

Yes it worked perfectly, thank you so much!

przell commented 4 years ago

Thanks for the quick fix! We have reinstalled and tested for our use case but we are still receiving the same error. Here our json file. Originally we are using local MODIS files. So I also attached one using the online archive. MODIStsp_GQ_offline_linux.json.txt MODIStsp_GQ_online_linux.json.txt and also the error messages: MODIStsp_GQ_offline_linux.json: 

GDAL version in use: 3.0.4
[Fri Jun 19 09:29:30 2020] MODIStsp --> Starting processing
[Fri Jun 19 09:29:30 2020] Accessing http server at:  https://e4ftl01.cr.usgs.gov/MOLT/MOD09GQ.006/
[Fri Jun 19 09:29:35 2020] Retrieving list of available ` Terra ` Files for Year 2008
Loading required namespace: gWidgetsRGtk2
Failed with error:  ‘there is no package called ‘gWidgetsRGtk2’’
[Fri Jun 19 09:29:39 2020] HDF File: MOD09GQ.A2008001.h18v04.006.2015169042051.hdf already exists on your system. Skipping download!
[Fri Jun 19 09:29:39 2020] [Fri Jun 19 09:29:39 2020] 1 files for date: 2008.01.01 were successfully downloaded!
[Fri Jun 19 09:29:41 2020] Processing and Reprojecting Terra b1_Red files for date: 2008_01_01
Error in MODIStsp_process_bands(out_folder_mod, modislist, outproj_str,  : 
  Internal error in out_res_sel, outproj_str or full_ext. Aborting!

MODIStsp_GQ_online_linux.json: 

GDAL version in use: 3.0.4
[Fri Jun 19 09:39:53 2020] MODIStsp --> Starting processing
[Fri Jun 19 09:39:53 2020] Accessing http server at:  https://e4ftl01.cr.usgs.gov/MOLT/MOD09GQ.006/
[Fri Jun 19 09:40:07 2020] Retrieving list of available ` Terra ` Files for Year 2011
Loading required namespace: gWidgetsRGtk2
Failed with error:  ‘there is no package called ‘gWidgetsRGtk2’’
[Fri Jun 19 09:40:09 2020] Downloading Terra Files for date: 2011_01_01 : 1 of:  1
[Fri Jun 19 09:40:52 2020] [Fri Jun 19 09:40:52 2020] 1 files for date: 2011.01.01 were successfully downloaded!
[Fri Jun 19 09:40:52 2020] Processing and Reprojecting Terra b1_Red files for date: 2011_01_01
Error in MODIStsp_process_bands(out_folder_mod, modislist, outproj_str,  : 
  Internal error in out_res_sel, outproj_str or full_ext. Aborting!

Best, Peter

lbusett commented 4 years ago

@przell

your problem is a different one that I will have to investigate. Could you please open a new issue? Closing this now.

Lorenzo

lbusett commented 4 years ago

Hi Lorenzo, we are also having trouble with the Projections. On Ubuntu 18.04, gdal 3.0.4, proj 7.0.0 and MODIStsp 1.4.0. We are using the non-gui processing. Currently the parameter output_proj4 = "32632" and output_proj4 = "+proj=utm +zone=32 +datum=WGS84 +units=m +no_defs +type=crs" in the json file fail with the error: "Internal error in out_res_sel, outproj_str or full_ext. Aborting!"

Having a quick glimpse at your functions I have noted the following:

  • MODIStsp_process.R: uncommented this in line 164 # outproj_str <- sf::st_as_text(sf::st_crs(check_projection(outproj_str))). Is this intended?
  • MODIStsp_process_bands.R: you are refering to outproj$wkt in line 357 -> how can this work if the input is a string and not converted since it is uncommented
  • proj_bbox.R: You are handling the projection differently here. Might be worth to align this.

I can't provide the used json now since I don't have acces to the VM of the colleague who has encountered this problem. But could try to get hold of it, if it helps you.

Sorry this was just a quick review from my side. No extensive testing. Forgive me if I have missed something or am completely wrong. Hopefully this is of help. Best, Peter

@przell In the meantime, I tested on my machine with your options file, and everything seems to work fine. Can I ask you to share your / your colleague's session info?

przell commented 4 years ago

Here are the spatial dependencies:

> library(sf)
Linking to GEOS 3.8.0, GDAL 3.0.4, PROJ 7.0.0

And here is the session Info:

> sessionInfo()
R version 3.6.0 (2019-04-26)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 18.04.2 LTS

Matrix products: default
BLAS:   /usr/lib/x86_64-linux-gnu/openblas/libblas.so.3
LAPACK: /usr/lib/x86_64-linux-gnu/libopenblasp-r0.2.20.so

locale:
 [1] LC_CTYPE=C.UTF-8       LC_NUMERIC=C           LC_TIME=C.UTF-8        LC_COLLATE=C.UTF-8     LC_MONETARY=C.UTF-8    LC_MESSAGES=C.UTF-8   
 [7] LC_PAPER=C.UTF-8       LC_NAME=C              LC_ADDRESS=C           LC_TELEPHONE=C         LC_MEASUREMENT=C.UTF-8 LC_IDENTIFICATION=C   

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] rgdal_1.5-10   sp_1.3-2       MODIStsp_1.4.0

loaded via a namespace (and not attached):
 [1] Rcpp_1.0.4.6        lattice_0.20-38     prettyunits_1.1.1   png_0.1-7           class_7.3-15        ps_1.3.3            zoo_1.8-8          
 [8] rprojroot_1.3-2     assertthat_0.2.1    digest_0.6.25       mime_0.9            R6_2.4.1            backports_1.1.7     stats4_3.6.0       
[15] e1071_1.7-3         httr_1.4.1          pillar_1.4.1        rlang_0.4.6         curl_4.3            rstudioapi_0.11     data.table_1.12.8  
[22] callr_3.4.3         raster_3.0-12       webshot_0.5.2       stringr_1.4.0       htmlwidgets_1.5.1   munsell_0.5.0       shiny_1.4.0.2      
[29] compiler_3.6.0      httpuv_1.5.4        pkgconfig_2.0.3     base64enc_0.1-3     pkgbuild_1.0.8      htmltools_0.4.0     gdalUtilities_1.1.0
[36] mapedit_0.6.0       tidyselect_0.2.5    tibble_2.1.3        codetools_0.2-16    gWidgets_0.0-54     mapview_2.7.8       fansi_0.4.1        
[43] viridisLite_0.3.0   withr_2.2.0         crayon_1.3.4        dplyr_0.8.1         later_1.1.0.1       sf_0.9-4            bitops_1.0-6       
[50] grid_3.6.0          jsonlite_1.6.1      satellite_1.0.2     xtable_1.8-4        DBI_1.1.0           magrittr_1.5        units_0.6-6        
[57] scales_1.0.0        KernSmooth_2.23-15  cli_2.0.2           stringi_1.4.6       promises_1.1.1      remotes_2.1.1       leaflet_2.0.3      
[64] xml2_1.2.0          xts_0.12-0          tools_3.6.0         leafem_0.1.1        glue_1.4.1          purrr_0.3.4         crosstalk_1.1.0.1  
[71] processx_3.4.2      parallel_3.6.0      fastmap_1.0.1       colorspace_1.4-1    classInt_0.4-3     
> MODIStsp(
+   gui = FALSE,
+   options_file = "/mnt/CEPH_PROJECTS/ECOPOTENTIAL/IT-GRANPARADISO/Stephanie/Data/NDVI/MOD09GQ/MODIStsp_GQ_online_linux.json",
+  # options_file = "//usr/local/lib/R/site-library/MODIStsp/testdata/test_MOD13A2.json",
+  # spatial_file_path = NULL,
+ #  scroll_window = FALSE,
+  # test = NULL,
+ #  n_retries = 20,
+   verbose = TRUE
+ )
MODIStsp would like to save a "MODIStsp_Previous.json" file 
containing information about its last successfull run in the folder 
`.../your-r-library/MODIStsp/ExtData/.` 

 Do you authorize this? 

1: Yes - "MODIStsp_Previous.json" will be saved permanently and updated after
after each successfull run of the tool. You will not see this message anymore.

2: No - Previous options will be written to `tempdir` and will be lost when you
exit `R`. You will see this message at each `MODIStsp` execution.

Choice (1/2): 2
GDAL version in use: 3.0.4
[Tue Jun 23 07:20:37 2020] MODIStsp --> Starting processing
[Tue Jun 23 07:20:37 2020] Accessing http server at:  https://e4ftl01.cr.usgs.gov/MOLT/MOD09GQ.006/
[Tue Jun 23 07:20:48 2020] Retrieving list of available ` Terra ` Files for Year 2011
Loading required namespace: gWidgetsRGtk2
Failed with error:  ‘there is no package called ‘gWidgetsRGtk2’’
[Tue Jun 23 07:20:52 2020] HDF File: MOD09GQ.A2011001.h18v04.006.2015216011131.hdf already exists on your system. Skipping download!
[Tue Jun 23 07:20:52 2020] [Tue Jun 23 07:20:52 2020] 1 files for date: 2011.01.01 were successfully downloaded!
[Tue Jun 23 07:20:52 2020] Processing and Reprojecting Terra b1_Red files for date: 2011_01_01
Error in MODIStsp_process_bands(out_folder_mod, modislist, outproj_str,  : 
  Internal error in out_res_sel, outproj_str or full_ext. Aborting!

**************************************************************************************************************************************************

I hope this helps. Thanks for the support!

lbusett commented 4 years ago

@przell

You seem to have a quite old version of the "raster" package.

Could you try to update it and see if it solves the issue?

Lorenzo