search

ropensci / MODIStsp

An "R" package for automatic download and preprocessing of MODIS Land Products Time Series
https://docs.ropensci.org/MODIStsp
GNU General Public License v3.0
155 stars 51 forks source link

MODIStsp GDAL No Match #186

Closed RickHeeres closed 4 years ago

RickHeeres commented 4 years ago

Hi,

I'm experiencing problems while downloading MODIS data using MODIStsp() in R.

[Sun Nov 17 17:54:46 2019] Processing Terra LST_Day_1km files for date: 2014_01_01 Error in gdal_chooseInstallation(hasDrivers = of) : No installations match. In addition: Warning message: In system(cmd, intern = TRUE) : running command '"C:\OSGeo4W64\bin\gdalbuildvrt.exe" -sd "1" -srcnodata "0" -vrtnodata "0" "...\RtmpuOc27T/mstp_temp\file412c14fa134e.vrt" "...\RtmpuOc27T/MODIStsp/HDFs/MOD11A1.A2014001.h18v02.006.2016179224152.hdf"' had status 1

This is the error which occurs every time. I checked all the environmental paths, packages and checked the GDAL installation and version in R itself. Everything seems to work fine, but the gdal_chooseInstallation keeps stating there is no match...

Do you have a solution for this? (I already re-installed R, R studio, all packages and OSGeo4W64) I'm using Windows 10.

I can provide more information if needed.

Thank you in advance.

RickHeeres commented 4 years ago

I saw that this issue (warning message) was also experienced by another Windows user before. I also get that message when I start up MODIStsp()

GDAL version in use: 3.0.2

Loading required packages … (everything still fine)

But then...

(rsession.exe:20512): Gtk-WARNING **: Failed to set text from markup due to error parsing markup: Error on line 1 char 61: Element 'markup' was closed, but the currently open element is 'span'

(rsession.exe:20512): Gtk-WARNING **: Failed to set text from markup due to error parsing markup: Error on line 1 char 55: Element 'markup' was closed, but the currently open element is 'span'

And this continues with some other char numbers: 62, 60, 64.

Maybe this can be of help also for solving the problem.

lbusett commented 4 years ago

@RickHeeres

thanks for signaling this. Concerning the first message: did you check if your GDAL installation has the HDF4 driver? You can check it using this command from a terminal:

gdalinfo --formats|grep HDF4

COncerning the warning, it is simething that happens sometimes when the GUI crashes unexpectedly and/or when using a "Old" options file on a newer versionì. Could you try to first remove completely the package, using

remove.packages("MODIStsp")

, restart R/RStudio and try reinstalling?

Lorenzo

RickHeeres commented 4 years ago

This is the result from the terminal:

...res>gdalinfo --formats|grep HDF4 HDF4 -raster- (ros): Hierarchical Data Format Release 4 HDF4Image -raster- (rw+): HDF4 Dataset

I tried to reinstal it (with restarting Rstudio) multiple times. But makes no difference.

Also his message comes up now when I do library(MODIStsp):

Registered S3 method overwritten by 'xts': method from as.zoo.xts zoo

Maybe this causes problems?

lbusett commented 4 years ago

No, that is not a problem. Could you pleasr save your processing options from the gui as json file and cut and paste its contents here?

RickHeeres commented 4 years ago

Yes. Here is both session info and the processing options (just … for password):

[1] ... Software/R/R-3.6.1/library

And:

{ "sel_prod": "Surf_Temp_Daily_005dg (M*D11C1)", "sensor": "Terra", "prod_version": "6", "start_date": "2004-01-01", "end_date": "2016-01-01", "bandsel": [1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], "indexes_bandsel": 0, "quality_bandsel": [1, 1, 0, 0, 0, 1, 1, 0, 0, 0], "start_x": 18, "end_x": 18, "start_y": 2, "end_y": 2, "user": "RickHeeres", "password": "...", "use_aria": false, "download_server": "http", "download_range": "full", "proj": "Native", "output_proj4": "+init=epsg:4008 +proj=longlat +ellps=clrk66 +no_defs", "out_res_sel": "Native", "out_res": "Native", "full_ext": "Define Custom Area", "resampling": "near", "out_format": "GTiff", "ts_format": "R rasterStack", "compress": "None", "nodata_change": "No", "scale_val": "No", "delete_hdf": "No", "reprocess": "No", "bbox": [14.4934, 66.494, 21.7382, 67.8348], "out_folder": "...\Data\Data Layers\MODIS", "out_folder_mod": "...\Data\Data Layers\MODIS", "MODIStspVersion": "1.3.9", "custom_indexes": [] }

lbusett commented 4 years ago

Thanks. As a first try, could you try using full paths for the output folders instead than relative ones and/or using paths with no spaces?

RickHeeres commented 4 years ago

In both the "out_folder" as the mod I changed the first part manually when sending it here. There is a full path which is D:\Wolverines_Sweden\Thesis\Data\Data_Layers\MODIS now.

Restarted R, installed all packages and tried with paths mentioned above. But still not working. But now in the processing options the folders are mentioned like this:

"out_folder": "D:\\Wolverines_Sweden\\Thesis\Data\\Data_Layers\\MODIS", "out_folder_mod": "D:\\Wolverines_Sweden\\Thesis\Data\\Data_Layers\\MODIS"

lbusett commented 4 years ago

@RickHeeres

I managed to replicate the issue on a Windows machine with GDAL3/PROJ6 installed. So it is either a GDAL3/PROJ6 issue, or a windows one. I'll keep you posted.

Lorenzo

supernitt commented 4 years ago

Hi, I also have a problem associated with gdal. Previously, the MODIStsp works well. But, I recently reinstall Windows 10 OS and any software included OSGEO4W (For GDAL installation), R 3.5.3, and MODIStsp library. I downloaded the NDVI product and while processing (reprojecting), I have got the error message as follows;

[Wed Nov 20 21:39:57 2019] MODIStsp --> Starting processing [Wed Nov 20 21:39:57 2019] Accessing http server at: https://e4ftl01.cr.usgs.gov/MOLT/MOD13Q1.006/ [Wed Nov 20 21:39:59 2019] Retrieving list of available Terra Files for Year 2019 [Wed Nov 20 21:40:14 2019] HDF File: MOD13Q1.A2019001.h27v07.006.2019024152658.hdf already exists on your system. Skipping download! [Wed Nov 20 21:40:14 2019] HDF File: MOD13Q1.A2019001.h28v07.006.2019024152716.hdf already exists on your system. Skipping download! [Wed Nov 20 21:40:15 2019] HDF File: MOD13Q1.A2019001.h27v08.006.2019024152651.hdf already exists on your system. Skipping download! [Wed Nov 20 21:40:16 2019] HDF File: MOD13Q1.A2019001.h28v08.006.2019024152603.hdf already exists on your system. Skipping download! [Wed Nov 20 21:40:16 2019] [Wed Nov 20 21:40:16 2019] 4 files for date: 2019.01.01 were successfully downloaded! [Wed Nov 20 21:40:20 2019] Processing and Reprojecting Terra NDVI files for date: 2019_01_01 Error in gdal_chooseInstallation(hasDrivers = of) : No installations match. In addition: Warning message: In system(cmd, intern = TRUE) : running command '"C:\OSGeo4W64\bin\gdalbuildvrt.exe" -sd "1" -srcnodata "-3000" -vrtnodata "-3000" "C:\Users\SUPERN~1\AppData\Local\Temp\RtmpkrL9aN/mstp_temp\file4745376bc9.vrt" "D:\supernitt\Work\Depa\MODIS/MOD13Q1.A2019001.h27v07.006.2019024152658.hdf" "D:\supernitt\Work\Depa\MODIS/MOD13Q1.A2019001.h28v07.006.2019024152716.hdf" "D:\supernitt\Work\Depa\MODIS/MOD13Q1.A2019001.h27v08.006.2019024152651.hdf" "D:\supernitt\Work\Depa\MODIS/MOD13Q1.A2019001.h28v08.006.2019024152603.hdf"' had status 1

How can I fix this issue? Could you please give me any suggestion?

Thank you so much Nitt

lbusett commented 4 years ago

Hi @supernitt 

I am on it - hopefully I will soon have at least a temporary fix!

Lorenzo

lbusett commented 4 years ago

@RickHeeres @supernitt

I have a tentative patch - could you try installing the gdalUtilities package, and then install from github: 

install.packages("gdalUtilities")
remotes::install_github("ropensci/MODIStsp", ref = "feature/gdalutilities")
supernitt commented 4 years ago 
install.packages("gdalUtilities")
remotes::install_github("ropensci/MODIStsp", ref = "feature/gdalutilities")

Thank you I have installed the package, but still displayed some errors in remote install process

Warning: Failed to load RGtk2 dynamic library, attempting to install it. Please install GTK+ from http://ftp.gnome.org/pub/gnome/binaries/win32/gtk+/2.22/gtk+-bundle_2.22.1-20101227_win32.zip If the package still does not load, please ensure that GTK+ is installed and that it is on your PATH environment variable IN ANY CASE, RESTART R BEFORE TRYING TO LOAD THE PACKAGE AGAIN Error: package or namespace load failed for 'MODIStsp': .onLoad failed in loadNamespace() for 'RGtk2', details: call: NULL error: node stack overflow Error: loading failed Execution halted *** arch - x64 ERROR: loading failed for 'i386'

Thank you Nitt

lbusett commented 4 years ago

Hi nik,

Please try removing MODIStsp, and then installing again from a fresh session. If you still have problems with RGtk2, try installing that package beforehand.

See also

https://ropensci.github.io/MODIStsp/articles/installation.html

RickHeeres commented 4 years ago

@lbusett

It is working now. No messages what so ever.

Thank you so much!

supernitt commented 4 years ago

Hi nik,

Please try removing MODIStsp, and then installing again from a fresh session. If you still have problems with RGtk2, try installing that package beforehand.

See also

https://ropensci.github.io/MODIStsp/articles/installation.html

Hi @lbusett ,

It is working now. After remote installing, I downloaded GTK+ directly and set the environment path (both win 64 and win 32.)

The installation of MODIStsp needs the GTK+ for win32. Use the link warning in the remote installation to download the zip file.

http://ftp.gnome.org/pub/gnome/binaries/win32/gtk+/2.22/gtk+-bundle_2.22.1-20101227_win32.zip

Thank you Nitt

lbusett commented 4 years ago

@RickHeeres @supernitt

Glad it works now. However, consider that this is only a temporary fix, based on "piggybacking" on the GDAL utilities provided in sf, that currently are still at version 2.4 something. I'll work on proper support for GDAL 3/PROJ6 in the coming days.

Lorenzo

thechibo commented 4 years ago

Dear Lorenzo,

Congratulations on the great work. This package is really going to help me with my thesis. I had the same issues as above. I tried what you suggested and it worked. I still get the Gtk warning though, Should I be concerned? Is there any chance it distorts the Geotiffs I download? Thank you in advance.

John

lbusett commented 4 years ago

@thechibo

the warning should not change anything, and may/should go away if you completely remove the package and reinstall.

However, be careful: I cannot guarantee that this fix always works correctly: transition towards GDAL3 is proving a mess, and if you update packages such as sp and rgdal in the near future you could get into trouble. I am working on a more reliable solution in these days. I'd suggest you to check proper "positioning" of some of the outputs to verify all is working properly, in particular if you are reprojecting the outputs (which I do NOT encourage to do).

Lorenzo