search

ropensci / MODIStsp

An "R" package for automatic download and preprocessing of MODIS Land Products Time Series
https://docs.ropensci.org/MODIStsp
GNU General Public License v3.0
155 stars 51 forks source link

How to process modis hdf files already available on disk using MODIStsp #142

Closed hgbzzw closed 5 years ago

hgbzzw commented 6 years ago

Hi, I need to download MCD18A1 radiation data from May to Nov. in 2011. However, MCD18A1 data can only be downloaded before 2009 in the https://e4ftl01.cr.usgs.gov/MOTA/MCD18A1.006/. I found the MCD18A1 data in 2011 can be download in the https://ladsweb.modaps.eosdis.nasa.gov/search/order/4/MCD18A1--6. Thus, I want to know how to process MODIS HDF files already available on disk using MODIStsp. Thanks a lot!

lbusett commented 6 years ago

Hi,

you can find some instructions for offline processing at:

https://ropensci.github.io/MODIStsp/articles/faq.html#working-with-already-downloaded-hdf-files

Let us know if you have problems!

hgbzzw commented 6 years ago

I placed all the HDF files (h23-27, V04-06, 15files) in 'G:\VIP\GWR\MCD18A1\HDF',like this

files then set the parameters in MODIStsp GUI like this image then start processing. no results in my folder. image R control shows that image

Thanks a lot!

hgbzzw commented 6 years ago

It didn't do the mosaicing and reprojecting work. I do not know why?

lbusett commented 6 years ago

could you try changing the start / end dates to 2011-01-01 and 2011-12-31 ?

hgbzzw commented 6 years ago

I changed the start/end dates to 2011-01-01 and 2011-12-31, it also didn't work.

lbusett commented 6 years ago

Sorry it did not work. Was just a wild guess. I'll look into this more in detail, but considee that i will be on holiday for the next couple of week, so it will take some time.

hgbzzw commented 6 years ago

Thanks a lot! waiting for you.

webb-e commented 5 years ago

Was this ever resolved? I am experiencing the same problem.

MODIStsp() GDAL version in use: 2.2.3

(rsession.exe:12596): Gtk-WARNING **: Failed to set text from markup due to error parsing markup: Error on line 1 char 87: Element 'markup' was closed, but the currently open element is 'span'

(rsession.exe:12596): Gtk-WARNING **: Failed to set text from markup due to error parsing markup: Error on line 1 char 87: Element 'markup' was closed, but the currently open element is 'span'

(rsession.exe:12596): Gtk-WARNING **: Failed to set text from markup due to error parsing markup: Error on line 1 char 91: Element 'markup' was closed, but the currently open element is 'span'

(rsession.exe:12596): Gtk-WARNING **: Failed to set text from markup due to error parsing markup: Error on line 1 char 91: Element 'markup' was closed, but the currently open element is 'span' [Tue Feb 19 08:25:25 2019] MODIStsp --> Starting processing [Tue Feb 19 08:26:29 2019] Accessing http server at: https://e4ftl01.cr.usgs.gov/MOTA/MCD64A1.006/ [Tue Feb 19 08:26:29 2019] Retrieving list of available Combined Files for Year 2000 [Tue Feb 19 08:26:29 2019] Retrieving list of available Combined Files for Year 2001 [Tue Feb 19 08:26:29 2019] Retrieving list of available Combined Files for Year 2002 [Tue Feb 19 08:26:30 2019] Retrieving list of available Combined Files for Year 2003 [Tue Feb 19 08:26:30 2019] Retrieving list of available Combined Files for Year 2004 [Tue Feb 19 08:26:30 2019] Retrieving list of available Combined Files for Year 2005 [Tue Feb 19 08:26:30 2019] Retrieving list of available Combined Files for Year 2006 [Tue Feb 19 08:26:30 2019] Retrieving list of available Combined Files for Year 2007 [Tue Feb 19 08:26:30 2019] Retrieving list of available Combined Files for Year 2008 [Tue Feb 19 08:26:31 2019] Retrieving list of available Combined Files for Year 2009 [Tue Feb 19 08:26:31 2019] Retrieving list of available Combined Files for Year 2010 [Tue Feb 19 08:26:31 2019] Retrieving list of available Combined Files for Year 2011 [Tue Feb 19 08:26:31 2019] Retrieving list of available Combined Files for Year 2012 [Tue Feb 19 08:26:31 2019] Retrieving list of available Combined Files for Year 2013 [Tue Feb 19 08:26:31 2019] Retrieving list of available Combined Files for Year 2014 [Tue Feb 19 08:26:32 2019] Retrieving list of available Combined Files for Year 2015 [Tue Feb 19 08:26:32 2019] Retrieving list of available Combined Files for Year 2016 [Tue Feb 19 08:26:32 2019] Retrieving list of available Combined Files for Year 2017 [Tue Feb 19 08:26:32 2019] Retrieving list of available Combined Files for Year 2018 [Tue Feb 19 08:26:32 2019] Creating Virtual Files and R time series for layer Burn_Date [Tue Feb 19 08:26:32 2019] Total Processing Time: 1.12606896559397 [Tue Feb 19 08:26:32 2019] MODIStsp processed files are in: D:\MODIS_burns [Tue Feb 19 08:26:32 2019] Original downloaded MODIS HDF files are in: D:\MODIS_burns_originalHDF

ex1

lbusett commented 5 years ago

Hi, 

completely forgot about this...sorry @hgbzzw

@webb-e so, you have some HDFs in D:/MODIS_burns_originalHDF and you want to run the processing offline, is that correct? What exactly happens when you try?

Lorenzo

webb-e commented 5 years ago

That is correct. MODIStsp creates a folder called "Burned_Monthly_500m_v6" in the folder where I asked it to save the processed tiff files, but the folder is empty. The R output is above.

I should note that the program only runs for a few seconds before ending, so I don't think it starts processing anything.

lbusett commented 5 years ago

Yeah, that's probable. May I ask you to save your options to JSON (using the save options button in the GUI) , open the JSON file in a text editor and cut and paste it here ? That way, I can more easily replicate your analysis.

webb-e commented 5 years ago

{ "sel_prod": "Burned_Monthly_500m (MCD64A1)", "sensor": "Combined", "prod_version": "6", "start_date": "2000-11-01", "end_date": "2018-12-31", "bandsel": [1, 0, 0, 0, 0], "indexes_bandsel": 0, "quality_bandsel": [0, 0, 0, 0, 0], "start_x": 4, "end_x": 26, "start_y": 1, "end_y": 3, "user": "##### "password": ##### "use_aria": false, "download_server": "offline", "download_range": "full", "proj": "Native", "output_proj4": "+proj=sinu +lon_0=0 +x_0=0 +y_0=0 +a=6371007.181 +b=6371007.181 +units=m +no_defs", "out_res_sel": "Native", "out_res": "Native", "full_ext": "Select MODIS Tiles", "resampling": "near", "out_format": "GTiff", "ts_format": "R rasterStack", "compress": "None", "nodata_change": "No", "scale_val": "No", "delete_hdf": "No", "reprocess": "No", "bbox": ["NA", "NA", "NA", "NA"], "out_folder": "D:\MODIS_burns", "out_folder_mod": "D:\MODIS_burns_originalHDF", "MODIStspVersion": "1.3.7", "custom_indexes": [] }

lbusett commented 5 years ago

@webb-e

sorry for the hiatus. This should be now fixed on the "devel" branch. Could you test it?

Lorenzo

webb-e commented 5 years ago

Hmm. So, this worked for the files in 2001 and 2002, but not for any of the other files. Here is a warning I got:

50: In system(cmd, intern = TRUE) : running command '"C:\OSGeo4W64\bin\gdal_translate.exe" -a_nodata "-1" -ot "Int16" -of "GTiff" -a_srs "+proj=sinu +lon_0=0 +x_0=0 +y_0=0 +a=6371007.181 +b=6371007.181 +units=m +no_defs" -co "COMPRESS=None" "C:\Users\webbe\AppData\Local\Temp\RtmpY9l5IB/mstp_temp\file1f5c7bfe4.vrt" "D:\MODIS_burns/Burned_Monthly_500m_v6/Burn_Date/MCD64A1_Burn_Date_2004_336.tif"' had status 1

(but those warnings might be because there were no burns at that date? the file exists though) Looking back, it looks like I also got this warning:


Input file size is 50400, 48000...10...20...30...40...ERROR 4: HDF4_EOS:EOS_GRID:"D:\MODIS_burns_originalHDF/MCD64A1.A2017121.h21v03.006.2017192063953.hdf":MOD_Grid_Monthly_500m_DB_BA:Burn Date' does not exist in the file system, and is not recognized as a supported dataset name.

| >

lbusett commented 5 years ago

That's strange. I tested it on 2010 data and it worked. Maybe you have some corrupted / incomplete HDF. Could you:

1) check if file "D:\MODIS_burns_originalHDF/MCD64A1.A2017121.h21v03.006.2017192063953.hdf" exists?

2) open a terminal and run:

"C:\OSGeo4W64\bin\gdalinfo.exe" "D:\MODIS_burns_originalHDF/MCD64A1.A2017121.h21v03.006.2017192063953.hdf"

(It's a single command - no carriage return)

webb-e commented 5 years ago

I tried it on only 2010 data with no luck. Here is the output for my entire session

> library(MODIStsp)
> MODIStsp()
Loading required package: gWidgetsRGtk2
Loading required package: RGtk2
Loading required package: gWidgets
Loading required package: cairoDevice
[Thu Mar 07 06:23:42 2019] Welcome to MODIStsp!We will now search for a valid GDAL installation - pleasewait! (this will happen only once)
GDAL version in use: 2.3.2

(rsession.exe:9144): Gtk-WARNING **: Failed to set text from markup due to error parsing markup: Error on line 1 char 82: Element 'markup' was closed, but the currently open element is 'span'

(rsession.exe:9144): Gtk-WARNING **: Failed to set text from markup due to error parsing markup: Error on line 1 char 82: Element 'markup' was closed, but the currently open element is 'span'

(rsession.exe:9144): Gtk-WARNING **: Failed to set text from markup due to error parsing markup: Error on line 1 char 87: Element 'markup' was closed, but the currently open element is 'span'

(rsession.exe:9144): Gtk-WARNING **: Failed to set text from markup due to error parsing markup: Error on line 1 char 87: Element 'markup' was closed, but the currently open element is 'span'

(rsession.exe:9144): Gtk-WARNING **: Failed to set text from markup due to error parsing markup: Error on line 1 char 83: Element 'markup' was closed, but the currently open element is 'span'

(rsession.exe:9144): Gtk-WARNING **: Failed to set text from markup due to error parsing markup: Error on line 1 char 83: Element 'markup' was closed, but the currently open element is 'span'

(rsession.exe:9144): Gtk-WARNING **: Failed to set text from markup due to error parsing markup: Error on line 1 char 85: Element 'markup' was closed, but the currently open element is 'span'

(rsession.exe:9144): Gtk-WARNING **: Failed to set text from markup due to error parsing markup: Error on line 1 char 85: Element 'markup' was closed, but the currently open element is 'span'
[Thu Mar 07 06:27:27 2019] MODIStsp --> Starting processing
[Thu Mar 07 06:28:32 2019] Accessing http server at:  https://e4ftl01.cr.usgs.gov/MOTA/MCD64A1.006/
[Thu Mar 07 06:28:32 2019] Retrieving list of available ` Combined ` Files for Year 2010
[Thu Mar 07 06:28:33 2019] HDF File: MCD64A1.A2010305.h09v02.006.2017012031115.hdf already exists on your system. Skipping download!
[Thu Mar 07 06:28:33 2019] HDF File: MCD64A1.A2010305.h10v02.006.2017012034002.hdf already exists on your system. Skipping download!
[Thu Mar 07 06:28:33 2019] HDF File: MCD64A1.A2010305.h11v02.006.2017012044442.hdf already exists on your system. Skipping download!
[Thu Mar 07 06:28:33 2019] HDF File: MCD64A1.A2010305.h12v02.006.2017012060858.hdf already exists on your system. Skipping download!
[Thu Mar 07 06:28:33 2019] HDF File: MCD64A1.A2010305.h13v02.006.2017012060854.hdf already exists on your system. Skipping download!
[Thu Mar 07 06:28:33 2019] HDF File: MCD64A1.A2010305.h14v02.006.2017012035142.hdf already exists on your system. Skipping download!
[Thu Mar 07 06:28:33 2019] HDF File: MCD64A1.A2010305.h15v02.006.2017012035934.hdf already exists on your system. Skipping download!
[Thu Mar 07 06:28:33 2019] HDF File: MCD64A1.A2010305.h16v02.006.2017012035108.hdf already exists on your system. Skipping download!
[Thu Mar 07 06:28:33 2019] HDF File: MCD64A1.A2010305.h17v02.006.2017012042700.hdf already exists on your system. Skipping download!
[Thu Mar 07 06:28:33 2019] HDF File: MCD64A1.A2010305.h18v02.006.2017012052517.hdf already exists on your system. Skipping download!
[Thu Mar 07 06:28:33 2019] HDF File: MCD64A1.A2010305.h19v02.006.2017012052506.hdf already exists on your system. Skipping download!
[Thu Mar 07 06:28:33 2019] HDF File: MCD64A1.A2010305.h20v02.006.2017012042936.hdf already exists on your system. Skipping download!
[Thu Mar 07 06:28:33 2019] HDF File: MCD64A1.A2010305.h21v02.006.2017012034303.hdf already exists on your system. Skipping download!
[Thu Mar 07 06:28:33 2019] HDF File: MCD64A1.A2010305.h22v02.006.2017012042527.hdf already exists on your system. Skipping download!
[Thu Mar 07 06:28:34 2019] HDF File: MCD64A1.A2010305.h23v02.006.2017012035613.hdf already exists on your system. Skipping download!
[Thu Mar 07 06:28:34 2019] HDF File: MCD64A1.A2010305.h24v02.006.2017012060509.hdf already exists on your system. Skipping download!
[Thu Mar 07 06:28:34 2019] HDF File: MCD64A1.A2010305.h25v02.006.2017012040140.hdf already exists on your system. Skipping download!
[Thu Mar 07 06:28:34 2019] HDF File: MCD64A1.A2010305.h26v02.006.2017012035017.hdf already exists on your system. Skipping download!
[Thu Mar 07 06:28:34 2019] HDF File: MCD64A1.A2010305.h06v03.006.2017012031138.hdf already exists on your system. Skipping download!
[Thu Mar 07 06:28:34 2019] HDF File: MCD64A1.A2010305.h07v03.006.2017012031138.hdf already exists on your system. Skipping download!
[Thu Mar 07 06:28:34 2019] HDF File: MCD64A1.A2010305.h08v03.006.2017012034549.hdf already exists on your system. Skipping download!
[Thu Mar 07 06:28:34 2019] HDF File: MCD64A1.A2010305.h09v03.006.2017012031116.hdf already exists on your system. Skipping download!
[Thu Mar 07 06:28:34 2019] HDF File: MCD64A1.A2010305.h10v03.006.2017012034007.hdf already exists on your system. Skipping download!
[Thu Mar 07 06:28:34 2019] HDF File: MCD64A1.A2010305.h11v03.006.2017012041935.hdf already exists on your system. Skipping download!
[Thu Mar 07 06:28:34 2019] HDF File: MCD64A1.A2010305.h12v03.006.2017012043006.hdf already exists on your system. Skipping download!
[Thu Mar 07 06:28:34 2019] HDF File: MCD64A1.A2010305.h13v03.006.2017012040601.hdf already exists on your system. Skipping download!
[Thu Mar 07 06:28:34 2019] HDF File: MCD64A1.A2010305.h14v03.006.2017012072356.hdf already exists on your system. Skipping download!
[Thu Mar 07 06:28:34 2019] HDF File: MCD64A1.A2010305.h15v03.006.2017012034514.hdf already exists on your system. Skipping download!
[Thu Mar 07 06:28:34 2019] HDF File: MCD64A1.A2010305.h17v03.006.2017012052211.hdf already exists on your system. Skipping download!
[Thu Mar 07 06:28:35 2019] HDF File: MCD64A1.A2010305.h18v03.006.2017012035541.hdf already exists on your system. Skipping download!
[Thu Mar 07 06:28:35 2019] HDF File: MCD64A1.A2010305.h19v03.006.2017012053141.hdf already exists on your system. Skipping download!
[Thu Mar 07 06:28:35 2019] HDF File: MCD64A1.A2010305.h20v03.006.2017012042936.hdf already exists on your system. Skipping download!
[Thu Mar 07 06:28:35 2019] HDF File: MCD64A1.A2010305.h21v03.006.2017012055030.hdf already exists on your system. Skipping download!
[Thu Mar 07 06:28:35 2019] HDF File: MCD64A1.A2010305.h22v03.006.2017012034242.hdf already exists on your system. Skipping download!
[Thu Mar 07 06:28:35 2019] HDF File: MCD64A1.A2010305.h23v03.006.2017012041922.hdf already exists on your system. Skipping download!
[Thu Mar 07 06:28:35 2019] HDF File: MCD64A1.A2010305.h24v03.006.2017012051758.hdf already exists on your system. Skipping download!
[Thu Mar 07 06:28:35 2019] HDF File: MCD64A1.A2010305.h25v03.006.2017012060259.hdf already exists on your system. Skipping download!
[Thu Mar 07 06:28:35 2019] HDF File: MCD64A1.A2010305.h26v03.006.2017012060253.hdf already exists on your system. Skipping download!
[Thu Mar 07 06:28:35 2019] [Thu Mar 07 06:28:35 2019] 38 files for date: 2010.11.01 were successfully downloaded!
[Thu Mar 07 06:28:38 2019] Processing Terra Burn_Date files for date: 2010_11_01
Checking gdal_installation...
Scanning for GDAL installations...
Checking the gdalUtils_gdalPath option...
GDAL version 2.3.2
GDAL command being used: "C:\OSGeo4W64\bin\gdal_translate.exe" -a_nodata "-1" -ot "Int16" -of "GTiff" -a_srs "+proj=sinu +lon_0=0 +x_0=0 +y_0=0 +a=6371007.181 +b=6371007.181 +units=m +no_defs" -co "COMPRESS=None" "C:\Users\webbe\AppData\Local\Temp\RtmpS8dn05/mstp_temp\file23b84b6d2c26.vrt" "D:\MODIS_burns/Burned_Monthly_500m_v6/Burn_Date/MCD64A1_Burn_Date_2010_305.tif"
Input file size is 50400, 48000...10...20...30...40...ERROR 4: `HDF4_EOS:EOS_GRID:"D:\MODIS_burns_originalHDF/MCD64A1.A2010305.h21v03.006.2017012055030.hdf":MOD_Grid_Monthly_500m_DB_BA:Burn Date' does not exist in the file system, and is not recognized as a supported dataset name.
[Thu Mar 07 06:28:52 2019] HDF File: MCD64A1.A2010335.h09v02.006.2017012034459.hdf already exists on your system. Skipping download!
[Thu Mar 07 06:28:52 2019] HDF File: MCD64A1.A2010335.h10v02.006.2017012055026.hdf already exists on your system. Skipping download!
[Thu Mar 07 06:28:53 2019] HDF File: MCD64A1.A2010335.h11v02.006.2017012053012.hdf already exists on your system. Skipping download!
[Thu Mar 07 06:28:53 2019] HDF File: MCD64A1.A2010335.h12v02.006.2017012034241.hdf already exists on your system. Skipping download!
[Thu Mar 07 06:28:53 2019] HDF File: MCD64A1.A2010335.h13v02.006.2017012060749.hdf already exists on your system. Skipping download!
[Thu Mar 07 06:28:53 2019] HDF File: MCD64A1.A2010335.h14v02.006.2017012041934.hdf already exists on your system. Skipping download!
[Thu Mar 07 06:28:53 2019] HDF File: MCD64A1.A2010335.h15v02.006.2017012053453.hdf already exists on your system. Skipping download!
[Thu Mar 07 06:28:53 2019] HDF File: MCD64A1.A2010335.h16v02.006.2017012062803.hdf already exists on your system. Skipping download!
[Thu Mar 07 06:28:53 2019] HDF File: MCD64A1.A2010335.h17v02.006.2017012065541.hdf already exists on your system. Skipping download!
[Thu Mar 07 06:28:53 2019] HDF File: MCD64A1.A2010335.h18v02.006.2017012055655.hdf already exists on your system. Skipping download!
[Thu Mar 07 06:28:53 2019] HDF File: MCD64A1.A2010335.h19v02.006.2017012050811.hdf already exists on your system. Skipping download!
[Thu Mar 07 06:28:53 2019] HDF File: MCD64A1.A2010335.h20v02.006.2017012050707.hdf already exists on your system. Skipping download!
[Thu Mar 07 06:28:53 2019] HDF File: MCD64A1.A2010335.h21v02.006.2017012063858.hdf already exists on your system. Skipping download!
[Thu Mar 07 06:28:53 2019] HDF File: MCD64A1.A2010335.h22v02.006.2017012080331.hdf already exists on your system. Skipping download!
[Thu Mar 07 06:28:53 2019] HDF File: MCD64A1.A2010335.h23v02.006.2017012065014.hdf already exists on your system. Skipping download!
[Thu Mar 07 06:28:53 2019] HDF File: MCD64A1.A2010335.h24v02.006.2017012053202.hdf already exists on your system. Skipping download!
[Thu Mar 07 06:28:54 2019] HDF File: MCD64A1.A2010335.h25v02.006.2017012034548.hdf already exists on your system. Skipping download!
[Thu Mar 07 06:28:54 2019] HDF File: MCD64A1.A2010335.h26v02.006.2017012050150.hdf already exists on your system. Skipping download!
[Thu Mar 07 06:28:54 2019] HDF File: MCD64A1.A2010335.h06v03.006.2017012041703.hdf already exists on your system. Skipping download!
[Thu Mar 07 06:28:54 2019] HDF File: MCD64A1.A2010335.h07v03.006.2017012041704.hdf already exists on your system. Skipping download!
[Thu Mar 07 06:28:54 2019] HDF File: MCD64A1.A2010335.h08v03.006.2017012055021.hdf already exists on your system. Skipping download!
[Thu Mar 07 06:28:54 2019] HDF File: MCD64A1.A2010335.h09v03.006.2017012044940.hdf already exists on your system. Skipping download!
[Thu Mar 07 06:28:54 2019] HDF File: MCD64A1.A2010335.h10v03.006.2017012061329.hdf already exists on your system. Skipping download!
[Thu Mar 07 06:28:54 2019] HDF File: MCD64A1.A2010335.h11v03.006.2017012051703.hdf already exists on your system. Skipping download!
[Thu Mar 07 06:28:54 2019] HDF File: MCD64A1.A2010335.h12v03.006.2017012041058.hdf already exists on your system. Skipping download!
[Thu Mar 07 06:28:54 2019] HDF File: MCD64A1.A2010335.h13v03.006.2017012052319.hdf already exists on your system. Skipping download!
[Thu Mar 07 06:28:54 2019] HDF File: MCD64A1.A2010335.h14v03.006.2017012050905.hdf already exists on your system. Skipping download!
[Thu Mar 07 06:28:54 2019] HDF File: MCD64A1.A2010335.h15v03.006.2017012052444.hdf already exists on your system. Skipping download!
[Thu Mar 07 06:28:54 2019] HDF File: MCD64A1.A2010335.h17v03.006.2017012054531.hdf already exists on your system. Skipping download!
[Thu Mar 07 06:28:54 2019] HDF File: MCD64A1.A2010335.h18v03.006.2017012055654.hdf already exists on your system. Skipping download!
[Thu Mar 07 06:28:54 2019] HDF File: MCD64A1.A2010335.h19v03.006.2017012050809.hdf already exists on your system. Skipping download!
[Thu Mar 07 06:28:54 2019] HDF File: MCD64A1.A2010335.h20v03.006.2017012050704.hdf already exists on your system. Skipping download!
[Thu Mar 07 06:28:55 2019] HDF File: MCD64A1.A2010335.h21v03.006.2017012050706.hdf already exists on your system. Skipping download!
[Thu Mar 07 06:28:55 2019] HDF File: MCD64A1.A2010335.h22v03.006.2017012054801.hdf already exists on your system. Skipping download!
[Thu Mar 07 06:28:55 2019] HDF File: MCD64A1.A2010335.h23v03.006.2017012065009.hdf already exists on your system. Skipping download!
[Thu Mar 07 06:28:55 2019] HDF File: MCD64A1.A2010335.h24v03.006.2017012062220.hdf already exists on your system. Skipping download!
[Thu Mar 07 06:28:55 2019] HDF File: MCD64A1.A2010335.h25v03.006.2017012034551.hdf already exists on your system. Skipping download!
[Thu Mar 07 06:28:55 2019] HDF File: MCD64A1.A2010335.h26v03.006.2017012044436.hdf already exists on your system. Skipping download!
[Thu Mar 07 06:28:55 2019] [Thu Mar 07 06:28:55 2019] 38 files for date: 2010.12.01 were successfully downloaded!
[Thu Mar 07 06:28:58 2019] Processing Terra Burn_Date files for date: 2010_12_01
Checking gdal_installation...
Scanning for GDAL installations...
Checking the gdalUtils_gdalPath option...
GDAL version 2.3.2
GDAL command being used: "C:\OSGeo4W64\bin\gdal_translate.exe" -a_nodata "-1" -ot "Int16" -of "GTiff" -a_srs "+proj=sinu +lon_0=0 +x_0=0 +y_0=0 +a=6371007.181 +b=6371007.181 +units=m +no_defs" -co "COMPRESS=None" "C:\Users\webbe\AppData\Local\Temp\RtmpS8dn05/mstp_temp\file23b84db15e1.vrt" "D:\MODIS_burns/Burned_Monthly_500m_v6/Burn_Date/MCD64A1_Burn_Date_2010_335.tif"
Input file size is 50400, 48000...10...20...30...40...ERROR 4: `HDF4_EOS:EOS_GRID:"D:\MODIS_burns_originalHDF/MCD64A1.A2010335.h21v03.006.2017012050706.hdf":MOD_Grid_Monthly_500m_DB_BA:Burn Date' does not exist in the file system, and is not recognized as a supported dataset name.
[Thu Mar 07 06:29:11 2019] Creating Virtual Files and R time series for layer Burn_Date
[Thu Mar 07 06:29:11 2019] Total Processing Time: 1.7282917658488
[Thu Mar 07 06:29:11 2019] MODIStsp processed files are in: `D:\MODIS_burns`
[Thu Mar 07 06:29:11 2019] Original downloaded MODIS HDF files are in: `D:\MODIS_burns_originalHDF`
Warning messages:
1: package ‘gWidgetsRGtk2’ was built under R version 3.5.2 
2: package ‘RGtk2’ was built under R version 3.5.2 
3: package ‘gWidgets’ was built under R version 3.5.2 
4: package ‘cairoDevice’ was built under R version 3.5.2 
5: In system(cmd, intern = TRUE) :
  running command '"C:\OSGeo4W64\bin\gdal_translate.exe" -a_nodata "-1" -ot "Int16" -of "GTiff" -a_srs "+proj=sinu +lon_0=0 +x_0=0 +y_0=0 +a=6371007.181 +b=6371007.181 +units=m +no_defs" -co "COMPRESS=None" "C:\Users\webbe\AppData\Local\Temp\RtmpS8dn05/mstp_temp\file23b84b6d2c26.vrt" "D:\MODIS_burns/Burned_Monthly_500m_v6/Burn_Date/MCD64A1_Burn_Date_2010_305.tif"' had status 1
6: In system(cmd, intern = TRUE) :
  running command '"C:\OSGeo4W64\bin\gdal_translate.exe" -a_nodata "-1" -ot "Int16" -of "GTiff" -a_srs "+proj=sinu +lon_0=0 +x_0=0 +y_0=0 +a=6371007.181 +b=6371007.181 +units=m +no_defs" -co "COMPRESS=None" "C:\Users\webbe\AppData\Local\Temp\RtmpS8dn05/mstp_temp\file23b84db15e1.vrt" "D:\MODIS_burns/Burned_Monthly_500m_v6/Burn_Date/MCD64A1_Burn_Date_2010_335.tif"' had status 1

I checked a couple of the hdf files in the way you suggested and the console printed out their metatdata (so this means they are not corrupt?)

webb-e commented 5 years ago

Also, I just ran the data just for 2001-2003 and it did not work. I suspect last time I had some old files in the folder, so in fact when I ran the program yesterday it did not work for any years (based on their file creation date, the 2001-2003 files do not look like they were processed yesterday when I rant the program)

lbusett commented 5 years ago

I wonder if this could be something similar to #114.... On Windows, there seems to be a "limit" to how many tiles we are able to "stitch together". To verify this, could you to :

1) start again the processing, and after it crashes open a terminal and run the command that is giving the error? In practice, it should be something similar to:

"C:\OSGeo4W64\bin\gdal_translate.exe" -a_nodata "-1" -ot "Int16" -of "GTiff" -a_srs "+proj=sinu +lon_0=0 +x_0=0 +y_0=0 +a=6371007.181 +b=6371007.181 +units=m +no_defs" -co "COMPRESS=None" "C:\Users\webbe\AppData\Local\Temp\RtmpS8dn05/mstp_temp\file23b84b6d2c26.vrt" "D:\MODIS_burns/Burned_Monthly_500m_v6/Burn_Date/MCD64A1_Burn_Date_2010_305.tif"

but with a different name for the vrt file.

2) Try the same processing, but on a smaller "extent" . For example, you could change the "start_x" and "end_x" lines of the json options file to :

"start_x": 20, "end_x": 26,

webb-e commented 5 years ago

image

(I used the same vrt file as earlier since I had not closed my R session. I also searched for mstp_temp in all of the C: drive and didn't find anything) image

I tried a smaller extent and it does appear to have worked (there are output files but I have not checked them yet). I used this: "start_x": 4, "end_x": 10, "start_y": 1, "end_y": 3,

However, I will note that earlier I directly downloaded the data and MODIStsp processed the large extent. Actually, it did crash when I used: "start_x": 4, "end_x": 27, "start_y": 1, "end_y": 3,

I'm not sure why, but when I changed the extent to x:26 instead of x:27, it worked. The plan was just to add that last tile in after the fact.

The reason that I don't have the processed data from the first download is because when I went to download that last tile (h27, v3), it replaced all of my previously processed files. I'm new to this and didn't know that was going to happen, so now I am just trying to process the files that I originally downloaded. However, it seems like the fastest option might be to just re-download them, since processing worked that way.

lbusett commented 5 years ago

Ok. Then I'd suggest you to try to process 4/26 1/3 using "http" download option. Consider that even in that case, you will not need to re-download the HDFs if you do not change the HDF output folder. MODIstsp will just check to verify if you already have all the HDFs for the specified spatial/temporal selection.

If it works, good. otherwise, I suggest you to try to do two separate processings (e.g, 4-->12 and 13-->26), (with different output folders) and see if by reducing the number of tiles you can avoid the problem. If it works, you can then merge the results afterward.

webb-e commented 5 years ago

@lbusett I ended up dividing the processing into two parts (4/26 1/2 and 4/27 3) and am now stitching them together. So you are right, this is related to issue 114. Glad I found a workaround. Thank you for your help with this!

lbusett commented 5 years ago

Closing here. Keeping open in #114