modis no data values

[mcorbiere]

Product 'ndvi8' has multiple no data values (reads as -32768). Product 'indices' also has this problem in certain bands (reads as 32767).

mcorbier@nile:$ gdalinfo /scratch/mcorbiere/forestdisturbance/Modis_tiles/h12v04/mod-ndvi-h12v04/h12v04_2007001_MOD_ndvi8.tif -stats Driver: GTiff/GeoTIFF Files: /scratch/mcorbiere/forestdisturbance/Modis_tiles/h12v04/mod-ndvi-h12v04/h12v04_2007001_MOD_ndvi8.tif Size is 4800, 4800 ... Band 1 Block=4800x1 Type=Int16, ColorInterp=Gray Description = ndvi Minimum=-32760.000, Maximum=32766.000, Mean=1448.274, StdDev=6820.874 NoData Value=-32768 Unit Type: other Offset: 0, Scale:9.99999974737875e-05 Metadata: STATISTICS_MAXIMUM=32766 STATISTICS_MEAN=1448.274008222 STATISTICS_MINIMUM=-32760 STATISTICS_STDDEV=6820.8741990155

[ircwaves]

Per conversation between @mcorbiere and myself, this is indicating that bad data are created in the process of computing the ndvi8 product for MODIS. Not indicating that there are actually multiple nodata values.

[bhbraswell]

For better or worse this product happens almost entirely inside gippy. I think it's possible that the nodata value for reflectance is not being set properly.

            # NDVI (8-day) - Terra only
            if val[0] == "ndvi8":
                VERSION = "1.0"
                meta['VERSION'] = VERSION
                sensor = 'MOD'
                fname = '%s_%s_%s' % (bname, sensor, key)

                refl = gippy.GeoImage(allsds)
                refl.SetBandName("RED", 1)
                refl.SetBandName("NIR", 2)
                refl.SetNoData(-28762)

                fouts = dict(Indices(refl, {'ndvi': fname}, meta))
                imgout = gippy.GeoImage(fouts['ndvi'])

Should it really be 28762? If not, then for example if there was a case when RED=-32768 and NIR is some reasonable value (like 0.2), then NDVI would end up being close to 32768. Flip NIR and RED, and you'd get a case where NDVI would end up being close to -32768.

[ircwaves]

-28762 looks like a strange typo of -32768, to me.

[ircwaves]

Should it be 32767 based on this:

icooke@north:~$ gdalinfo \
  HDF4_EOS:EOS_GRID:"/titan/data/modis/tiles/h10v05/2009017/MCD43A4.A2009017.h10v05.005.2009036021552.hdf":MOD_Grid_BRDF:Nadir_Reflectance_Band1 \
  | grep NoData
  NoData Value=32767

[bhbraswell]

Voila

On Jan 14, 2016, at 3:46 PM, ircwaves notifications@github.com wrote:

Should it be 32767 based on this:

icooke@north:~$ gdalinfo \ HDF4_EOS:EOS_GRID:"/titan/data/modis/tiles/h10v05/2009017/MCD43A4.A2009017.h10v05.005.2009036021552.hdf":MOD_Grid_BRDF:Nadir_Reflectance_Band1 \ | grep NoData NoData Value=32767 — Reply to this email directly or view it on GitHub https://github.com/Applied-GeoSolutions/gips/issues/31#issuecomment-171774693.

[mcorbiere]

That would be great if that solves the ndvi8 problem, but I don't think it will solve the indices problem since I can't find a 'typo' like that in the indices section of modis.py

mcorbier@nile:/scratch/mcorbiere/forestdisturbance/Modis_tiles/h12v04/mcd-indices-h12v04$ gdalinfo h12v04_2009177_MCD_indices.tif -stats
Driver: GTiff/GeoTIFF
Files: h12v04_2009177_MCD_indices.tif
Size is 2400, 2400
Pixel Size = (463.312716527916507,-463.312716527500129)
...
Band 1 Block=2400x1 Type=Int16, ColorInterp=Gray
  Description = NDVI
  Minimum=-10001.000, Maximum=9999.000, Mean=5426.168, StdDev=4077.892
  NoData Value=32767
  Offset: 0,   Scale:9.99999974737875e-05
  Metadata:
    STATISTICS_MAXIMUM=9999
    STATISTICS_MEAN=5426.1677114303
    STATISTICS_MINIMUM=-10001
    STATISTICS_STDDEV=4077.8917803624
Band 2 Block=2400x1 Type=Int16, ColorInterp=Undefined
  Description = LSWI
  Minimum=-10001.000, Maximum=9979.000, Mean=2055.597, StdDev=2490.112
  NoData Value=32767
  Offset: 0,   Scale:9.99999974737875e-05
  Metadata:
    STATISTICS_MAXIMUM=9979
    STATISTICS_MEAN=2055.5965027778
    STATISTICS_MINIMUM=-10001
    STATISTICS_STDDEV=2490.1123738488
Band 3 Block=2400x1 Type=Int16, ColorInterp=Undefined
  Description = VARI
  Minimum=-32768.000, Maximum=32765.000, Mean=1769.367, StdDev=6734.199
  NoData Value=32767
  Offset: 0,   Scale:9.99999974737875e-05
  Metadata:
    STATISTICS_MAXIMUM=32765
    STATISTICS_MEAN=1769.3668487182
    STATISTICS_MINIMUM=-32768
    STATISTICS_STDDEV=6734.1993114719
Band 4 Block=2400x1 Type=Int16, ColorInterp=Undefined
  Description = BRGT
  Minimum=26.000, Maximum=9999.000, Mean=1898.288, StdDev=3165.329
  NoData Value=32767
  Offset: 0,   Scale:9.99999974737875e-05
  Metadata:
    STATISTICS_MAXIMUM=9999
    STATISTICS_MEAN=1898.2883517361
    STATISTICS_MINIMUM=26
    STATISTICS_STDDEV=3165.3291494447
Band 5 Block=2400x1 Type=Int16, ColorInterp=Undefined
  Description = SATVI
  Minimum=-14978.000, Maximum=9970.000, Mean=1027.257, StdDev=2765.962
  NoData Value=32767
  Offset: 0,   Scale:9.99999974737875e-05
  Metadata:
    STATISTICS_MAXIMUM=9970
    STATISTICS_MEAN=1027.2565833333
    STATISTICS_MINIMUM=-14978
    STATISTICS_STDDEV=2765.9619199185
mcorbier@nile:/scratch/mcorbiere/forestdisturbance/Modis_tiles/h12v04/mcd-indices-h12v04$

[bhbraswell]

I’m pretty sure you wouldn’t see it there. In the driver code, that value was provided as the missing value for the reflectance data (not the indices). Then gippy indices goes off and does its thing, which produces all sorts of weird values due to what Ian and I just determined. What’s interesting is that the output image doesn’t appear to have a missing value set at all, yet the file says that it is -32767. I assume gippy sets it to that as default. Nevertheless I think it doesn’t really matter. The driver code just needs to specify the appropriate value for reflectance missing value and that should fix the problem.

On Jan 14, 2016, at 3:53 PM, mcorbiere notifications@github.com wrote:

That would be great if that solves the ndvi8 problem, but I don't think it will solve the indices problem since I can't find a 'typo' like that in the indices section of modis.py

mcorbier@nile:/scratch/mcorbiere/forestdisturbance/Modis_tiles/h12v04/mcd-indices-h12v04$ gdalinfo h12v04_2009177_MCD_indices.tif -stats Driver: GTiff/GeoTIFF Files: h12v04_2009177_MCD_indices.tif Size is 2400, 2400 Coordinate System is: PROJCS["unnamed", GEOGCS["Unknown datum based upon the custom spheroid", DATUM["Not_specified_based_on_custom_spheroid", SPHEROID["Custom spheroid",6371007.181,0]], PRIMEM["Greenwich",0], UNIT["degree",0.0174532925199433]], PROJECTION["Sinusoidal"], PARAMETER["longitude_of_center",0], PARAMETER["false_easting",0], PARAMETER["false_northing",0], UNIT["metre",1, AUTHORITY["EPSG","9001"]]] Origin = (-6671703.117999999783933,5559752.598333000205457) Pixel Size = (463.312716527916507,-463.312716527500129) Metadata: AREA_OR_POINT=Area Image Structure Metadata: INTERLEAVE=PIXEL Corner Coordinates: Upper Left (-6671703.118, 5559752.598) ( 93d20'36.35"W, 50d 0' 0.00"N) Lower Left (-6671703.118, 4447802.079) ( 78d19'27.97"W, 40d 0' 0.00"N) Upper Right (-5559752.598, 5559752.598) ( 77d47'10.29"W, 50d 0' 0.00"N) Lower Right (-5559752.598, 4447802.079) ( 65d16'13.31"W, 40d 0' 0.00"N) Center (-6115727.858, 5003777.339) ( 77d46'54.29"W, 45d 0' 0.00"N) Band 1 Block=2400x1 Type=Int16, ColorInterp=Gray Description = NDVI Minimum=-10001.000, Maximum=9999.000, Mean=5426.168, StdDev=4077.892 NoData Value=32767 Offset: 0, Scale:9.99999974737875e-05 Metadata: STATISTICS_MAXIMUM=9999 STATISTICS_MEAN=5426.1677114303 STATISTICS_MINIMUM=-10001 STATISTICS_STDDEV=4077.8917803624 Band 2 Block=2400x1 Type=Int16, ColorInterp=Undefined Description = LSWI Minimum=-10001.000, Maximum=9979.000, Mean=2055.597, StdDev=2490.112 NoData Value=32767 Offset: 0, Scale:9.99999974737875e-05 Metadata: STATISTICS_MAXIMUM=9979 STATISTICS_MEAN=2055.5965027778 STATISTICS_MINIMUM=-10001 STATISTICS_STDDEV=2490.1123738488 Band 3 Block=2400x1 Type=Int16, ColorInterp=Undefined Description = VARI Minimum=-32768.000, Maximum=32765.000, Mean=1769.367, StdDev=6734.199 NoData Value=32767 Offset: 0, Scale:9.99999974737875e-05 Metadata: STATISTICS_MAXIMUM=32765 STATISTICS_MEAN=1769.3668487182 STATISTICS_MINIMUM=-32768 STATISTICS_STDDEV=6734.1993114719 Band 4 Block=2400x1 Type=Int16, ColorInterp=Undefined Description = BRGT Minimum=26.000, Maximum=9999.000, Mean=1898.288, StdDev=3165.329 NoData Value=32767 Offset: 0, Scale:9.99999974737875e-05 Metadata: STATISTICS_MAXIMUM=9999 STATISTICS_MEAN=1898.2883517361 STATISTICS_MINIMUM=26 STATISTICS_STDDEV=3165.3291494447 Band 5 Block=2400x1 Type=Int16, ColorInterp=Undefined Description = SATVI Minimum=-14978.000, Maximum=9970.000, Mean=1027.257, StdDev=2765.962 NoData Value=32767 Offset: 0, Scale:9.99999974737875e-05 Metadata: STATISTICS_MAXIMUM=9970 STATISTICS_MEAN=1027.2565833333 STATISTICS_MINIMUM=-14978 STATISTICS_STDDEV=2765.9619199185 mcorbier@nile:/scratch/mcorbiere/forestdisturbance/Modis_tiles/h12v04/mcd-indices-h12v04$

— Reply to this email directly or view it on GitHub https://github.com/Applied-GeoSolutions/gips/issues/31#issuecomment-171776270.

[ircwaves]

branch created 31-modis-ndv

[ircwaves]

@bhbraswell Looks like -28762 is the right NDV for MOD09Q1*. Not sure why I had grabbed a MCD43A file before. Anyway, I don't see any evidence of multiple NoDataValues, or issues with the output.

• https://lpdaac.usgs.gov/dataset_discovery/modis/modis_products_table/mod09q1

Per your mention:

I think it's possible that the nodata value for reflectance is not being set properly.

The NoData value does appear to be correct in the HDF, so the line setting it is unnecessary. The histogram of NDVI8 doesn't seem totally unreasonable to me, given the snowy cloudy winter composite. Interested in your opinion?

[bhbraswell]

Absolutely reasonable. Sorry for the confusion regarding MCD43/MOD09. I have very little experience with MOD09.

[ircwaves]

I believe the end result of this was that it was not an issue.