MODIS export fails (data is all NaNs)

[lledoux]

I have run many different iterations of gips_export (using my virtual environment /scratch/lledoux/cmsindo/test-venv/ ) of MODIS data that has been correctly produced (so archived data is valid) for a certain area (relatively small, about 3 Ag fields in Indiana /scratch/lledoux/optis/optis_indiana_fields_polys_dissolve.shp) . Throughout these iterations, the data that is exported is all NaNs. This is what I have tried (and the results)"

• using --alltouch does not help

• using a shapefile in a different projection does not help

• the data export correctly (I get valid data) when a larger shapefile is used (same projection as original shapefile, many times the size of the fields) for the same area in Indiana

• using the network-based gips does not help

[bhbraswell]

This appears to be a cookie-cutter type issue, except that when you look at an image it appears that at least a couple pixels should be fully contained within the region.

To clarify, Lindsay has verified that there are valid data in the time and space range of the shape file request. Those values show up with a larger extent (@lledoux was it the same projection?).

[lledoux]

The two shapefiles (the larger and the smaller) are in the same projection (UTM Zone 16N), and the successful export of the data using the larger shapefile did produce data in the same projection as the larger shapefile (the UTM 16N).

[bhbraswell]

I assume the underlying science task was for Optis, is that right? I'll loop in the PI @stephenhagen so he is able to track this.

[lledoux]

Yes, for Optis.

[ircwaves]

@lledoux -- please add the command that you ran to export.

[ircwaves]

@lledoux -- also, what date are you encountering the problem? I randomly grabbed 2017-282, and can see how easy NaNs are to get for this field.

That feature is at the edge of the h11v04 tile, so GIPS' modis/tiles.shpsays "that field is h11v04", and then the export export says "I've got nothing for ya".

[lledoux]

I've tried running: (test-venv) lmelendy@rio:/scratch/lledoux/optis$ gips_project modis -s optis_indiana_fields_polys_dissolve.shp -d 2016-08-01,2017-11-14 -p ndvi8 --outdir . --res 30 30 and

(test-venv) lmelendy@rio:/scratch/lledoux/optis$ gips_project modis -s optis_indiana_fields_polys_dissolve.shp -d 2016-08-01,2017-11-14 -p ndvi8 --outdir . --res 30 30 --alltouch

and both fail

[ircwaves]

OK, so I took this a step further loaded up all the imagery, and Lindsay's geometry. It looks to me like the modis tiles vector geometries are off. Here's (maybe) the code and maybe a clearer picture (all in sinusoidal projection):

# in QGIS console
import os
from glob import glob

ndvi8 = glob('/titan/data/modis6/tiles/h11v05/2017*/*ndvi8.tif')
print(len(ndvi8))
for img in ndvi8:
    iface.addRasterLayer(img, os.path.basename(img))
iface.addVectorLayer('/scratch/icooke/optis_indiana_fields_polys_dissolve_sinu.shp', 'optis_indiana_fields_polys_disolve_sinu', 'ogr')
iface.addVectorLayer('/home/icooke/src/gips/gips/data/modis/tiles.shp', 'tiles', 'ogr')

@bhbraswell it looks like the delta between the edge of imagery for h11v05 and the tiles.shp boundary is 13.3km. Given that, I think the modis tile vector needs to be updated. Do you agree? As it stands, this is a blocker bug to small feature exports within these 13km error regions.

[bhbraswell]

Agree