S2 EVI & SATVI have too many no-data values

[lledoux]

Oh jeeze, I feel like I should have caught this sooner, but- it seems like the EVI Sentinel-2 products have a lot of no data values (-32768) where there shouldn't be any (no data values here are white):

This is the corresponding NDVI image:

[lledoux]

Also- Sentinel-2 SATVI images are all NAN's (-32768):

[ra-tolson]

@lledoux can you point me at the problematic scenes you found? tile & date would be ideal.

[ircwaves]

@ags-tolson let's get together after standup and discuss what you've found so far. I've verified this issue using 19TCH and 2017-283.

[lledoux]

Sorry for the delay! An example is in /titan/data/sentinel2/tiles/48PUS/2017031/

[ra-tolson]

those are great details, thanks to you both.

[ircwaves]

Re-capping our conversation yesterday, It looks like setting the gain on both the input and the output geoimages should address this. When you've implemented and run this, it should be discernible from the gdalinfo -stats ${TIF} output whether the gain is having the desired effect (N.B.: gdalinfo does not use gains nor offsets when computing image statistics).

[ra-tolson]

Tentatively there's been success with SetGain(0.0001) for indices products. Remaining tasks are:

• [x] install SIXS et al on my container to set up for atmo correction
• [x] run rad and ref and confirm they look good post-SetGain
• [x] run surface 'satvi' and 'evi' and confirm they too look good post-SetGain

[ra-tolson]

Wellp, turns out all four of the central products in the driver needed revision. Tentative fix is in branch 433-draft-main-prod-fixes. Once I started adding gains to things it started breaking "downstream" products; next is testing that the indices get reasonable results.

I call them 'draft fixes' because I'm not confident my math is right; I'd really like someone knowledgeable to verify the resulting products.

[ra-tolson]

Place to look for products resulting from that branch:

/scratch/tolson/pub/433-trial-central-prods/

There's some asset jp2s in there too for comparison.

[ircwaves]

Which branch generated these?

[ra-tolson]

It's the one I mentioned above, 433-draft-main-prod-fixes.

[ircwaves]

Sorry. I wish GH had a way to auto ref branches. I stick them in as links for lack of a better way. ../compare/dev...433-draft-main-prod-fixes

[lledoux]

So in order to be able to process correct EVI and SATVI, do I need to have that particular gips branch? I just tried processing both products (on rio, in my virtual environment that is 'up-to-date' with origin/master) yesterday, and I'm still having the same problem...

EVI (for tile 48PTT, 2015365):

• The screenshot of the SATVI for the same tile and date would not be helpful to show, as it is all NaNs.

Here is the reflectance image for reference:

[ircwaves]

@lledoux @bhbraswell -- I think you may need to re-process the 'ref' product in that directory (use --overwrite should update it). If that fixes it, then there is a lurking bug in the guts of the sentinel2 driver.

icooke@north:/titan/data/sentinel2/tiles/48PTT/2015365$ ls -l *ref.tif
-rw-rw-r-- 1 10087 staff 361726198 Oct 25 15:00 48PTT_2015365_S2A_ref.tif

[lledoux]

It turns out that the reflectance images that had already been processed before the EVI and SATVI fix were mucking up some things. Once the ref images are re-processed, the EVI and SATVI look correct.