Reflection in GHG-EMIT Plume data

[siddharth0248]

Some of the COGs have weird reflections displayed on the dashboard.

[siddharth0248]

@aboydnw please look into this bug

[aboydnw]

Thanks @siddharth0248 @danielfdsilva reproduction steps for the team:

1. Open GHG center dashboard
2. Go to EMIT Methane Emissions Plume dataset
3. Click to Explore Data
4. Change date to August 14th to see an example, although other dates have a similar issue
5. After the map zooms in, zoom out slightly and you will notice a shadow that was originally just outside the border of the screen

@siddharth0248 let me know if I got any of those steps incorrect, I tried to work backwards from your screenshot

[danielfdsilva]

This is very strange. Looking at the tile requests there doesn't seem to be anything wrong - the tiles that come in look clean. However I don't see this happening with any other dataset. Could there be something weird going on with the tiler response? Tiles too small, misconfigured values? (just throwing things out here, not sure if they make sense) cc @smohiudd @vincentsarago

[danielfdsilva]

I don't know if this is useful at all, but in the example given the shadow is caused by the rendering of this tile:

https://2qncpyg24c.execute-api.us-west-2.amazonaws.com/api/raster/mosaic/tiles/0717b7cadd852b7131c309717ea06910/WebMercatorQuad/8/157/101@1x?assets=ch4-plume-emissions&resampling=bilinear&bidx=1&colormap_name=plasma&rescale=1%2C1500&nodata=-9999

If I block this specific url, then the shadow does not appear in this case. Not sure why this happens, given that the tile looks clean. 🤔

Edit: updating mapbox to the latest version (2.15.0) did not solve the problem.

[smohiudd]

This one is very strange. @danielfdsilva, can confirm that I can reproduce the issue with the same tile you have linked above. I checked the underlying COG and confirmed it's valid. I even recreated it just in case and still getting the same issue. I also rendered this with rio-viz (which doesn't use mapbox) and I'm not getting the shadow.

I'm only seeing this issue on 2022-08-14 and 2023-05-04 so far.

[smohiudd]

This may be an issue with the tile resampling. I tried nearest and one of the problem area is fixed (2022-08-14):

However still have an issue with 2023-05-04:

[danielfdsilva]

this is so strange...

[smohiudd]

Maybe it was a caching issue but looks like 05-04 is fixed now:

Can someone else do a search and see if they can find any more problem tiles? I can't seem to find anymore:

https://github.com/NASA-IMPACT/veda-config-ghg/pull/54

https://deploy-preview-54--ghg-demo.netlify.app/

[j08lue]

Hmm, the issue persists, at least for me: https://deploy-preview-54--ghg-demo.netlify.app/data-catalog/jpl-ch4-plume-emissions/explore?datetime=2023-05-04T00%3A00%3A00.000Z&position=-46.4651%7C-23.5963%7C10.95

I also see the UI pulling this weird tile:

https://2qncpyg24c.execute-api.us-west-2.amazonaws.com/api/raster/mosaic/tiles/82e82d2c69aa16fdaa88d5cf0f157c74/WebMercatorQuad/7/47/72@1x?assets=ch4-plume-emissions&resampling=nearest&bidx=1&colormap_name=plasma&rescale=1%2C1500&nodata=-9999

which is an overview of the whole raster. Why does this even get pulled when we are all the way zoomed in on the raster?

Btw, the issue is absent when loading the WMTS into QGIS:

@vincentsarago may find this interesting. 😄

[j08lue]

I don't know if this is useful at all, but in the example given the shadow is caused by the rendering of this tile:

@danielfdsilva, I think your comment above https://github.com/NASA-IMPACT/veda-data/issues/33 is the best lead we have.

I wonder when this tile gets pulled and why: it gets loaded even when the map is at a much higher zoom level: try to zoom far in (13 or so) and then reload - the app will request the tiles at that zoom level plus this one at level 7 or 8. Y tho?

[smohiudd]

I recreated the affected COG yesterday and I think that may have fixed it - I'm not seeing the issue anymore for 05-04 (can someone else confirm?).

Here is the COG info for the old and new files.

Old COG:

COG: True
Compression: ZSTD
ColorSpace: None

Profile
    Width:            71
    Height:           129
    Bands:            1
    Tiled:            True
    Dtype:            float32
    NoData:           -9999.0
    Alpha Band:       False
    Internal Mask:    False
    Interleave:       BAND
    ColorMap:         False
    ColorInterp:      ('gray',)
    Scales:           (1.0,)
    Offsets:          (0.0,)

Geo
    Crs:              EPSG:4326
    Origin:           (-46.450625050282454, -23.630816986890196)
    Resolution:       (0.000542232520256367, -0.000542232520256367)
    BoundingBox:      (-46.450625050282454, -23.700764982003268, -46.41212654134425, -23.630816986890196)
    MinZoom:          11
    MaxZoom:          11

Image Metadata
    AREA_OR_POINT: Area

Image Structure
    COMPRESSION: ZSTD
    INTERLEAVE: BAND
    LAYOUT: COG

Band 1
    ColorInterp: gray

IFD
    Id      Size           BlockSize     Decimation           
    0       71x129         256x256       0
    1       36x65          256x256       2
    2       18x33          256x256       4
    3       9x17           256x256       8
    4       5x9            256x256       14
    5       3x5            256x256       24
    6       2x3            256x256       36
    7       1x2            256x256       71
    8       1x1            256x256       71

New COG

COG: True
Compression: DEFLATE
ColorSpace: None

Profile
    Width:            71
    Height:           129
    Bands:            1
    Tiled:            True
    Dtype:            float32
    NoData:           -9999.0
    Alpha Band:       False
    Internal Mask:    False
    Interleave:       BAND
    ColorMap:         False
    ColorInterp:      ('gray',)
    Scales:           (1.0,)
    Offsets:          (0.0,)

Geo
    Crs:              EPSG:4326
    Origin:           (-46.450625050282454, -23.630816986890196)
    Resolution:       (0.000542232520256367, -0.000542232520256367)
    BoundingBox:      (-46.450625050282454, -23.700764982003268, -46.41212654134425, -23.630816986890196)
    MinZoom:          11
    MaxZoom:          11

Image Metadata
    AREA_OR_POINT: Area
    OVR_RESAMPLING_ALG: MODE

Image Structure
    COMPRESSION: DEFLATE
    INTERLEAVE: BAND
    LAYOUT: COG

Band 1
    ColorInterp: gray

IFD
    Id      Size           BlockSize     Decimation           
    0       71x129         512x512       0

[j08lue]

Nice, I cannot reproduce it anymore either. https://deploy-preview-54--ghg-demo.netlify.app/data-catalog/jpl-ch4-plume-emissions/explore?datetime=2023-05-04T00%3A00%3A00.000Z&position=-46.4651%7C-23.5963%7C10.95

[vincentsarago]

Oh sorry I was on holidays so didn't see this issue until now. is there anything I can help with @j08lue ?

[smohiudd]

Hey @vincentsarago, I think we have it figured out. It was an issue with the COG creation and titiler parameter setting.

[j08lue]

is there anything I can help with @j08lue?

Hope you had a nice break! Thanks for asking - I think we now have a theory explaining these rendering artifacts we were getting with Mapbox: the COGs had too many overviews considering their tiny size and/or some of the overviews contained stray values, which Mapbox for some reason loads, even when zoomed all the way in. 🤷

Cloud-optimizing the files again seems to resolve the issue - maybe because we removed those overviews.

This confirms my sense that COG production is challenging for our users (data providers) and we need to put more guardrails on that process or handle it on our side...

[smohiudd]

I recreated all the COGs for this dataset. Strangely, still having issues with a few areas (see spreadsheet @siddharth0248 created)

You can check the preview if you want to see the problem areas:

• Aug 26 2022 | USA (New Mexico)
• Sep 1 2022 | Iraq
• Apr 25 2023 | China (North)

@vincentsarago any thoughts on what could be causing this?

cc: @moradology

[vincentsarago]

🙏 @smohiudd

I've looked at the preview and while looking at the tile requests the application does, I can't see anything wrong, meaning that the app requests 2 tiles (bunch of other fails)

Tile 12/863/1657

Tile 12/863/1658

at this point I don't see any issue with the tiler itself because the two tile images it returns are fine. My guess is that there is something at mapbox/react level

[anayeaye]

That Carlsbad, NM plume in the last example was featured/re-posted in Source New Mexico source last year.

Not helpful to this issue, but I thought it looked familiar...

[j08lue]

My guess is that there is something at mapbox/react level

@vincentsarago, we noticed that the issue seems to be related to an extra tile that gets requested: https://github.com/NASA-IMPACT/veda-data/issues/33#issuecomment-1695270293

In the New Mexico case, it is this request:

https://2qncpyg24c.execute-api.us-west-2.amazonaws.com/api/raster/mosaic/tiles/837b2dc1de8199f4f41bff9fc1c6c9ff/WebMercatorQuad/6/13/25@1x?assets=ch4-plume-emissions&resampling=nearest&bidx=1&colormap_name=plasma&rescale=1%2C1500&nodata=-9999

When I block that request, the artifact does not show up.

The tile returned to me looks like it should - pretty much empty with just a few pixels of data. So that would also hint at a problem with the client (Mapbox), rather than TiTiler.

[moradology]

I don't think this is an extra tile. It looks like a very zoomed out view (Z=6 here) of the data in total. Very strange that blocking it would avoid the issue.

Developing an idea (thank you to @jsignell for bouncing ideas back and forth) that perhaps this has to do with quantization and our handling of missing data: 1. lower zoom levels being gone seems to clear things up, 2. there appear to be some resampling/overview artifacts at some zoom levels (perhaps including 6), 3. tile rendering libraries often overzoom/underzoom a given layer while loading the tiles under/above them and we know 404s come back for the strange gradient tile's location so perhaps it just uses that really nasty overzoom for all time as it waits for 200?

I propose that if 404s were returned as 200s (just an alpha channel to 0 png), this artifact would go away

[j08lue]

I don't think this is an extra tile. It looks like a very zoomed out view (Z=6 here) of the data in total

That is what it is - but it even gets requested when you load the map all zoomed in:

1. Open https://deploy-preview-54--ghg-demo.netlify.app/data-catalog/jpl-ch4-plume-emissions/explore?datetime=2022-08-26T00%3A00%3A00.000Z&position=-104.0642%7C32.393%7C14.9
2. Zoom all the way in on the plume and the artifact (like z=15)
3. Reload and watch the requests - one request for z=6 should show up
4. Block that and reaload, the artifact does not appear

So I meant "extra" in terms of not needed to render the current zoom level.

[moradology]

Apologies for misunderstanding you (and with such confidence). This is certainly odd behavior

[j08lue]

No worries!

Interestingly, btw, the artifacts do not show up in the Jupyter Lite app that @jsignell made (which is using Leaflet) https://nasa-impact.github.io/veda-interactive-emission-plumes/voici/render/plumes.html - look for 26 August 2022 and check the plume in New Mexico.

[smohiudd]

@moradology I tried changing up the zoom extent values in the mdx file but still getting the same artifact.

[moradology]

@smohiudd Out of curiosity, is the strange phenomenon of the Z=6 tile still there after zoom extent rules it out?

[jsignell]

Just wanted to swing by and mention that the dashboard is using the mosaicked version of the rasters (/api/raster/mosaic/tiles) and in the interactive viz I am using closer to the rasters themselves: /api/raster/stac/tiles