search

US-GHG-Center / veda-config-ghg

Veda config for GHG
https://ghg-demo.netlify.app
Other
3 stars 15 forks source link

[R2] Add new MiCASA and LPJ datasets and update dataset overview pages to new format #333

Closed siddharth0248 closed 2 months ago

siddharth0248 commented 3 months ago

Add new datasets for R2

netlify[bot] commented 3 months ago

Deploy Preview for ghg-demo ready!

Name Link
Latest commit 4e661649dea906ca2a0bc1fd652cbb9693e86dde
Latest deploy log https://app.netlify.com/sites/ghg-demo/deploys/662bd1cf4a93c80008deae01
Deploy Preview https://deploy-preview-333--ghg-demo.netlify.app
Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

siddharth0248 commented 2 months ago

@slesaad @j08lue can one of you point out why titiler increasing the spatial resolution of the LPJ data. I tried to visualize the data in QGIS and in our portal. Any variable in mdx that is doing this

image

image

slesaad commented 2 months ago

@siddharth0248 we had this same issue a while ago, which was fixed by pinning rio_tiler to a post release that @vincentsarago created https://github.com/US-GHG-Center/ghgc-backend/pull/59/files

we recently removed that pin because I understood that the fix was incorporated in the new releases (see: https://github.com/NASA-IMPACT/veda-backend/pull/267/files).

@vincentsarago these are the requirements used for this deployment - https://github.com/US-GHG-Center/ghgc-backend/blob/dev/raster_api/runtime/setup.py. any idea what's up?

j08lue commented 2 months ago

Some resources for debugging this:

UI preview to debug the issue:

Example tile requests:

Can you point out the exact assets (COGs) that are getting rendered, @siddharth0248?

It's not this, I can see, but a preview URL like this would be super helpful: https://dev.ghg.center/ghgcenter/api/raster/cog/preview?url=s3://ghgc-data-store-dev/NASA_GSFC_ch4_wetlands_monthly/NASA_GSFC_ch4_wl_ch4_wetlands_v22_x720_y360_t12_202112.tif&colormap_name=magma&rescale=0%2C0.3&nodata=-9999

vincentsarago commented 2 months ago

@vincentsarago these are the requirements used for this deployment - https://github.com/US-GHG-Center/ghgc-backend/blob/dev/raster_api/runtime/setup.py. any idea what's up?

well, as far I can see, this shows titiler.core requirement "titiler.core>=0.15.5,<0.16" which should depends on rio-tiler >6.2.5 (https://github.com/cogeotiff/rio-tiler/issues/654 was fixed in 6.2.7), I don't see why pip won't install rio-tiler 6.4.5 (latest version)

j08lue commented 2 months ago

when can we get this resolved ? or any pointers for me ?

As far as I understand, the issue with the rendering is not related to this PR or the files, but only the rio-tiler version in the dev stage of our data services, which for some reason is out of date.

Therefore, you should in principle be ok to merge this PR and the data shoud look right in production.

I would like to verify this first, though, but it looks like the production tiler does not have access to the dev bucket? This does not load:

https://ghg.center/api/raster/cog/map?url=s3://ghgc-data-store-dev/lpjwsl-wetlandch4-daygrid-v2/ensemble_mean/LPJ_EOSIM_L2_CH4e_dch4e_ensemble_mean_19900101day.tif&colormap_name=magma&rescale=0%2C2&nodata=0

and this throws access denied:

https://ghg.center/api/raster/cog/info?url=s3://ghgc-data-store-dev/lpjwsl-wetlandch4-daygrid-v2/ensemble_mean/LPJ_EOSIM_L2_CH4e_dch4e_ensemble_mean_19900101day.tif&colormap_name=magma&rescale=0%2C2&nodata=0

So we would need to fix the access (@slesaad or @amarouane-ABDELHAK). Or were you planning on moving the data to a production bucket, @siddharth0248, and could fast-forward that?

siddharth0248 commented 2 months ago

Changed the min max values for each of the layer. This helps in distinguishing high and super high values. eg, USA and China

Original/Proposed values in document image

Changed values

image

@deborahUAH please have a look

vincentsarago commented 2 months ago

After talking with @siddharth0248 and @SwordSaintLancelot we found that the dataset had a weird transform (origin of the file being lower right, instead of top right). For some Reason QGIS is able to work with it, but rio-tiler couldn't.

I've added a fix in rio-tiler https://github.com/cogeotiff/rio-tiler/pull/696 which I hope to get merged/released today

deborahUAH commented 2 months ago

@siddharth0248 regarding images above about changing visualization scale. NBE is a new layer right? Confirm choice of color ramp and min/max for visualization with data provider.

siddharth0248 commented 2 months ago

The content has been checked by the team.

hanbyul-here commented 2 months ago

I have the same comments with @j08lue , If the data is not on prod endpoint, it means that it is not ready to be deployed to production. Please change the endpoint to prod when it is ready.

SwordSaintLancelot commented 2 months ago

We can change the API back to prod, once we create the STAC collection and ingest items.

j08lue commented 2 months ago

We can change the API back to prod, once we create the STAC collection and ingest items.

Can we wait with merging this until then?

j08lue commented 2 months ago

@siddharth0248 please check the ECCO Overview page - there was a merge conflict and I am not sure I kept the right edit. Looks like this now:

image

j08lue commented 2 months ago

MiCasa also still needs the additional layer info for E&A:

image

siddharth0248 commented 2 months ago

@Jonas I haven't added the media section, because how don't know how you grabbed it. But rest of the Info needed for the data card and layers have been updated. I have also added monthly layer for LPJ_EOSIM

j08lue commented 2 months ago

I haven't added the media section, because how don't know how you grabbed it

Yeah, @faustoperez made the thumbnails last time

I think you can just take a screen grab from the rendered map of whichever extent and zoom level best brings out the characteristics of the dataset. You could also use the /raster/cog/preview endpoint of the tiler, if you anyways want a global coverage thumbnail, e.g.: https://dev.ghg.center/ghgcenter/api/raster/cog/preview?url=s3://ghgc-data-store-dev/MiCASA/v1/daily/2001/01/MiCASAv1_flux_NBE_x3600_y1800_daily_20010101.tif&colormap_name=purd&rescale=0%2C1.5