[EPIC] WCS "clip and ship"- v2.0

[tephenavies]

From https://github.com/TerriaJS/terriajs/issues/2937#issuecomment-420085999:

• The parameters width and height determine the output grid of the resulting coverage, this means that the resulting coverage is not native resolution.
• The resulting GeoTIFF only includes the bands associated with the active band (Red, Blue, Green for this example).
• Our belief is that our stakeholders will expect to be able to acquire native resolution and all bands associated with a given product. ACTION: to validate this assumption

[AnaBelgun]

First user feedback since re-testing:
"I have had a play with the functionality. It looks simple, but really useful.

It seems that the resolution of the resultant GeoTiff is determined by the size of the area requested. A small area over Canberra returned a much higher resolution tiff than a request for most of NSW. I think this is a really neat functionality if you want to produce a quick picture, but might be less useful if people are actually wanting to do analysis on the data.

Maybe an option to explicitly set the output resolution? Or at least a pop up message that the output resolution will be more course than the satellite’s native resolution if you choose too large an area.

I could only load in ‘clear observations’ which seemed to only have one band in the output.

In terms of the workflow, it was very straight forward, which I liked. I loaded the resulting GeoTiff into ArcGIS to test it, and it seemed to load correctly there.

One other comment would be to make Australia the default location of the map when you load up Terria. For me, it was somewhere in the Atlantic near Africa, so I had to move the map to find Australia before I could get started. Just a minor suggestion though.

It would be good to have a more complex dataset to play with to really test the functionality. Something with time slices and bands to see how that works with the export functionality?"

[AnaBelgun]

From the feedback above I would create the following issues:

• message to advise user about the (coarse) resolution/different than native (@philipgrimmett)
• implement option to get native resolution if possible/feasible
• implement option to display in more bands if feasible

@steve9164 see if this is reasonable break down

[AnaBelgun]

following Paul's investigation of colour bands, to implement a workflow which allows users to tick/select different bands/styles

[juan-guerschman]

found about this issue today now that our layers got WCS and I was exploring the feature. Same comments from me, as a user myself I'd like to have more control over the specifics of the output(s) returned. In particular having back something with the native resolution of the data. If it could be a guide, the Worldview site has a neat tool which allows the user to decide on these things: https://worldview.earthdata.nasa.gov

[AnaBelgun]

We need a review of current v1.0 and what can be improved for v2.0 from UX/UI perspective; added to planning

[wibily]

Ana: ticket is to make it better not complete (ie v1.1 not v2)

[AnaBelgun]

@philipgrimmett @nf-s Update 1 Nov 2021:

• to test: https://maps.dea.ga.gov.au/#share=s-thUx3Apjwffi84xdqCqb1gPqMJC
• drawing polygon doesn't work very well (it doesn't let you create another shape than rectangular?)
• there is no close function for the pop up of downloading extent: i.e you can't get out of it if changed your mind and want to select another dataset etc
• download native resolution - high priority
• band - medium priority
• @nf-s would you please have a look at the other linked tickets and see if styling can be included in this timeframe

[philipgrimmett]

Figma prototype

https://www.figma.com/proto/jtdgEiY0gxtZlGtxSJTzcZ/Terria---2021?page-id=8310%3A7172&node-id=8312%3A7904&viewport=286%2C48%2C0.48&scaling=contain&starting-point-node-id=8312%3A7904

To discuss

• How do we currently assess whether a selection can be exported RAW (native)?
• If a selection can't be exported RAW, what other options are available?
• Are we able to export all bands? Will this have significant impact on download size/speed?

[nf-s]

Version 1 ready for review

Test URL - http://ci.terria.io/wcs-2/#configUrl=https://raw.githubusercontent.com/TerriaJS/saas-catalogs-public/main/de-australia/map-config/prod.json&share=s-qEsTdaPNVmQt1GEa7HkjRVMoghg

My assumptions

• WCS has Range subsetting extension https://portal.ogc.org/files/12-040
• WCS has CRS extension https://portal.ogc.org/files/54209
• WCS has KVP protocol extension https://portal.ogc.org/files/09-147r3
• Native CRS is determined by looking at domainSet.Grid.srsName and then (Envelope | EnvelopeWithTimePeriod).srsName

• Will use CoverageDescription.ServiceParameters.nativeFormat as default output format
• In GetCoverage requests, I am only specifying subsets for Lat/Long, time and WMS dimensions
  • Lat/Long subsets are always EPSG:4326 (no support for X/Y subsets in different CRS)
  • I am not specifying any scaling...

An example GetCoverage request

https://ows.dea.ga.gov.au/?service=WCS&request=GetCoverage&version=2.0.0&coverageId=ga_ls_ard_3&format=image/geotiff&subset=Long(114.10100132274897,114.25284999493852)&subset=Lat(-26.294307619640055,-26.171064499960185)&subset=time("2021-10-26")&subsettingCrs=EPSG:4326&outputCrs=EPSG:3577

[nf-s]

Problems

ODC OWS doesn't "align" requests to original grid

For example - here you can see two different geotiffs (from two GetCoverage requests) with different grids

Whereas, GeoServer will always return coverages with original grid - regardless of Lat/Long subset

[SpacemanPaul]

@nf-s yeah, I'm not surprised by the alignment issues. You might get better more consistent results by using EPSG:3857 instead of EPSG:4326 as the subsetting CRS.

To properly fix, I would have to add special handling for native-resolution requests use the native grid instead of creating their own. This is probably not going to happen soon, sorry.

[robbibt]

Just copying some Slack discussion here to document it:

@robbibt:

...this functionality is looking really nice - such an easy way to access the underlying high-res data! Any chance I could get some clarification about how this works behind the scenes (some of this may have been already discussed RE the alignment issue)

• Looks like the data is being re-projected to Australian Albers EPSG:3577 rather than being exported in its native CRS - is this true?
• If so, what resampling method is used when re-projecting the data? The exported clip seems subtly different from both the OWS rendering and the raw data itself, so would be good to know how the resampling is being handled here

@nf-s:

Hi robbibt there are some comments about my assumptions here https://github.com/TerriaJS/terriajs/issues/3109#issuecomment-963885147

• native CRS is guessed from WCS DescribeCoverage response, it can be overridden in catalog config - what is the native CRS supposed to be?
• I don't send an interpolation parameter - I believe most services use nearest-neighbor as default. I can add interpolation to catalog config if needed

@SpacemanPaul:

1) WCS "native CRS" is the product-native-CRS where possible. Where native-CRS is set at the dataset level, the OWS system admin sets a nominal coverage-wide native-CRS. This is unavoidable because OWS publishes whole products as layers/coverages, not individual datasets. The whole Terria WMS-WCS interaction would be impossible without it. 2) WCS allows for user-selection of interpolation method but OWS does not support this - yet. There is a github issue, mostly waiting for somebody to say "yes, there is a use case for this". It would be pretty easy actually.

Oh, and more significantly: 3) There is currently no way to get the unprojected "native data" unless you deliberately set your request spatial range in native CRS and using coordinates that are exactly on the native grid, as advertised in DescCoverage. I think I'm going to have to add code to mangle "native resolution" queries to quantise them onto the defined native grid.

[robbibt]

In response to the thread above: as clip and export now returns 30 m multiband data, I can see it being used primarily by users who want to actually quantitatively analyse the data, not just visualise it. Because of this, some of the questions around map projection and resampling become more important as they can affect the outputs of subsequent analyses.

The "ideal" scenario here would be that "clip and ship" returns data for a given extent exactly as it's stored on file (i.e. in its native grid), without any resampling or reprojection applied that could degrade or modify the data. It sounds like this won't be possible due to OWS's use of a nominal coverage-wide CRS (e.g. EPSG:3577) - I think that's probably OK, as at least it will be consistent across all DEA satellite and derived products.

If the data coming from "clip and ship" is going to be resampled to EPSG:3577 though, I'd really like it to be possible to control the resampling that's applied to the various bands - nearest neighbour resampling is probably fine for visualisation but it can really degrade the quality of the satellite data for precise applications (e.g. lots of our coastal work isn't possible on nearest neighbour resampled data). For multispectral data, I think "bilinear" or "average" resampling for all of the spectral bands and "nearest" for the "Fmask"/pixel quality band would be a sensible default that would make sure the data is as useful to users as possible.

Similarly, the lack of a consistent grid is going to be problematic for users who want to analyse this data - if different "clip and ships" return differently gridded data, it's going to be really challenging for users to combine and analyse the outputs of different exports (e.g. either spatially when combing different areas of the same satellite images, or temporally to compare different timesteps) - they'll have to apply a resampling step of their own which will further degrade the data and make it less useful for downstream workflows

Not sure if either of these are a "clip and ship" issue or a OWS issue though, so happy to move this request elsewhere if necessary!

[robbibt]

I'm still getting a bunch of errors that seem related to trying to clip and ship large areas of an image. To reproduce:

1. Visit http://ci.terria.io/wcs-2/#configUrl=https://raw.githubusercontent.com/TerriaJS/saas-catalogs-public/main/terria-cube/map-config/prod.json&share=s-iULPRQuZy9cBc46RNEoncsVMMCV
2. Click "export" and select an area covering the width of the satellite image:

3. After ~ 1 minute, the following error will appear:

I've also tried this on the "DEA Surface Reflectance (Sentinel-2)" layer and the error appears almost instantly - not sure if this is the same issue or whether the current URL is only set up to work on "DEA Surface Reflectance (Landsat)".

Is there an area limit that's causing the clip and ship to crash? If so, it might be good to find a way to communicate this to users so they can limit the area they choose.

[meh9]

@robbibt the OWS service is returning a very unexpected 502 Bad Gateway. Potentially this is because of an error in the service, or a size limit internal to the service that isn't reporting a suitable error message.

[nf-s]

When we sort out subsetting with the "native grid", we can add a resolution (or number of pixels) limit per request - we may also be able to give a rough estimate of file size before we fire off a GetCoverage request.

Also we can add interpolation quite easily on terria side when it has added it to odc-ows

[nf-s]

https://github.com/TerriaJS/terriajs/pull/5967 is merged!

[nf-s]

TODO turn comments here into separate tickets

[AnaBelgun]

Closing. If new issues are raised against the PR, will open a new ticket - v3.