Scaling, Subsetting "beyond" the coverage (BBOX + resolution interoperability), and slicing irregular coverages

[jerstlouis]

We would like to support a common simple mechanism to sub-set geospatial data across the OGC API standards family (e.g. the semantics of which, but not the result, would be defined in OGC API - Common - Part 2: Geospatial data).

This allows for example to easily cut coverage tiles with BBOX (potentially CRS84h including vertical elevation), width, height and dateTime parameters.

Recent attempts to do this with WCS and WCPS are showing that this is surprisingly difficult to achieve:

• It is not clear whether WCPS is required or this can be done with simply scaleFactor and subset parameters
• Using WCPS:
  • scale() is not necessarily implemented with n-D pyramids and thus may process too much data unnecessarily
  • extend() must be used to try to subset beyond the coverage's domainset before scaling, which may result in generating too much data unnecessarily
  • irregular coverages have limitations regarding slicing

A valid concern would be is that for the same reason this is difficult to achieve with WCS, it may be difficult to implement this functionality as a facade on top of WCS, so perhaps this could live in conformance classes that not all implementations would implement initially.

[jratike80]

Please define irregular coverages.

[jerstlouis]

@jratike80 the definition is at https://docs.opengeospatial.org/is/09-146r6/09-146r6.html#32

I am myself a bit confused by the concept, but my understanding is that you cannot expect something at regular interval along an axis, so that you can end up trying to slice at a value where there is nothing, even though there is someothing before and after on that axis (e.g. a coverage made up of many satellite imagery scenes taken on different dates at different locations).

[joanma747]

For a common way of subsetting my vote goes to the WCS one that, at least, does not have axis order issues. It supports n-dimensions and the subsetting of 0D or 1D data. I always get confused with the order of the BBOX param. Is it x1,y1,x2,y2 or x1,x2,y1,y2 or y1,x1,y2,y2... And even for a non-spatial expert, it is clearer because it is explicit.

&subset=Lat(30,42)&subset=Long(-125,-115)

Superclear!. I know what is lat and what is long!!. Wonderful!.

I personally will be happy to adopt it in WMS too.

See https://docs.geoserver.geo-solutions.it/edu/en/wcs/get.html for examples of scale and rangesubset.

[jerstlouis]

@joanma747 I also get confused with axis order, but mainly because it is CRS84 and that is backwards ;)

Would we drop BBOX completely in everyhing but Features? (where it has an intersecting meaning rather than clipping, another issue with it). A new conformance class could specify subset for clipping features.

@Schpidi Would it be thinkable to refine the syntax of subset? It is already so verbose, why does subset need to be repeated for every single axis?

Couldn't we specify &subset=Lat(30,42),Long(-125,-115) instead for OGC API - Coverages? Or even without the comma that's easy to parse: &subset=Lat(30,42)Long(-125,-115)

(Is that specifyable with OpenAPI?)

[Schpidi]

Good point, I guess it would be valid to adjust the syntax as you suggest. At least I can't think of a case that wouldn't be supported. Maybe we should decide on a separator that isn't used already in order to ease parsing, e.g. ;.

[pomakis]

I like the idea of collapsing it into a single parameter value. However, one problem with changing the syntax from WCS 2.0 is that it requires a code rewrite to evolve from WCS 2.0. Currently, our support for this parameter in the OGC API was accomplished with one call to an existing function.

[jerstlouis]

@Schpidi also is there anything preventing : in the URL? We use that now for hierarchical {collectionId} and it seems to work just fine without URL encoding... We could make this more similar to WCPS syntax e.g.:

&subset=Lat(30:42),Long(-125:-115)

and avoid using the comma twice.

[pvretano]

@jerstlouis we have already implemented the subset=Lat(30,42)&subset=Long(-125,-115) encoding but before deciding on an ultimate encoding we need to take several things into consideration (1) The current encoding is taken from WCS and so existing WCS client know how to handle it. If we change it we might be putting a barrier in front of people who simply want to facade an OGC API on top of existing implementations. (2) We need to take into consideration how the parameters would/could be encoding in OpenAPI. Right now, both approaches (multiple parameters or single parameter) are encoded as strings. It might be better, and this is just an example to accommodate OpenAPI, to encode the dimension names as the query parameters (e.g. ...lat=30,42&long=-125,-115...). Not saying that we should do it that way just saying we need to consider OpenAPI requirements.

[pomakis]

In WCS 2.0, the individual values can be strings in some sitiuations. E.g., Time("2020-08-11T12:32:00“,"2020-08-11T14:47:00"). Note that that these strings may contain colons and maybe even commas, etc, so a non-trivial parser will be needed either way if we merge this into a comma-separated list, even if we change one of the separators into commas.

[jerstlouis]

Interestingly there are already colons inside the time strings. I propose:

&subset=Lat(30:42),Long(-125:-115),Time("2020-08-11T12:32:00":"2020-08-11T14:47:00")

The parser just needs to ignore commas / colons within the quotes.

[jratike80]

Is it really so hard to write separate subsets? Giving all subsets as one string does not add clarity. Users do not useally write subsets by hand but they use applications which are written for them. Or they use the copy-paste-edit method by using some working query as a template. Helping developers to write more code and less text that does not add value is still a valid point.

For my mind the main question is "Do we get less errors with single subset string?"

BTW there is an error in &subset=Lat(30:42),Long(-125:-115),Time("2020-08-11T12:32:00“:"2020-08-11T14:47:00")

The error message shoud be "Invalid subset, dimension: Time". Works for me. (the error is wrong character used as the closing double quote).

[pvretano]

Looking at this from the OGC API - Common perspective ... so we have bbox, time and subset all of which more-or-less do similar things? So bbox=10,10,20,20&bbox-crs=http://www.opengis.net/def/crs/OGC/0/CRS84&time=2020-08-11/2020-08-12 is equivalent to subset=Lat(20:20),Lon(10:10),Time("2020-08-11":"2020-08-12")? Isn't that more confusing?

[jerstlouis]

@pvretano I have been pushing all this time for BBOX across the OGC APIs. But having implemented Maps recently and testing our server, I realize that every single time when testing this in the browser bar I mix up the min/max lat/lon. @joanma747 opened my eyes ;)

But getting it down to having to type subset only once makes it much friendlier for that user testing in browser bar, and therefore a better candidate for the Common way to do clipping aka sub-setting.

Why would I want to type subset 3 times?!?

[pvretano]

@jerstlouis presumably there is a nice client in front of the API. I don't think "it is cumbersome for a human to type in a browser" is a good argument for supporting something since most of this stuff is machine generated anyway. In code, which is easier to handle? From what I gather in this discussion, it does not matter one way or the other. Is that correct?

[jerstlouis]

@pvretano With these more browser-friendly API, I am sure there will be even more users tweaking addresses in the browser bar than with WxS. So why not make things friendlier for the user?

Shorter/friendlier URL also mean friendlier to debug for the developer, both in the code debugger and when testing things in the browser.

[ghobona]

Looking at the comment from @pvretano, I am wondering whether the Sprint should recommend to the OGC-NA that URNs be reintroduced for use as an alternative to HTTP URIs?

[jratike80]

Do not forget to test then also if &subset=Time("2020-08-11T12:32:00":"2020-08-11T14:47:00"),Long(-125:-115),Lat(30:42) gives the same result than &subset=Lat(30:42),Long(-125:-115),Time("2020-08-11T12:32:00":"2020-08-11T14:47:00")

But really, the difference in browser testing is minimal.

Trying to think about some other use cases. How about if developer would like to enhance the bbox based download application by adding a time control as a slider. Would it be more convenient to make the slider to adjust only the Time subset, or to slip the Time dimension into the universal subset string?

[jerstlouis]

@jratike80 as a client developer I would probably re-generate that whole string, the slider just slides that one property so it doesn't make any differences.

Good point on the equivalence of subset order. Do you know if WCS 2.0 ensures this is equivalent? (I would assume it is).

I am wondering how this relates to the irregular coverages issues... e.g. not being able to slice by time, can you only slice after you've subsetted a particular area? I would really want those to be equivalent (including if time was a slice rather than a trim).

[pvretano]

@ghobona I am not a fan of URNs because they are not resolvable in a browser. The reasons I like HTTP URIs such as http://www.opengis.net/def/crs/OGC/0/CRS84 (as opposed to urn:ogc:def:crs:OGC::CRS84) is that if plop that in a browser I get the definition.

[pvretano]

@jerstlouis I don't think there is an appreciable complexity or length difference between subset and bbox/bbox-crs/time. The main argument I see for subset is that it resolves the coord order confusion but even that is not as big a problem as it used to be; especially since OGC mandated that the CRS definition is the source of truth about coordinate order. One thing ... if we want to push subset to all OGC API's then we would need to also push some neutral and resource agnostic way to advertise which dimensions are available for subsetting. No? Otherwise, in features for example, getting the dimensions would involve getting the CRS definition, parsing the document, etc. Too much trouble if you ask me.

[jerstlouis]

the CRS definition is the source of truth about coordinate order

In my opinion any attempt to fix the axis ordering issue makes things worst.

resource agnostic way to advertise which dimensions are available for subsetting

Right. I would argue for Lat, Long to always be supported if "spatial" is specified in the extent, and "dateTime" (or just "time") if the extent reports "temporal".

OGC API - Coverages could say something about those being fixed to those names? I think that further helps interoperability/usability as well rather than those names being arbitrary and having to first check the domainset for what those are called. If the coverage axis are not in geographic coordinates, then it presumably have other names (e.g. E / N) which don't conflict with Lat / Long...

And we had discussed "extra" for the other dimensions, which the extent could also conveniently expose there.

Now I am not sure what the CRS implications are... but I am quite confident that it is greatly worth to try to have a consistent subsetting mechanism across the OGC API.

In WCS 2, is subset always in the output CRS? Is it possible to e.g. subset in the native CRS then have results returned in another CRS?

[pvretano]

@jerstlouis How does that make it worse? The CRS definition says use A,B order, you use A,B order! Anyway, nuff about that. No, I don't think that will work since extent is optional not mention that for some CRS's I don't think the dimension names are LAT or LONG. I think it would be better for the server to be explicit about it.

[jratike80]

In WCS 2, is subset always in the output CRS? Is it possible to e.g. subset in the native CRS then have results returned in another CRS?

If server implements the crs extension https://portal.opengeospatial.org/files/54209 then user can use subsettingCrs and outputCrs.

The biggest inconvenience in subsetting in WCS 2.0 is that user must know the official axis labels which are used in subsetting. They must be resolved from DescribeCoverage. Example: https://beta-karttakuva.maanmittauslaitos.fi/wcs/service/ows?service=WCS&version=2.0.1&request=DescribeCoverage&coverageID=korkeusmalli__korkeusmalli_2m

<gml:Envelope srsName="http://www.opengis.net/def/crs/EPSG/0/3067" axisLabels="E N" uomLabels="m m" srsDimension="2">

For EPSG:2393 axis labels are X and Y, where X means northing and Y easting.

[jerstlouis]

@jratike80 thanks, so that is quite similar to bbox-crs?

I wish we could extent the {collectionId}'s extent to describe valid subset parameters for CRS84, and that this could always be Lat, Long.

[jratike80]

Do you know if WCS 2.0 ensures this is equivalent? (I would assume it is).

Subsets in WCS 2.0 are independent and they can appear in any order.

[Schpidi]

20200902: Coverages SWG call: It was decided to not use / as delimiter for various reasons like not being usable in paths although ISO8601 and WMS would use / for intervals. The proposal for the subset parameter on the table is: &subset=Lat(30:42),Long(-125:-115),Time("2020-08-11T12:32:00Z":"2020-08-11T14:47:00Z"). i.e. to have a single subset parameter and use : as delimiter per axis and , as delimiter between axes. This is also discussed in https://github.com/opengeospatial/oapi_common/issues/167 as proposal to make this part of common.

We will vote on this proposal in our next call.

[Schpidi]

20200909: Coverages SWG call: NOTUC to adopt the proposal and merge #88 once extended to fully reflect the proposal.

Please open a separate issue if an individual aspect of this one isn't covered yet.