Pattern/docs for passing geometry filters on to remote APIs (pass-through provider)

[rgwozdz]

From @AvidDabbler:

Its been a while since I looked at the docs, but I think my big issue with it was figuring out how to run server side envelope queries. Would be nice to have some best practices or a method to be able to do the filtering that way. For reference I was using AWS Athena, I would design the query, pulling in all of the data and then filtering, being able to just have the data with the envelope would have been nice and having something like a query builder would have been beneficial. This could all be user error or me not reading the docs well enough.

Originally posted by @AvidDabbler in https://github.com/koopjs/koop/issues/529#issuecomment-1482897456

[rgwozdz]

Thanks @AvidDabbler; this is a very good call out and points to something we need to work through. We want to support lots of different output plugins and these plugins have different specs for things like geometry filter/envelope. For example, Geoservices uses a geometry parameter with a particular shape, while vector-tiles uses an implicit envelope based on x/y/z route parameters. We need to standardize on something so that providers can look for expected parameters is one place.

It might make sense to standardize on the Geoservices spec, since most applications for Koop are using the geoservices output-plugin. I think one of the downsides to that is have to use a where parameter for key/value filters.

[AvidDabbler]

Yeah. IDK that there is a great solution for all data sources. Using where parameters for data types would be great, but if you are dealing with databases and are not concerned with geometry queries you could use prisma. I believe geo functions are on the roadmap. But for geo queries at the moment you might need to just have some built in methods just for envelope queries and allow a method with the passed params to allow the user quick access to write the customized where clause.

On Fri, Mar 24, 2023 at 9:53 AM Rich Gwozdz @.***> wrote:

Thanks @AvidDabbler https://github.com/AvidDabbler; this is a very good call out and points to something we need to work through. We want to support lots of different output plugins and these plugins have different specs for things like geometry filter/envelope. For example, Geoservices uses a geometry parameter with a particular shape, while vector-tiles uses an implicit envelope based on x/y/z route parameters. We need to standardize on something so that providers can look for expected parameters is one place.

It might make sense to standardize on the Geoservices spec, since most applications for Koop are using the geoservices output-plugin. I think one of the downsides to that is have to use a where parameter for key/value filters.

— Reply to this email directly, view it on GitHub https://github.com/koopjs/koop/issues/530#issuecomment-1482934503, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACAUJTCL65HSE5JCR2S5PD3W5WYORANCNFSM6AAAAAAWGU3AQU . You are receiving this because you were mentioned.Message ID: @.***>

[rgwozdz]

Yeah, I'm talking about something slightly different. Different data sources are going to have their own api/syntax for doing any kind of filtering (e.g., elastic vs mongo), and it's the job of the provider to turn the incoming request params into a query it can pass on to the remote data source. But the trouble is, the request param keys for defining filters are going to vary based on which output-plugin route is handling the request. So a typical GeoServices request might look like:

elastic/rest/services/:id/FeatureServer/0/query?geometryType=esriGeometryEnvelope&geometry={xmin: -104, ymin: 35.6, xmax: -94.32, ymax: 41}

In this case, the elastic provider needs to parse geometry and geometryType and convert it to its equivalent elasticsearch query syntax, then send it on to elastic.

But if a request arrives on the vector-tile output plugin route, it looks like this:

elastic/:id/VectorTileServer/:z/:x/:y.pbf

In this case, the elastic provider needs to parse and covert the z, x, and y params to the elastic equivalent of a geometry envelope. So even with just these two output-plugins, provider plugins have to handle two separate possible conversions to pass on geometry filters to the remote data source. It would ease provider-plugin development if they could depend on a standardized set of filter parameters, regardless of which output-plugin the request flows through.

[rgwozdz]

Closing due to inactivity.