search

geopython / pygeoapi

pygeoapi is a Python server implementation of the OGC API suite of standards. The project emerged as part of the next generation OGC API efforts in 2018 and provides the capability for organizations to deploy a RESTful OGC API endpoint using OpenAPI, GeoJSON, and HTML. pygeoapi is open source and released under an MIT license.
https://pygeoapi.io
MIT License
483 stars 258 forks source link

OGRProvider: Get ESRI FeatureServer fully working #137

Closed justb4 closed 4 years ago

justb4 commented 5 years ago

The OGRProvider currently works with standard OGR Source Types (Drivers) like GeoPackage (GPKG), WFS v2, GeoJSON, SQL/SpatiaLite and even ESRI Shapefiles (.zipped and expanded) and probably Simple Feature GML or even complex GML with mappings in .gfs files (not yet tried) .

OGR (GDAL >= 2.3) should also be able to fetch from ESRI FeatureServer (ESRIJSON). This sort-of works but not for all API calls, in particular further work is needed to add support for:

Using the OGR-Driver-specific SourceHelper class any particular handling can be delegated there.

Hopefully an ESRI expert, I am not ;-) , can pick this up.

PeaceNlove commented 5 years ago

I don't think GDAL is needed when a more recent Arcgis Server Mapservice or Featureservice is called, because 10.5 and higher support GeoJSON as output format. The only thing you need to do is to map the incoming request parameters to a Esri mapservice request and request geojson, the result can be passed to the client without any modification.

justb4 commented 5 years ago

True, but there is more than just GeoJSON from Features. We get lots of functionality for free with GDAL, I see the following advantages:

But the proof is in the pudding: if via GDAL does not work out we can develop and ArcGIS FeatureServer Provider.

PeaceNlove commented 5 years ago

I don't get from the documentation (https://gdal.org/drivers/vector/esrijson.html#) how GDAL handles authentication, as far as I can see it's just a simple url GET request which it can do. I know Esri and you need to make a oauth token call first and add this token to the request. Paging, getfeature by id and bbox queries can be added to the GET request for instance, get feature by id with the sample service from gdal doc is as simple as: http://sampleserver3.arcgisonline.com/ArcGIS/rest/services/Hydrography/Watershed173811/FeatureServer/0/1?f=json bbox: &geometry=-180%2C-90%2C180%2C90%0D%0A&geometryType=esriGeometryEnvelope&inSR=4326

I'm not sure which functions in class ESRIJSONHelper(SourceHelper): i should modify to achieve a token request and to modify the configured url with required bbox or token

pvgenuchten commented 5 years ago

@PeaceNlove personally i'm not in favour to use a product like pygeoapi as a proxy to bypass authentication. It could be ok if pygeoapi would transparantly forward the user tokens to the authenticated services, but i think this type of man-in-the-middle is not possible with this type of technology. we should prevent anonymous users access the authenticated services by impersonating a pygeoapi user.

justb4 commented 5 years ago

Ok, for Auth I was referring to generic GDAL/OGR functionality like documented here. If this is what one would prefer or not is another issue. Most ESRI services run on intranets anyway IMHO. Also HTTP proxy stuff (to reach the ESRI server or any other URL), often a headache in corporate/governmental intranets is handled by GDAL.

But the pygeoapi project is open for contributions, so we would be very happy if someone offers to contribute an ESRI FeatureServer Provider. Like said: I am no ESRI expert, never even used ESRI software. My line of thinking is more: if we can get ESRI FeatureServer support with minimal effort via GDAL that would make a lot of people happy (and have some great demos!).

PeaceNlove commented 5 years ago

@pvgenuchten If you do Oauth properly, you don't impersonate a user, but you generate appID and appSecret and let that app generate tokens with these,. This way, it's very easy to revoke permissions for a specific app. @justb4 those are very old methods for authentication and modern Esri setups don't use them anymore. It's all tokens nowadays. I also disagree on the 'Most ESRI services run on intranets anyway' part, because ArcGIS Online is huge and attracting a whole new type of GIS users. Therefore a lot of services are internetservices and could be used with pygeoapi

Anyway, if you can give me some directions on what methods I should modify in class ESRIJSONHelper to add some url parameters to the url before the gdal functions are called I could give this a try.

justb4 commented 5 years ago

I tweaked a bit with the current OGRProvider ESRIJSON Driver, but it quickly got messy. All in all, I think, following @PeaceNlove suggestion, that it would be better to start a separate ESRIFeatureServer Provider. Disadvantages OGR Method:

So I see advantages in a specific ESRIFeatureServer driver:

@PeaceNlove is this something you could work on? Then we'll close this issue and open a new issue for ESRIFeatureServer Provider, phasing out support via OGRProvider.

justb4 commented 5 years ago

This looks like a handy util to use for ESRIFeatureServer: https://github.com/Schwanksta/python-arcgis-rest-query/blob/master/arcgis/arcgis.py It is on Pypi as well: https://pypi.org/project/arcgis-rest-query/ best to use that and contribute.

francbartoli commented 4 years ago

I wasn't aware of this while fixing #390. I've added tests and everything seems to works. Let's see what happens with the covid-19 endpoint. The only limitation is with features that don't have a geometry member, the OGR connection skips them as of now

justb4 commented 4 years ago

As I look to this now an ESRI FS Provider would be a better way to go (than via OGR) . I spent quite some hours, but still Paging, Query by id, bbox and attribute filtering does not work. The ESRI FS API is very similar to OAPIF so mapping is straightforward. I think we should also map the endpoint to Collections (iso specifying each Layer in the pygeoapi config). And further, ESRI FS Authentication very involved with Tokens and special HTTP headers.

I've implemented ESRI FS interworking in GeohealthCheck: esrifs.py Plugin. With some template strings and the requests package, interworking is quite straightforward and we have better control over e.g. bugfixing and when ESRI Auth is required ("hell"). But will take some time, unless @francbartoli you think that GDAL v3 (not yet tried) has improvements for ESRI FS and it could be done?