Closed justb4 closed 4 years ago
I don't think GDAL is needed when a more recent Arcgis Server Mapservice or Featureservice is called, because 10.5 and higher support GeoJSON as output format. The only thing you need to do is to map the incoming request parameters to a Esri mapservice request and request geojson, the result can be passed to the client without any modification.
True, but there is more than just GeoJSON from Features. We get lots of functionality for free with GDAL, I see the following advantages:
But the proof is in the pudding: if via GDAL does not work out we can develop and ArcGIS FeatureServer Provider.
I don't get from the documentation (https://gdal.org/drivers/vector/esrijson.html#) how GDAL handles authentication, as far as I can see it's just a simple url GET request which it can do. I know Esri and you need to make a oauth token call first and add this token to the request. Paging, getfeature by id and bbox queries can be added to the GET request for instance, get feature by id with the sample service from gdal doc is as simple as: http://sampleserver3.arcgisonline.com/ArcGIS/rest/services/Hydrography/Watershed173811/FeatureServer/0/1?f=json bbox: &geometry=-180%2C-90%2C180%2C90%0D%0A&geometryType=esriGeometryEnvelope&inSR=4326
I'm not sure which functions in class ESRIJSONHelper(SourceHelper): i should modify to achieve a token request and to modify the configured url with required bbox or token
@PeaceNlove personally i'm not in favour to use a product like pygeoapi as a proxy to bypass authentication. It could be ok if pygeoapi would transparantly forward the user tokens to the authenticated services, but i think this type of man-in-the-middle is not possible with this type of technology. we should prevent anonymous users access the authenticated services by impersonating a pygeoapi user.
Ok, for Auth I was referring to generic GDAL/OGR functionality like documented here. If this is what one would prefer or not is another issue. Most ESRI services run on intranets anyway IMHO. Also HTTP proxy stuff (to reach the ESRI server or any other URL), often a headache in corporate/governmental intranets is handled by GDAL.
But the pygeoapi
project is open for contributions, so we would be very happy if someone offers to contribute an ESRI FeatureServer Provider. Like said: I am no ESRI expert, never even used ESRI software. My line of thinking is more: if we can get ESRI FeatureServer support with minimal effort via GDAL that would make a lot of people happy (and have some great demos!).
@pvgenuchten If you do Oauth properly, you don't impersonate a user, but you generate appID and appSecret and let that app generate tokens with these,. This way, it's very easy to revoke permissions for a specific app. @justb4 those are very old methods for authentication and modern Esri setups don't use them anymore. It's all tokens nowadays. I also disagree on the 'Most ESRI services run on intranets anyway' part, because ArcGIS Online is huge and attracting a whole new type of GIS users. Therefore a lot of services are internetservices and could be used with pygeoapi
Anyway, if you can give me some directions on what methods I should modify in class ESRIJSONHelper to add some url parameters to the url before the gdal functions are called I could give this a try.
I tweaked a bit with the current OGRProvider
ESRIJSON
Driver, but it quickly got messy. All in all, I think, following @PeaceNlove suggestion, that it would be better to start a separate ESRIFeatureServer
Provider. Disadvantages OGR Method:
So I see advantages in a specific ESRIFeatureServer
driver:
requests
needed for URL fetching@PeaceNlove is this something you could work on? Then we'll close this issue and open a new issue for ESRIFeatureServer
Provider, phasing out support via OGRProvider
.
This looks like a handy util to use for ESRIFeatureServer
:
https://github.com/Schwanksta/python-arcgis-rest-query/blob/master/arcgis/arcgis.py
It is on Pypi as well: https://pypi.org/project/arcgis-rest-query/ best to use that and contribute.
I wasn't aware of this while fixing #390. I've added tests and everything seems to works. Let's see what happens with the covid-19 endpoint. The only limitation is with features that don't have a geometry member, the OGR connection skips them as of now
As I look to this now an ESRI FS Provider would be a better way to go (than via OGR) . I spent quite some hours, but still Paging, Query by id, bbox and attribute filtering does not work. The ESRI FS API is very similar to OAPIF so mapping is straightforward. I think we should also map the endpoint to Collections (iso specifying each Layer in the pygeoapi
config). And further, ESRI FS Authentication very involved with Tokens and special HTTP headers.
I've implemented ESRI FS interworking in GeohealthCheck: esrifs.py
Plugin. With some template strings and the requests
package, interworking is quite straightforward and we have better control over e.g. bugfixing and when ESRI Auth is required ("hell"). But will take some time, unless @francbartoli you think that GDAL v3 (not yet tried) has improvements for ESRI FS and it could be done?
The OGRProvider currently works with standard OGR Source Types (Drivers) like GeoPackage (GPKG), WFS v2, GeoJSON, SQL/SpatiaLite and even ESRI Shapefiles (.zipped and expanded) and probably Simple Feature GML or even complex GML with mappings in .gfs files (not yet tried) .
OGR (GDAL >= 2.3) should also be able to fetch from
ESRI FeatureServer (ESRIJSON)
. This sort-of works but not for all API calls, in particular further work is needed to add support for:Using the OGR-Driver-specific SourceHelper class any particular handling can be delegated there.
Hopefully an ESRI expert, I am not ;-) , can pick this up.