search

Esri / arcgis-osm-editor

ArcGIS Editor for OpenStreetMap is a toolset for GIS users to access and contribute to OpenStreetMap through their Desktop or Server environment.
Apache License 2.0
395 stars 129 forks source link

Should ESRI run its own XAPI / Overpass API server? #157

Closed mboeringa closed 7 years ago

mboeringa commented 7 years ago

Hi @cloveman, @eggwhites and @ThomasEmge ,

This is a question that has been creeping up to me lately, especially with the recent service disruption of the main German (X)API instance caused by over-usage of some mobile App.

Now with a potential upcoming new release of the Editor toolbox, and especially the revamped and bug-fixed OSM File Loader (load only) tool combined with the Download OSM Data (XAPI) tool, it is clear ESRI will provide some real powertools for downloading and converting of OSM data to user friendly ArcGIS formats for e.g. cartography and analysis.

While some people will download entire country extracts from e.g. Geofabrik, it is likely many people will want to use the Download OSM Data (XAPI) tool to download custom extents of data for their region on the globe. With a large pool of power GIS users in ESRI's customer base, I can imagine that putting serious strain on the free third party services that provide the (X)API endpoints. Now the amount of data downloadable data from the (X)API, according to this page:

http://wiki.openstreetmap.org/wiki/Overpass_API

seems fairly liberal, but I still wonder if this could potentially result in heavy usage by ESRI customers (including myself here ;-)) at some point in time, and system administrators deciding it is over-usage. I also have the feeling from some other stuff I read, that large rather infrequent batch requests of data - which is a likely usage pattern for the ESRI tools - are far less of a problem than very frequent small requests like a potential mobile App, but still, it may be good to think about this.

It would be really sad to see ESRI users being limited or shut out because of over-use of existing infrastructure. If ESRI had its own copy of an XAPI server the Download OSM Data (XAPI) server could point to, this would be far less of a problem.

I have zero experience with all of this, so can't be of any help here, but I would be interested to hear the views of ESRI on this...

Also see this Wiki page with more info about OSM XAPI: http://wiki.openstreetmap.org/wiki/Xapi

cloveman commented 7 years ago

We do have an internal overpass server setup for demos and prototypes…and we have discussed the value in doing something like you suggest for those reasons you’ve outlined, but no current plans.

From: mboeringa notifications@github.com Reply-To: Esri/arcgis-osm-editor reply@reply.github.com Date: Tuesday, October 4, 2016 at 12:36 PM To: Esri/arcgis-osm-editor arcgis-osm-editor@noreply.github.com Cc: Clint Loveman cloveman@esri.com, Mention mention@noreply.github.com Subject: [Esri/arcgis-osm-editor] Should ESRI run its own XAPI / Overpass API server? (#157)

Hi @clovemanhttps://urldefense.proofpoint.com/v2/url?u=https-3Agithub.com_cloveman&d=DQMCaQ&c=n6-cguzQvX_tUIrZOS_4Og&r=JYw2HCRZvEKA6NAcKLB6aTkaNRCv8jCuviz0miw1zM8&m=I5QG6lFQrQzSwF1SS8tiYDdNFjRXVSY81xWmCvQ69P0&s=xeEgQljYLmbsOqDfsUOi2GtyORIXllCka9QmhqiMuvk&e=, @eggwhiteshttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_eggwhites&d=DQMCaQ&c=n6-cguzQvX_tUIrZOS_4Og&r=JYw2HCRZvEKA6NAcKLB6aTkaNRCv8jCuviz0miw1zM8&m=I5QG6lFQrQzSwF1SS8tiYDdNFjRXVSY81xWmCvQ69P0&s=l4VewgJEMlQ4StDq5vUQNvRde1L4rXgSEyIFUDfJKas&e= and @ThomasEmgehttps://urldefense.proofpoint.com/v2/url?u=https-3Agithub.com_ThomasEmge&d=DQMCaQ&c=n6-cguzQvX_tUIrZOS_4Og&r=JYw2HCRZvEKA6NAcKLB6aTkaNRCv8jCuviz0miw1zM8&m=I5QG6lFQrQzSwF1SS8tiYDdNFjRXVSY81xWmCvQ69P0&s=duAhyoifeaWu4puP2TeoYjoh1HQd3qSMEVaa-G6apSM&e= ,

This is a question that has been creeping up to me lately, especially with the recent service disruption of the main German (X)API instance caused by over-usage of some mobile App.

Now with a potential upcoming new release of the Editor toolbox, and especially the revamped and bug-fixed OSM File Loader (load only) tool combined with the Download OSM Data (XAPI) tool, it is clear ESRI will provide some real powertools for downloading and converting of OSM data to user friendly ArcGIS formats for e.g. cartography and analysis.

While some people will download entire country extracts from e.g. Geofabrik, it is likely many people will want to use the Download OSM Data (XAPI) tool to download custom extents of data for their region on the globe. With a large pool of power GIS users in ESRI's customer base, I can imagine that putting serious strain on the free third party services that provide the (X)API endpoints. Now the amount of data downloadable data from the (X)API, according to this page:

https://urldefense.proofpoint.com/v2/url?u=http-3A__wiki.openstreetmap.org_wiki_Overpass-5FAPI&d=DQIGaQ&c=n6-cguzQvX_tUIrZOS_4Og&r=do0Ib9WwrZ9Iv_kloDpkiesoOCAIzyw_lI-jHGUx3MX-YVWIJtgddKYiqpniT0rOG5TUSg_mbG0aNWQu2c6AGlsNONPvBDeL3AXh-GLMHPD6IHv-0Djbl3Rt9QavMnVN85YURYIvZ_OYxpl4qaNHHA&m=ljeqp3FNKjuH3D5dzxCQ0waFm2UXYXfgipNsf-HFcGc&s=gZMq3OBSyC6ym0WpHm2zAfd0Ab-vRmwZ1c1yqOtKtFg&e= https://urldefense.proofpoint.com/v2/url?u=http-3A__wiki.openstreetmap.org_wiki_Overpass-5FAPI&d=DQMCaQ&c=n6-cguzQvX_tUIrZOS_4Og&r=JYw2HCRZvEKA6NAcKLB6aTkaNRCv8jCuviz0miw1zM8&m=I5QG6lFQrQzSwF1SS8tiYDdNFjRXVSY81xWmCvQ69P0&s=leyKGJIas_T_Ifc2wrhRSwCBGAzctxT0vGAswWtzWNk&e=

seems fairly liberal, but I still wonder if this could potentially result in heavy usage by ESRI customers (including myself here ;-)) at some point in time, and system administrators deciding it is over-usage. I also have the feeling from some other stuff I read, that large rather infrequent batch requests of data - which is a likely usage pattern for the ESRI tools - are far less of a problem than very frequent small requests like a potential mobile App, but still, it may be good to think about this.

It would be really sad to see ESRI users being limited or shut out because of over-use of existing infrastructure. If ESRI had its own copy of an XAPI server the Download OSM Data (XAPI) server could point to, this would be far less of a problem.

I have zero experience with all of this, so can't be of any help here, but I would be interested to hear of the views of ESRI on this...

Also see this Wiki page with more info about OSM XAPI: https://urldefense.proofpoint.com/v2/url?u=http-3A__wiki.openstreetmap.org_wiki_Xapi&d=DQIGaQ&c=n6-cguzQvX_tUIrZOS_4Og&r=do0Ib9WwrZ9Iv_kloDpkiesoOCAIzyw_lI-jHGUx3MX-YVWIJtgddKYiqpniT0rOG5TUSg_mbG0aNWQu2c6AGlsNONPvBDeL3AXh-GLMHPD6IHv-0Djbl3Rt9QavMnVN85YURYIvZ_OYxpl4qaNHHA&m=ljeqp3FNKjuH3D5dzxCQ0waFm2UXYXfgipNsf-HFcGc&s=ztZPIFGN5l__0v8-puGFVkxNaSDgFEumoAi_PYA2KRo&e= https://urldefense.proofpoint.com/v2/url?u=http-3A__wiki.openstreetmap.org_wiki_Xapi&d=DQMCaQ&c=n6-cguzQvX_tUIrZOS_4Og&r=JYw2HCRZvEKA6NAcKLB6aTkaNRCv8jCuviz0miw1zM8&m=I5QG6lFQrQzSwF1SS8tiYDdNFjRXVSY81xWmCvQ69P0&s=PHQAa91XB48KiQVuWcOa6VpQpKmRIjY3rkMIRAIo74c&e=

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://urldefense.proofpoint.com/v2/url?u=https-3Agithub.com_Esri_arcgis-2Dosm-2Deditor_issues_157&d=DQMCaQ&c=n6-cguzQvX_tUIrZOS_4Og&r=JYw2HCRZvEKA6NAcKLB6aTkaNRCv8jCuviz0miw1zM8&m=I5QG6lFQrQzSwF1SS8tiYDdNFjRXVSY81xWmCvQ69P0&s=FX6bDpsnzANPPLH6_cHCqNIaxITzqm_ZNSckJzuvw3Y&e=, or mute the threadhttps://urldefense.proofpoint.com/v2/url?u=https-3Agithub.com_notifications_unsubscribe-2Dauth_AHHZjeFnUxowPV3tdEg-2DW7kz-2DPnP1to2ks5qwqqqgaJpZM4KOEfs&d=DQMCaQ&c=n6-cguzQvX_tUIrZOS_4Og&r=JYw2HCRZvEKA6NAcKLB6aTkaNRCv8jCuviz0miw1zM8&m=I5QG6lFQrQzSwF1SS8tiYDdNFjRXVSY81xWmCvQ69P0&s=hftmUOtHnvrISgSmFk4lhvvs_CLGKo7a6XdN2kGkEfs&e=.

mboeringa commented 7 years ago

Hi @cloveman ,

Thanks for response. Yes, I guess it was logical ESRI already considered this. I assume ESRI has intermittent but regular contact with the OSMF, so in the rare case if any problems would come up due to usage, there is a channel for that... I am not a member of the OSMF myself (yet), nor have I attended the SOTM meetings, so just get my limited info from the internet.

mmd-osm commented 7 years ago

@mboeringa : OSMF is clearly not the right channel for this, see https://lists.openstreetmap.org/pipermail/talk/2016-September/076825.html

BTW: XAPI is quite dated, you should probably consider switching to Overpass QL.

mboeringa commented 7 years ago

@mmd-osm , thanks for the info, but I don't work for ESRI. I am just an experienced ArcGIS product user and programmer, and have helped out with some of the testing of the toolbox maintained in this Open Source repository as I actively use it and thus have an interest in this.

It will be up to @cloveman to potentially act on this.