Overpass protocol

[zstadler]

Feature

The IHM custom layer allows everyone to define a layer to be added to IHM. However, the process is technically challenging:

1. There has to be an online geojson source for the data.
2. A MapLibre style has to be defined

Adding a protocol that receives the data from Overpass-Turbo would greatly simplify the creation of data for a custom layer. The Overpass-Turbo wizard can further simplify the process. As a result, the layer data will also be kept up-to-date w.r.t. the underlying OSM data.

To address the complexity of defining a style, the protocol would have a default style. Potentially, this style will be similar to the Overpass-Turbo style.

It would be even better if the protocol would accept simple overrides to the style, such as a URL to an icon form point features or a color for line features.

Example Overpass-Turbo query:

• drinking water - following this FB post

I could not think of a layer of line features that would be of interest to hikers or bikers. If that's so, it would be fine for the protocol to show icons only - for point and area features.

Things I think the developers should know, images, links etc.

The code that translates Overpass API results into geojson is maintained by the Overpass-Turbo team, and it is available on Github and as an npm package osmtogeojson

For example, the above drinking water query can be defined as a layer using this syntax:

overpass-turbo:/1nyV

A possible extension could be adding formatting parameters, such as size and icon, as optional URL parameters.

[HarelM]

Reverse engineering overpass-turbo website:

1. The short link is redirected by the server to a long link: https://overpass-turbo.eu/s/1nyV --> https://overpass-turbo.eu/?Q=nw%0A%20%20%5Bamenity%3D%22drinking_water%22%5D%0A%20%20(area%3A3606195356)%3B%0Aout%20meta%20geom%3B%0A%0A%2F%2F%20rel%20(6195356)%3Bout%20geom%3B&C=31.40054;35.04639;7&R
2. The response from the long link is an html file with the relevant query inside it as far as I can understand and javascript to run the relevant code
3. This send a post request to https://overpass-api.de/api/interpreter with the body that is inside the html:

nw
  [amenity="drinking_water"]
  (area:3606195356);
out meta geom;

// rel (6195356);out geom;

So basically, if I'm understanding this correctly I need to do the following steps:

1. Send a call to the short URL to get the redirect address - the problem here is that I couldn't find any way to do it in javascript as this is done by the browser.
2. Take the redirect address Q parameter value (the query) and url decode it
3. Send it using post to the interpreter
4. Translate the results into geojson using osmtogeojson

Bottom line, I don't think this can be implemented with the short link due to how overpass works, please let me know what else is possible from your perspective as a user.

[zstadler]

Thanks for the analysis!

1. https://stackoverflow.com/a/33380848 and responseURL doc suggest it is possible to capture the redirect URL in JavaScript:

   const xhr = new XMLHttpRequest();
   xhr.open('GET', 'https://overpass-turbo.eu/s/1nyV');
   xhr.onload = () => {
    console.log(xhr.responseURL);
   };
   xhr.send(null); 

   prints

   https://overpass-turbo.eu/?Q=nw%0A%20%20%5Bamenity%3D%22drinking_water%22%5D%0A%20%20(area%3A3606195356)%3B%0Aout%20meta%20geom%3B%0A%0A%2F%2F%20rel%20(6195356)%3Bout%20geom%3B&C=31.40054;35.04639;7&R

2. The Q parameter can then be sent directly to the https://overpass-api.de/api/interpreter API as the data in an HTTP POST request. For example:

   curl -d 'data=nw%0A%20%20%5Bamenity%3D%22drinking_water%22%5D%0A%20%20(area%3A3606195356)%3B%0Aout%20meta%20geom%3B%0A%0A%2F%2F%20rel%20(6195356)%3Bout%20geom%3B' https://overpass-api.de/api/interpreter

[HarelM]

I tired that, but since the overpass site doesn't support CORS the xhr request doesn't get to the onload but rather gets to the onerror, in this method the responseURL field is empty and I couldn't find a field in the xhr object that holds the redirected address. Any thoughts or links to SO to solve this would help, I couldn't find anything :-(

[HarelM]

Moreover, I keep reading the following line in all the answers I read:

To answer your question: You can't handle redirects with XHR callbacks because the browser takes care of them automatically. You will only get back what at the redirected location.

https://stackoverflow.com/questions/15996779/cannot-handle-302-redirect-in-ajax-and-why

[HarelM]

Discussion summary: let's jump over the current problem with the short URL and focus on getting this tested assuming the user pastes the full query URL so we can see how this looks and feels...

[HarelM]

After skipping the URL redirect issue, I hard coded a lot of things and got the following to work: I've added a source with the protocol of overpass://some-url-i-made-up-to-see-that-I-get-it and I've added a very simple circle layer just to make sure the data is presented: I think a fuller less hard coded example is needed in order to better understand the manual work that is needed in order to create a custom layer. @zstadler can you please create a style somewhere (any public github repo will do) that has a style that should be constructed as follows:

...
sources: {
   ...
   overpass-example: { // or any other key you want to give it
       type: 'geojson',
       data: "overpass://query-to-overpass-url-encoded" // the example we used in this issue is nw%0A%20%20%5Bamenity%3D%22drinking_water%22%5D%0A%20%20(area%3A3606195356)%3B%0Aout%20meta%20geom%3B%0A%0A%2F%2F%20rel%20(6195356)%3Bout%20geom%3B
   }
},
layers: [
    ... // any way you would like to style this data that uses the overpass-example source
]

If this will work the only question left is if we insist on the short URL or not.

[zstadler]

Here is a style file using circles similar to Overpass-Turbo: Overpass-overlay-example.json

[HarelM]

Thanks!! Can you please remove the source-layer from the layer definition? it shouldn't be there for geojson source.

[zstadler]

Here it is

[HarelM]

Thanks!! this is how it looks :-)

Two things to consider

1. For some reason the layer is added twice (and removed once in between) which I need to investigate in order to improve performance and avoid sending too many requests to overpass
2. What to do about the short URL.

This is less than 15 lines of code in total :-)

[zstadler]

There is a free un-shorten API - https://unshorten.me/api.

For example, https://unshorten.me/s/https://overpass-turbo.eu/s/1nyV returns

https://overpass-turbo.eu/?Q=nw%0A%20%20%5Bamenity%3D%22drinking_water%22%5D%0A%20%20(area%3A3606195356)%3B%0Aout%20meta%20geom%3B%0A%0A%2F%2F%20rel%20(6195356)%3Bout%20geom%3B&C=31.40054;35.04639;7&R

[zstadler]

One other thing to consider is the Overpass-Turbo feature that prevents small features from vanishing when zooming-out.

Overpass Turbo seems to do so by temporarily replacing the geometry of the feature with a point geometry when the size of the bounding box is below a threshold. The original geometry is restored when the appropriate zoom-in is done.

Their code is made for Leaflet.

[HarelM]

The above doesn't support CORS unfortunately... :/

Access to XMLHttpRequest at 'https://unshorten.me/s/https://overpass-turbo.eu/s/1nyV' from origin 'http://localhost:5000' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource.

In the style users are free to define min zoom, max zoom etc and style however they see fit, I don't think there's a need here to complicate things...

[HarelM]

Bottom line, at this point I have no more intentions on pursuing the short URL issue. The style editor should take the query, url encode it using an online tool like https://www.urlencoder.org/, add a prefix of overpass:// and set that as the value for the data field in the geojson source, much like can be see in the following example from here:

{
  "version": 8,
  "name": "Overpass-overlay-example",
  "metadata": {
    "owner": "Zeev Stadler",
    "copyright": "Copyright Zeev Stadler 2022. All Rights Reserved."
  },
  "sources": {
    "overpass-points": {
      "type": "geojson",
      "data": "overpass://nw%0A%20%20%5Bamenity%3D%22drinking_water%22%5D%0A%20%20(area%3A3606195356)%3B%0Aout%20meta%20geom%3B%0A%0A%2F%2F%20rel%20(6195356)%3Bout%20geom%3B"
    }
  },
  "layers": [
    {
      "id": "overpass-point",
      "type": "circle",
      "metadata": {"IHM:overlay": true},
      "source": "overpass-points",
      "paint": {
        "circle-radius": 10,
        "circle-color": "#f22",
        "circle-stroke-width": 2,
        "circle-opacity": 0.3,
        "circle-stroke-color": "#d0f",
        "circle-stroke-opacity": 0.7
      }
    }
  ]
}

[zstadler]

I've added a proxy for the unshorten API at https://israelhiking.osm.org.il/unshorten

For example, the body of the response to https://israelhiking.osm.org.il/unshorten/overpass-turbo.eu/s/1nyV is

https://overpass-turbo.eu/?Q=nw%0A%20%20%5Bamenity%3D%22drinking_water%22%5D%0A%20%20(area%3A3606195356)%3B%0Aout%20meta%20geom%3B%0A%0A%2F%2F%20rel%20(6195356)%3Bout%20geom%3B&C=31.40054;35.04639;7&R

[zstadler]

Some Overpass-Turbo query links are LZW-compressed, as noted by a ?q= parameter rather than the ?Q= parameter used for plain text queries.

For example, https://overpass-turbo.eu/s/1ocA is a link to same "drinking water" query above, but it is compressed. The body of the response to https://israelhiking.osm.org.il/unshorten/overpass-turbo.eu/s/1ocA is

https://overpass-turbo.eu/?q=bncKICBbYW1lbml0eT0iZHJpbmvEkWdfd2F0ZXIiXcSCIChhcmVhOjM2MDYxOTUzNTYpOwpvdXQgxId0YSBnZW9txLEKLy8gxKJsxJ_EqcSrxK3ErzvEs8S1xLvEvTs&c=BYAznW9WWH&R

Luckily, the de-compression code is written in JavaScript and it is available in the Overpass-Turbo repository:

• parameter handling
• query decompression

[amiad]

@HarelM How I add label to the point? I want to add data from tag of the node.

[HarelM]

Using the style.json. overpass is just the underlying data source.

[amiad]

I tried to add a layer but I must have made a mistake. I'm not a geojson expert.

https://gist.github.com/amiad/e4e7187e906a716ec279fdd0bb28b051

[zstadler]

@amiad,

Maputnik is your friend.

• "Open" allows you to load a style from a file or a URL
• "Data Sources" allows you to use a copy of the GeoJSON from Overpass-Turbo as a temporary data source
• Provides useful error messages
• Provides a list of available fonts
• More...

Also, out geom; would be sufficient.

[amiad]

תודה, אחרי העזרה ממך ומהראל הצלחתי להגדיר הכול כמו שרציתי.

[zstadler]

Great!

Would you be interested in publishing your work

1. As an additional overlay, similar to the waterhole status overlay?
2. As a new Wiki page for overpass-based overlays?

[amiad]

כן, רק אני לא בטוח מה העניין לציבור. המטרה האישית שלי היא להציג את סוג העץ גם במקרה שהוא לא קיים ב־name (או שמשתמש חרוץ מחק אותו משם)

[HarelM]

דוגמא לשימוש ביכולת, אולי מישהו אחר ירצה דוגמה שעובדת...