Overpass protocol - Githubissues

zstadler commented 2 years ago

Feature

The IHM custom layer allows everyone to define a layer to be added to IHM. However, the process is technically challenging:

There has to be an online geojson source for the data.
A MapLibre style has to be defined

Adding a protocol that receives the data from Overpass-Turbo would greatly simplify the creation of data for a custom layer. The Overpass-Turbo wizard can further simplify the process. As a result, the layer data will also be kept up-to-date w.r.t. the underlying OSM data.

To address the complexity of defining a style, the protocol would have a default style. Potentially, this style will be similar to the Overpass-Turbo style.

It would be even better if the protocol would accept simple overrides to the style, such as a URL to an icon form point features or a color for line features.

Example Overpass-Turbo query:

drinking water - following this FB post

I could not think of a layer of line features that would be of interest to hikers or bikers. If that's so, it would be fine for the protocol to show icons only - for point and area features.

Things I think the developers should know, images, links etc.

The code that translates Overpass API results into geojson is maintained by the Overpass-Turbo team, and it is available on Github and as an npm package osmtogeojson

For example, the above drinking water query can be defined as a layer using this syntax:

overpass-turbo:/1nyV

A possible extension could be adding formatting parameters, such as size and icon, as optional URL parameters.

HarelM commented 2 years ago

Reverse engineering overpass-turbo website:

The short link is redirected by the server to a long link: https://overpass-turbo.eu/s/1nyV --> https://overpass-turbo.eu/?Q=nw%0A%20%20%5Bamenity%3D%22drinking_water%22%5D%0A%20%20(area%3A3606195356)%3B%0Aout%20meta%20geom%3B%0A%0A%2F%2F%20rel%20(6195356)%3Bout%20geom%3B&C=31.40054;35.04639;7&R
The response from the long link is an html file with the relevant query inside it as far as I can understand and javascript to run the relevant code
This send a post request to https://overpass-api.de/api/interpreter with the body that is inside the html:

nw
  [amenity="drinking_water"]
  (area:3606195356);
out meta geom;

// rel (6195356);out geom;

So basically, if I'm understanding this correctly I need to do the following steps:

Send a call to the short URL to get the redirect address - the problem here is that I couldn't find any way to do it in javascript as this is done by the browser.
Take the redirect address Q parameter value (the query) and url decode it
Send it using post to the interpreter
Translate the results into geojson using osmtogeojson

Bottom line, I don't think this can be implemented with the short link due to how overpass works, please let me know what else is possible from your perspective as a user.

zstadler commented 2 years ago

Thanks for the analysis!

https://stackoverflow.com/a/33380848 and responseURL doc suggest it is possible to capture the redirect URL in JavaScript:

const xhr = new XMLHttpRequest();
xhr.open('GET', 'https://overpass-turbo.eu/s/1nyV');
xhr.onload = () => {
 console.log(xhr.responseURL);
};
xhr.send(null);

prints

https://overpass-turbo.eu/?Q=nw%0A%20%20%5Bamenity%3D%22drinking_water%22%5D%0A%20%20(area%3A3606195356)%3B%0Aout%20meta%20geom%3B%0A%0A%2F%2F%20rel%20(6195356)%3Bout%20geom%3B&C=31.40054;35.04639;7&R

The Q parameter can then be sent directly to the https://overpass-api.de/api/interpreter API as the data in an HTTP POST request. For example:

curl -d 'data=nw%0A%20%20%5Bamenity%3D%22drinking_water%22%5D%0A%20%20(area%3A3606195356)%3B%0Aout%20meta%20geom%3B%0A%0A%2F%2F%20rel%20(6195356)%3Bout%20geom%3B' https://overpass-api.de/api/interpreter

HarelM commented 2 years ago

I tired that, but since the overpass site doesn't support CORS the xhr request doesn't get to the onload but rather gets to the onerror, in this method the responseURL field is empty and I couldn't find a field in the xhr object that holds the redirected address. Any thoughts or links to SO to solve this would help, I couldn't find anything :-(

HarelM commented 2 years ago

Moreover, I keep reading the following line in all the answers I read:

To answer your question: You can't handle redirects with XHR callbacks because the browser takes care of them automatically. You will only get back what at the redirected location.

https://stackoverflow.com/questions/15996779/cannot-handle-302-redirect-in-ajax-and-why

HarelM commented 2 years ago

Discussion summary: let's jump over the current problem with the short URL and focus on getting this tested assuming the user pastes the full query URL so we can see how this looks and feels...

HarelM commented 2 years ago

After skipping the URL redirect issue, I hard coded a lot of things and got the following to work: I've added a source with the protocol of overpass://some-url-i-made-up-to-see-that-I-get-it and I've added a very simple circle layer just to make sure the data is presented: I think a fuller less hard coded example is needed in order to better understand the manual work that is needed in order to create a custom layer. @zstadler can you please create a style somewhere (any public github repo will do) that has a style that should be constructed as follows:

...
sources: {
   ...
   overpass-example: { // or any other key you want to give it
       type: 'geojson',
       data: "overpass://query-to-overpass-url-encoded" // the example we used in this issue is nw%0A%20%20%5Bamenity%3D%22drinking_water%22%5D%0A%20%20(area%3A3606195356)%3B%0Aout%20meta%20geom%3B%0A%0A%2F%2F%20rel%20(6195356)%3Bout%20geom%3B
   }
},
layers: [
    ... // any way you would like to style this data that uses the overpass-example source
]

If this will work the only question left is if we insist on the short URL or not.

zstadler commented 2 years ago

Here is a style file using circles similar to Overpass-Turbo: Overpass-overlay-example.json

HarelM commented 2 years ago

Thanks!! Can you please remove the source-layer from the layer definition? it shouldn't be there for geojson source.

zstadler commented 2 years ago

Here it is

HarelM commented 2 years ago

Thanks!! this is how it looks :-)

Two things to consider

For some reason the layer is added twice (and removed once in between) which I need to investigate in order to improve performance and avoid sending too many requests to overpass
What to do about the short URL.

This is less than 15 lines of code in total :-)

zstadler commented 2 years ago

There is a free un-shorten API - https://unshorten.me/api.

For example, https://unshorten.me/s/https://overpass-turbo.eu/s/1nyV returns

https://overpass-turbo.eu/?Q=nw%0A%20%20%5Bamenity%3D%22drinking_water%22%5D%0A%20%20(area%3A3606195356)%3B%0Aout%20meta%20geom%3B%0A%0A%2F%2F%20rel%20(6195356)%3Bout%20geom%3B&C=31.40054;35.04639;7&R

zstadler commented 2 years ago

One other thing to consider is the Overpass-Turbo feature that prevents small features from vanishing when zooming-out.

Overpass Turbo seems to do so by temporarily replacing the geometry of the feature with a point geometry when the size of the bounding box is below a threshold. The original geometry is restored when the appropriate zoom-in is done.

Their code is made for Leaflet.

HarelM commented 2 years ago

The above doesn't support CORS unfortunately... :/

Access to XMLHttpRequest at 'https://unshorten.me/s/https://overpass-turbo.eu/s/1nyV' from origin 'http://localhost:5000' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource.

In the style users are free to define min zoom, max zoom etc and style however they see fit, I don't think there's a need here to complicate things...

HarelM commented 2 years ago

Bottom line, at this point I have no more intentions on pursuing the short URL issue. The style editor should take the query, url encode it using an online tool like https://www.urlencoder.org/, add a prefix of overpass:// and set that as the value for the data field in the geojson source, much like can be see in the following example from here:

{
  "version": 8,
  "name": "Overpass-overlay-example",
  "metadata": {
    "owner": "Zeev Stadler",
    "copyright": "Copyright Zeev Stadler 2022. All Rights Reserved."
  },
  "sources": {
    "overpass-points": {
      "type": "geojson",
      "data": "overpass://nw%0A%20%20%5Bamenity%3D%22drinking_water%22%5D%0A%20%20(area%3A3606195356)%3B%0Aout%20meta%20geom%3B%0A%0A%2F%2F%20rel%20(6195356)%3Bout%20geom%3B"
    }
  },
  "layers": [
    {
      "id": "overpass-point",
      "type": "circle",
      "metadata": {"IHM:overlay": true},
      "source": "overpass-points",
      "paint": {
        "circle-radius": 10,
        "circle-color": "#f22",
        "circle-stroke-width": 2,
        "circle-opacity": 0.3,
        "circle-stroke-color": "#d0f",
        "circle-stroke-opacity": 0.7
      }
    }
  ]
}

zstadler commented 2 years ago

I've added a proxy for the unshorten API at https://israelhiking.osm.org.il/unshorten

For example, the body of the response to https://israelhiking.osm.org.il/unshorten/overpass-turbo.eu/s/1nyV is

https://overpass-turbo.eu/?Q=nw%0A%20%20%5Bamenity%3D%22drinking_water%22%5D%0A%20%20(area%3A3606195356)%3B%0Aout%20meta%20geom%3B%0A%0A%2F%2F%20rel%20(6195356)%3Bout%20geom%3B&C=31.40054;35.04639;7&R

zstadler commented 2 years ago

Some Overpass-Turbo query links are LZW-compressed, as noted by a ?q= parameter rather than the ?Q= parameter used for plain text queries.

For example, https://overpass-turbo.eu/s/1ocA is a link to same "drinking water" query above, but it is compressed. The body of the response to https://israelhiking.osm.org.il/unshorten/overpass-turbo.eu/s/1ocA is

https://overpass-turbo.eu/?q=bncKICBbYW1lbml0eT0iZHJpbmvEkWdfd2F0ZXIiXcSCIChhcmVhOjM2MDYxOTUzNTYpOwpvdXQgxId0YSBnZW9txLEKLy8gxKJsxJ_EqcSrxK3ErzvEs8S1xLvEvTs&c=BYAznW9WWH&R

Luckily, the de-compression code is written in JavaScript and it is available in the Overpass-Turbo repository:

amiad commented 1 year ago

@HarelM How I add label to the point? I want to add data from tag of the node.

HarelM commented 1 year ago

Using the style.json. overpass is just the underlying data source.

amiad commented 1 year ago

I tried to add a layer but I must have made a mistake. I'm not a geojson expert.

https://gist.github.com/amiad/e4e7187e906a716ec279fdd0bb28b051

zstadler commented 1 year ago

@amiad,

Maputnik is your friend.

"Open" allows you to load a style from a file or a URL
"Data Sources" allows you to use a copy of the GeoJSON from Overpass-Turbo as a temporary data source
Provides useful error messages
Provides a list of available fonts
More...