Offline navigation via OSM Scout Server

[jonnius]

Would you be interested in letting unav use the OSM Scout Server for offline navigation in uNav? If so I would include Mapnik in OSM Scout Server to provide the map tiles.

[jonnius]

@rinigus FYI

[costales]

Hi, sure, but I think OSM Scout is using vectorial maps and uNav is using tiles map (under openlayers). Best regards.

[rinigus]

Hi! OSM Scout Server has also Mapnik-based rendering into raster tiles as well: https://github.com/rinigus/osmscout-server#raster-tiles . Its actually was built originally with raster tile support that later extended to Mapbox GL vector tiles.

When compared to the online servers, there speed is better if you request larger tiles, as 1024x1024. That way border area handling will not impact as much as in the case of smaller tiles. Other than that as well as not using some small timeout (timeouts on mobile will create issues as device may enter sleep in the middle of the tile generation), there is not that much different from regular servers.

[costales]

WIP :)

[rinigus]

Excellent! Good luck with it. Let me know if there are any questions.

Note, that there is also API for geocoding, reverse geocoding, routing. So, tiles are just a part of it.

[jonnius]

Make sure to use OSM Scout Server Full if you need Mapnik.

[costales]

Hi,

uNav is not finding the tiles.

I installed OSM Scout Server Full and as Profile in the 2nd wizard screen I choosed: Recommended for raster tiles maps and in the last one: Automatic activation. Then I downloaded a map.

I'm using with this tile URL:

http://localhost:8553/v1/tile?style=mapnik&daylight=1&z={z}&x={x}&y={y}

Is that URL OK? Is possible to set a .png filename?

Thanks in advance!

[rinigus]

Hi,

URL looks fine. On UT, automatic activation doesn't work as you don't use systemd. So you have to run the server in the background and disable suspending the server. Here instructions from open-store:

Preparations:

• Disable app suspension for OSM Scout Server using the UT Tweak Tool: https://open-store.io/app/ut-tweak-tool.sverzegnassi

As for .png, I would prefer not to change the API and keep it as it is. modRana, web browsers through leaflet are all fine with such URL. Let's see what will happen when it works in the background.

The server should also display URLs accessed on its window allowing you to debug the connection.

Let me know how it went.

[costales]

Thanks for the reply @rinigus

I disable the app suspension, but no lucky yet.

OSM Scout Server Full sees the requests:

but uNav doesn't show the tiles:

One question: I don't see any button to start the OSM Scout server. Is that right?

Thanks in advance!

[rinigus]

As soon as you start the app, it would serve the tiles and all the rest. Try to unzoom for lower z values. Maybe you are in the region without the map data... As for the log on the server, it looks to be getting the requests correctly

[rinigus]

Another check is to set daylight=1 to daylight=0. It should change the tile color to darker one, even if there is no data there.

Please also check if server's map manager did download the data. I.e. your region is marked as available.

[costales]

Hi,

I did a so simple uNav and did tests.

Leaflet with Carto (works): https://github.com/costales/unavtest1/blob/master/nav/index.html#L28

Leaflet with OSM Scout (works): https://github.com/costales/unavtest2/blob/master/nav/index.html#L28

openLayers with Carto (works): https://github.com/costales/unavtest3/blob/master/nav/index.html#L22

openLayers with OSM Scout (not works): https://github.com/costales/unavtest4/blob/master/nav/index.html#L22

I think is an issue with openLayers. I will investigate more.

Best regards.

[rinigus]

Thanks, great to hear it works with leaflet. I could see it is not loading for me with openlayers either (using https://github.com/rinigus/osmscout-server/blob/master/examples/js/openlayers.html). leaflet, in the same folder, has no issues. strange and don't know why

[costales]

Is there a way to invoque the server with a filename .png? (I'm not requesting to change OSM Server, just if it is possible to do that test with the current OSM Server release).

Best regards.

[rinigus]

No, unfortunately, it's impossible. API is parsed via HTTP URL and I would have to rewrite request mapper used in the server. All communication with the server is expected to be via HTTP and, for map matching, via DBus. So, we are rather limited with .png extension.

[rinigus]

This fixed the server example with openlayers: https://github.com/rinigus/osmscout-server/commit/d2ba707c2b98e137b52660bb47dcfab07c0f188d

Best wishes

PS: getting late over here, will be able to catch up tomorrow

[costales]

Hi, I will try it tomorrow. Good night and thank you so much for the support!

[rinigus]

Good luck!

[costales]

Hi.

Tiles working as expecting :)))

Thanks a lot!!

[rinigus]

Excellent! Next step: geocoder? Nearby search? Router?

[costales]

Excellent! Next step: geocoder? Nearby search? Router?

I want to rewrite uNav first, do it more simple and lighter, it will take me probably 1 or 2 months.

Yes, I'll include everything for offline mode :)))

A hug.

[costales]

Hi,

After a lot of tests the raster tiles are so slow for using in uNav. For example, when I do zoom is taking 1 minute to render so much tiles :(

I'm playing then with the vector maps, but with a not complete luck. Could you help me in this please?

For example, the map [tutorial] is loaded with this code, but the style is so poor.

I need to apply a nice style (mapbox, osmbright, ...?), but I don't find how to do it with openlayers. I saw that you have a style here:

http://localhost:8553/v1/mbgl/style?style=osmbright

How could I load it in openlayers?

Thanks in advance!!

[rinigus]

Will be happy to help. I will check it out on how to apply in openlayers. Let me think how you could reuse as much as possible...

[costales]

Thanks a lot!

I was investigating more without luck.

I found this example using olms, but when I run uNav is looks weird:

A hug.

[rinigus]

Several comments regarding the change, in no particular order. Note that I don't know much about OpenLayers, take that into account when reading below.

1. Switching to vector styles makes sense. It is way faster as part of rendering job is done while importing data into vector tiles and it is also performed on GPU. As a result, you get faster response. I guess, in our case, you could check performance of Pure Maps on the same hardware and see what to expect with the offline maps. Originally, I was using raster tiles as well. After the test and corresponding switch, I never look back and would recommend it to anyone.

2. When testing Mapnik style, how did you render the tiles? Have you used large tiles, such as 1024x1024, as suggested in OSM Scout Server readme? Are tiles cached once generated by uNav? While caching will not help with the first load, it maybe helpful otherwise... (This was a side note, going back to vector).

3. With vector tiles you can integrate routes and other objects between the layers of the map. That way, Pure Maps puts route under map labels. However, to make it possible, you may have to use Mapbox GL JS or Mapbox GL Native (either official version in Qt 5.9 [tech preview] or unofficial one at https://github.com/rinigus/mapbox-gl-qml). I would recommend the unofficial plugin, but I could be biased. Note that Mapbox GL JS/Native support rendering of raster tiles as well. For example, HERE maps as shown by Pure Maps are using this feature.

4. OSM Scout Server does not know how create raster tiles from vector data, unfortunately. So, we have to use vector tiles directly and cannot go around it by generation of raster tiles.

5. When showing vector maps, you would have to load the style and not vector data directly. Otherwise, you would have to start redoing every rendering step that is made by style usually. So, in this respect, let's focus on style rendering approach and then we have to load data through the style and not access vector tiles directly. I have updated README of the server accordingly.

6. Example at https://openmaptiles.org/docs/website/openlayers/ suggests to use https://github.com/openlayers/ol-mapbox-style, as you did. But notice code at https://github.com/openmaptiles/www.openmaptiles.org/tree/master/maps and that in the example they call olms with the style as an argument.

7. I don't know how good is olms in converting mapbox gl style and whether it is fully supporting all used features. That would come out during experiments, I guess. But there is a risk that you would try to use converter instead of using the original approach developed for these maps (Mapbox GL JS/Native). On the other hand, I presume that full uNav is written using OpenLayers and it is an extra work to move out from it.

I hope that this info is helpful. I will be happy to help further, just ask :) . It would make sense to share as much as we can in these projects and surely switch to vector tiles could only improve that.

[rinigus]

What you see in your tests are essentially data that comes from vector tiles. Unfortunately, it is not super simple and you have to consider 2 parts:

• The data layers are imported and distributed as vector tiles. In this case, import layers are described in https://github.com/rinigus/mapbox-gl-importer/tree/master/layers

• Rendering is described by styles. The server is using styles from https://github.com/rinigus/mapbox-gl-styles . Again, we have layers at https://github.com/rinigus/mapbox-gl-styles/tree/master/styles/src/layers combined together into the style as in https://github.com/rinigus/mapbox-gl-styles/tree/master/styles/src/osmbright

To get into it more, I would suggest to read Mapbox GL tutorials and look at their style studio.

[costales]

Hi there!

1.

Switching to vector styles makes sense. It is way faster as part of rendering job is done while importing data into vector tiles and it is also performed on GPU. As a result, you get faster response. I guess, in our case, you could check performance of Pure Maps on the same hardware and see what to expect with the offline maps. Originally, I was using raster tiles as well. After the test and corresponding switch, I never look back and would recommend it to anyone.

Yes, I tried several times to switch to vector maps, but no results.

1.

When testing Mapnik style, how did you render the tiles? Have you used large tiles, such as 1024x1024, as suggested in OSM Scout Server readme? Are tiles cached once generated by uNav? While caching will not help with the first load, it maybe helpful otherwise... (This was a side note, going back to vector).

Yes, 1024px. The tiles are not cached in uNav. They are working right, but not when uNav ask for a query of 30 tiles because the user did a zoom fast.

1.

With vector tiles you can integrate routes and other objects between the layers of the map. That way, Pure Maps puts route under map labels. However, to make it possible, you may have to use Mapbox GL JS or Mapbox GL Native (either official version in Qt 5.9 [tech preview] or unofficial one at https://github.com/rinigus/mapbox-gl-qml). I would recommend the unofficial plugin, but I could be biased. Note that Mapbox GL JS/Native support rendering of raster tiles as well. For example, HERE maps as shown by Pure Maps are using this feature.

Yes, but in uNav everything is using openlayers and switch to other library would be tedious.

1.

OSM Scout Server does not know how create raster tiles from vector data, unfortunately. So, we have to use vector tiles directly and cannot go around it by generation of raster tiles.

One question, the OSM Scout Server is using MVT format?

1.

When showing vector maps, you would have to load the style and not vector data directly. Otherwise, you would have to start redoing every rendering step that is made by style usually. So, in this respect, let's focus on style rendering approach and then we have to load data through the style and not access vector tiles directly. I have updated the README of the server accordingly.

openLayers is complex, but in all examples I saw they declare the vector tiles and then a style. I think is openlayers who deal with that, but I'm not sure.

1.

Example at https://openmaptiles.org/docs/website/openlayers/ suggests to use https://github.com/openlayers/ol-mapbox-style, as you did. But notice code at https://github.com/openmaptiles/www.openmaptiles.org/tree/master/maps and that in the example they call olms with the style as an argument. 2.

I don't know how good is olms in converting mapbox gl style and whether it is fully supporting all used features. That would come out during experiments, I guess. But there is a risk that you would try to use converter instead of using the original approach developed for these maps (Mapbox GL JS/Native). On the other hand, I presume that full uNav is written using OpenLayers and it is an extra work to move out from it.

Yes, use mapbox would be a last option. openLayers is the most powerful map library I ever saw, there has to be a way to do this.

I need to invert time in other features, if not, I can't progress in the new release. I'll come back to this trying to do it working.

A hug and thank you so much :)

[rinigus]

I understand that switching away from OpenLayers would be a major effort. It took me some time to switch Poor Maps (older name at that time) from raster to vector, so I know exactly what is involved in it.

Replies to your questions:

1. Yes, OSM Scout Server does use MVT format. That's why you were able to render the coastlines and such above.

2. Maybe OpenLayers can reuse Mapbox GL style. It is surely worth to test.

Good luck with the new release and this part can surely wait...

[costales]

Finally I got it, using olms! You were right, I have to use only the style URL and not the tiles.

uNav 3 will be released with your vector maps for offline use! :)))

Another question more please, do you know an online services same as OSM Server for routes and search? I am asking it for do the same code for online & offline responses from the APIs.

Thank you!

[rinigus]

Excellent news - congratulations!

Re search: no, unfortunately the search engine I wrote myself and it has an API that was designed for it.

Re routing: I am using Stadia Maps https://stadiamaps.com by default as they are using the same routing engine as OSM Scout Server: Valhalla. Free tier gives 2500 requests per day which was so far sufficient. Valhalla project at https://github.com/valhalla/valhalla, API described in docs at https://github.com/valhalla/valhalla/blob/master/docs/api/turn-by-turn/api-reference.md

Best wishes!

[jonnius]

Just for the record: does that mean unav3 won't need OSM Scout Server with Mapnik? In that case I would unpublish it from the OpenStore.

[costales]

Hi!

It means that uNav will use OSM Scout Server Slim, not the Full. But, please, wait a few weeks, I need to finish the release and test it a lot with these maps.

Thanks to all of you! A hug!

[costales]

Please @jonnius if you can, remove the Full version from the store if you want. Thanks a lot!

[jonnius]

Ok. Looking forward to the uNav release with Offline navigation!

[costales]

Thanks :) I'm with the ~50% complete... I am looking too xD

Thanks to all of you for OSM Scout Server! A hug!

[rinigus]

Looking forward, Great that you work on it!

[costales]

Hi @rinigus

Is a way to do reverse geocoding? For a latitude & longitude returns the place name.

Thanks in advance.

[rinigus]

@costales , sure, there is reverse geocoder: https://github.com/rinigus/osmscout-server#poi-search-near-a-reference-position-or-route

For list of possible POI types, request as in https://github.com/rinigus/osmscout-server#list-of-available-poi-types

Note that the reverse geocoder can also find POIs along the route. As for example, when you drive, show me the cafe within 200 meters from the route starting from current location. Handy on long trips!

[costales]

Hi,

But I don't need POIs in this case :) I would need for example, the name of the street of those coordinates.

For example: http://photon.komoot.de/reverse?lon=10&lat=52 would give for that lon & lat the name of "Geographischer Punkt".

How would be the URL in OSM Scout for returning that? Something like this one?

http://localhost:8553/v1/guide?limit=1&lng={lng}&lat={lat}

Thanks!

[rinigus]

Streets are tricky, it may not find them. But the houses with numbers/names are also considered as POIs. Default values are given in https://github.com/rinigus/osmscout-server/blob/master/src/requestmapper.cpp#L723 . So, without radius specified, it will be 100 meters.

In Pure Maps, I am exactly using this way to find the objects when user is long tapping on the screen. However, I specify radius depending on map zoom as it would be roughly when covering it by a finger.

As for streets, it is tricky as all objects are stored as points in the geocoder. So, while streets are there, it does not have full polyline. Thus, if you tap on a street, but not next to the stored point, you will miss it. Its mainly for storage reasons and avoiding to have larger databases.

[costales]

Hi,

If it doesn't return anything, it's OK.

Then, using this http://localhost:8553/v1/guide?limit=1&lng={lng}&lat={lat} would be the right way? With limit=1 would give the nearest? Any other URL compose for better information about the user click on map and give him the name of that point?

Thanks!

[costales]

Hi,

With this URL http://localhost:8553/v1/guide?limit=1&lng=-5.668974148516006&lat=43.53230762045101 I'm getting in the OSM Scout log:

Request: /v1/guide?limit=1&lng=-5.668974148516006&lat=43.53230762045101 Error while reading guide query parameters

[rinigus]

Just checked, you have to use slightly different:

http://localhost:8553/v1/guide??radius=XXXX&limit=1&lng=XXXX&lat=XXXXX&poitype=any

as you see, poitype has to be specified as well

[rinigus]

PS: note that radius is not obligatory

[rinigus]

PPS: you need to specify either poitype or name (or both). hence the placeholder any

[costales]

Hi,

I'm so sorry, but it's not working for me. I tried from the shell for know what is happening:

wget http://localhost:8553/v1/guide?limit=1&lng=-5.668974&lat=43.532307&poitype=any Resolving localhost (localhost)... ::1, 127.0.0.1 Connecting to localhost (localhost)|::1|:8553... failed: Connection refused. Connecting to localhost (localhost)|127.0.0.1|:8553... connected. HTTP request sent, awaiting response... 400 Bad Request 2020-08-12 23:12:00 ERROR 400: Bad Request.

Appears the server is not working, but it gives tiles and searches...

[costales]

From clickable logs I'm getting:

qml: [JS] (file:///opt/ click.ubuntu.com/navigator.costales/3.0/nav/index.html:0) Access to XMLHttpRequest at ' http://localhost:8553/v1/guide?limit=10&lng=-6.598679253710262&lat=43.77830779959967&poitype=any' from origin 'file://' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource.

Any idea?

[rinigus]

Re wget example: are you sure you don't need to enclose URL in ''? When testing on PC, it works (small JSON returned)

Re clickable logs: no idea, sorry, as I don't know enough about UT clickable.

Let's try to get it to work with terminal. Please test search from terminal as well, maybe server got suspended by UT?

[costales]

Hi,

Yes, wget needed the quotes, with that worked :)

I tried ajax and XMLHttpRequest for getting the data and it's not working because of CORS policy:

qml: [JS] (file:///opt/click.ubuntu.com/navigator.costales/3.0/nav/index.html:0) Access to XMLHttpRequest at 'http://localhost:8553/v1/guide' from origin 'file://' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource.

It's a bit weird, because it's localhost. Is the server returning a header like this?

Access-Control-Allow-Origin: *

Thanks in advance

[rinigus]

TBH, I don't remember. I would expect it to return the same headers as the search does. Is it possible to somehow check it?