Maps no longer updating on new vector tile server, out-of-date with quests

[matkoniecz]

Yes, #835 is back and #747 continues to cause problems.

Firstly, thanks to @Akasch for his great work in setting up a new vector tile server for rescue StreetComplete from the demise of Mapzen.

I am opening this issue to mention that as the server is not updating at the moment (and I'm sure it is probably not trivial to do so easily or cheaply), and therefore the map is no longer up-to-date with the OSM data (and therefore StreetComplete quests).

Is it possible to restart updates or run cyclic updates (once a month? once 3 months?) or switch to an alternative background map to StreetComplete that pulls direct from osm.org rendered tiles?

Currently it is not fatal problem in areas that are not significantly changed but it will steadily get worse if nothing is changed.

[matkoniecz]

I tested it by reinstalling app (what purges data) and downloading area where I changed well-visible data over 10 days ago ( https://www.openstreetmap.org/way/134071005#map=17/50.07755/19.93830 in that case ).

From what I know data was supposed to be updated continuously, what worked for some time and stopped.

I am not sure is it possible that data update will return in any form (even limited one like reload every 3 months).

[rugk]

Hmm, it all looks as if we maybe need a better powerful server there. But @westnordost still rejects any donations. Or another method here. Or you can e.g. apply for the Thomas-Krenn-Award, which not only brings publicity, but they also donate hardware…

[westnordost]

A server is not a problem, terrestris runs one and the person maintaining it said it would be ok if SC used it. But the format is a little different, i.e. the stylesheet would need to be changed.

[westnordost]

Is this still an issue? I think the server just had a hickup, maybe?

[dbdean]

Absolutely, the maps in my area haven't fully updated in at least 4 months.

[westnordost]

Hm, damn. The admin of the server also told me that he is quite busy right now (with other things), so no chance one can expect this will be solved anytime soon

On May 28, 2018 12:06:31 AM GMT+02:00, David Dean notifications@github.com wrote:

Absolutely, the maps in my area haven't fully updated in at least 4 months.

[matkoniecz]

Since I created this issue (and weeks before) I have seen no updates and plenty of cases of outdated map.

[rugk]

A server is not a problem, terrestris runs one and the person maintaining it said it would be ok if SC used it. But the format is a little different, i.e. the stylesheet would need to be changed.

Well…, but if you run your own server, you could get everything in the format you like.

[davidak]

And when multiple trusted members would have access to the server, problems can be solved faster, even if one don't have the time. Documentation is also important, in best case directly in configuration management. Minimizing the bus-factor...

[Akasch]

I have deployed a test, now Germany should be a more recent (I think 23.4. catching up, and the old tiles will be cached between 7 and 14 Days), all other Data is more old.

If this test is working it would be possible to split the Server to multiple small ones. As small Server are a lot cheaper for combined capacity, and it is easy possible to use unused capacity on private virtual servers this could be a model for a community driven Tile-server. I know that at the moment there will be a Problem with the tiles on the Border, I will fix it in the next days.

I think that we should put the Discussion about the Tileserver into its own repo/issue tracker. I would put a repo with configs etc up in the next day at the https://github.com/Map-Data Organisation.

[rugk]

Well, a "community-driven" sever would have privacy problems, so is rather now who runs the server I get tiles from.

[matkoniecz]

would have privacy problems

What would be worse than it is now? Location of given user is identifiable anyway by analysis of changes made.

[Akasch]

At the Moment only the fronted Cache Server is getting the IP address from the User, the Server Producing the Tiles is only seeing the IP of the fronted web Servers, and at the moment the User Agent which is always "okhttp/3.5.0" for StreetComplete. I think that it is not possible to correlate users to requests.

The Structure at the moment is like this:

Cacheserver01 -----               |------------------> Tileserver Germany
                   |-----> Master Server
Cacheserver02 -----               |------------------> Tileserver World

The Master Server is routing the requests to the correct Tilesserver, and is serving the prerendered Tiles. For a better availability the master server could be duplicated.

[matkoniecz]

In that case there is almost no reason to consider privacy as an issue.

[rugk]

Ah yes, if users could just host the backend servers yeah. This would be useful.

To answer the previous question: users may/do not always answer quests, but when they have the app open tiles are always downloaded.

[westnordost]

Tiles seem to not load anymore, @Akasch

[Akasch]

@westnordost deployed a workaround, should work again, will look into the causes later

[dbdean]

Any movement on getting worldwide up-to-date maps happening again?

[cstackpole]

I was just in an area that I haven't been to in a long time. The tiles were all downloaded. Yet I had house quests without outlines. I too request a up-to-date map. Thank you for everyone helping out with this!

[francois2metz]

I would be happy to host the France tile server (or more depending of the load). The proposition to route each traffic country to a specific instance is nice IMHO.

But I have no idea how to setup a tile server and update it. Anyone to give me a link to some doc?

[pkoby]

I've moved to a new area and also have no information on my maps, despite having added a lot of the buildings to OSM a few months back. This is not really workable, as others have said, without the context for which building you're updating.

I understand the issues with servers and whatnot, but perhaps we could get an option to use offline maps? I use c:geo for geocaching, and it has this option. You can download maps from http://download.mapsforge.org/maps/, then point the app to use them instead of, say, Google maps or Mapnik via wifi or data.

The size of the maps is not trivial, but I'd like the ability to update the basemap as necessary by downloading the mapsforge option much more frequently (about every month they update).

EDIT: alternatively, Maps.me has a download page (http://direct.mapswithme.com/regular/daily/), but I'm not sure what the license would be on that.

[matkoniecz]

offline maps are discussed in #122 (tl;dr: "This is not possible yet with the map rendering library I use").

[westnordost]

Actually, it would be when implemented manually. A nested nested for-loop that downloads all tiles from zoom level A to B within a bounding box C,D,E,F and stores these tiles in a local SQLite database. Tangram-es can then be pointed at that database for offline usage. However, that is not an effort I am prepared to make, because I do not want to invest time and effort into the dealing with the map library.

(Needless to say, this would put quite a strain on the tileserver. I have the impression that even now, the map-data tileserver we use does not cope with the traffic)

[rugk]

I have the impression that even now, the map-data tileserver we use does not cope with the traffic

Only @Akasch can say for sure.

[ENT8R]

Is there something which speaks against using Mapbox Vector Tiles? I know that one has to adapt the stylesheet, but that should be no big problem... But at least those tiles are always up-to-date...

[matkoniecz]

The MapBox tileserver is only allowed to be used through the MapBox SDK. I initially decided against MapBox because they have some telemetry built-in and their terms and conditions require to have this by default on (only letting the user switch it off in the settings).

https://github.com/westnordost/StreetComplete/issues/747#issuecomment-354842669

[matkoniecz]

And at this point it would require significant(?) app rewrite to make that switch.

[matkoniecz]

Also

However, I set for Tangram-ES instead of MapBox early in development because of this. I have no control over what and when the map renderer calls home, which is adverse to my goal to make this app very light on data usage and able to work offline. I asked at MapBox a few years ago whether as an app developer, I can select the "opt out" to be the default, the answer was no.

https://github.com/westnordost/StreetComplete/issues/40#issuecomment-289603003

[ENT8R]

I just did some testing but with Tangram Play I could retrieve and render Mapbox Vector tiles... I can do some further testing but I think if it works with Tangram (JS) it should also work with Tangram-ES, shouldn't it?

[ENT8R]

I converted nearly everything now... Some things are still missing but you should get an overview; https://ent8r.github.io/streetcomplete-mapstyle/mapbox/ What I don't understand is that it is working because they say in their Terms of service:

The "Mapping APIs" means the Maps API, Styles API, Static API, Static (Classic) API and Tileset API. You must access and use map tiles, static map images, style files, glyphs, and sprites that we provide to you (collectively "Map Assets”) only through our APIs.

Is it illegal to provide such a website/map then?

[westnordost]

There is another vector tile server maintained by terrestris this project could use for free but I did not get into contact with them yet because it seemed that the current solution works ok.

What is the reason you think a switch is necessary now?

[dbdean]

Because our maps are six months out of date.

On Tue., 7 Aug. 2018, 20:50 Tobias Zwick, notifications@github.com wrote:

There is another vector tile server maintained by terrestris this project could use for free but I did not get into contact with them yet because it seemed that the current solution works ok.

What is the reason you think a switch is necessary now?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/westnordost/StreetComplete/issues/1011#issuecomment-411016152, or mute the thread https://github.com/notifications/unsubscribe-auth/AMIsPi4atNyN-VnoYWGx3xsCGuezwFfAks5uOXDygaJpZM4TM2N9 .

-- http://dbdean.com

[cstackpole]

Because our maps are six months out of date.

I second this. I've temporarily stopped recommending StreetComplete to the new mappers because they kept having issues/questions about why the map was "wrong". Instead of them getting a bad impression (of StreetComplete, OSM, ect) or getting frustrated, I just asked them to wait. :-(

[matkoniecz]

What is the reason you think a switch is necessary now?

Because maps are not updating since at least April (and I created this issue because maps were not updating for a long time), and I see no easier way to change that.

[pkoby]

Maybe we could have a basemap selection option? So default could stay Tangram, but allow users to select another source (like Mapbox), making clear the caveats with each layer if necessary. Then maybe include an option for pointing the app to a folder on the device where you can store mapsforge maps like c:geo does? I admit, I don't know the coding issues involved, but I think providing choice might solve some of these issues.

[ENT8R]

Outdated maps but also bad user reviews because of map issues were the main reason for me...

Then maybe include an option for pointing the app to a folder on the device where you can store mapsforge

Don't think this will work, Mapsforge is a completely different rendering library...

There is another vector tile server maintained by terrestris

Do you know the url of the documentation? I couldn't find anything except an example map: https://ows.terrestris.de/osmvectortiles/index.html

EDIT: I also found this page: https://ows.terrestris.de/dienste.html#openstreetmap-wms but is this already the complete documentation?

[westnordost]

Alright, I didn't know the maps were that outdated. So let's put this into v7.

I will get into contact with the terrestris-person in either case. The documentation can be expected to be tiny/nonexistent because it is one of those services that I think are essentially only public/free because it is not advertised and thus not (mis)used commercially (that much), instead only in OSM context. Is the page you linked enough to go by or should I ask for more information?

Also, for those that complain that the map is not updated anymore, what about nextzen? Has it been updated since Dec 2017?

[matkoniecz]

From what I remember nextzen was supposed to be static and not updated (that is why I ignored it so far).

[rugk]

A static map??

[ENT8R]

Yes, Nextzen still uses the data from December 2017: https://tile.nextzen.org/preview.html

[exploide]

Maybe this all highly depends on the region?! Changes I made on July, 1st are visible on the currently used SC map. (And the tile.nextzen.org map contains changes I made 9 days ago.)

But several weeks / few months ago, I also noticed that everything is very outdated (missing buildings, etc).

[ENT8R]

Because maps are not updating since at least April

I checked it now on my device and the data is younger than 2 months (saw a fence I added on 22.06.) in Germany. Is the age of the map/tiles that different throughout the world?

[exploide]

Maybe because of https://github.com/westnordost/StreetComplete/issues/1011#issuecomment-394478833

I have deployed a test, now Germany should be a more recent (I think 23.4. catching up, and the old tiles will be cached between 7 and 14 Days), all other Data is more old.

[cstackpole]

I hope this is helpful/related information. It was on the OSM chat a while back. https://www.esri.com/arcgis-blog/products/arcgis-living-atlas/mapping/new-osm-vector-basemap/

Their contact is osm at esri.com .

Could this be used?

[westnordost]

I find esri embracing osm is as dubious as Microsoft embracing open source. Also, the mentioned tileserver is labelled as "beta release, meaning it’s available for you to use and test in your maps and apps but you shouldn’t use the map or related items in a production application".

[westnordost]

I haven't heard anything from @weskamm yet, the admin of the OpenStreetMap WMS at terrestris, so trying out/migrating to another vector tile server will probably not make it into the upcoming v7 (due this Sunday).

[westnordost]

I did get answer from him now:

The WMS is at https://ows.terrestris.de/osm-vectortiles/osm:osm_world_vector@mapbox-vector-tiles@pbf/{z}/{x}/{-y}.pbf.

There is no documentation, but from the source code of his demo application, one can see what layers there are etc. Regarding the attributes of map features (tunnel=yes, building height,...), he wrote that they are usually what Imposm3 maps in its standard configuration.

@ENT8R Are you interested in adapting the style to be compatible with this?

[ENT8R]

Yes, of course! I will look into it the following days (when the weather is bad again :smile:).

[westnordost]

Maybe I'd give it a try because I want this to be part of the next beta which I want to release on the weekend - unless you also intended to do it then.

[ENT8R]

I will try to do this today. Probably I will post the first result in the afternoon...