Maps no longer updating on new vector tile server, out-of-date with quests

matkoniecz commented 6 years ago

Yes, #835 is back and #747 continues to cause problems.

Firstly, thanks to @Akasch for his great work in setting up a new vector tile server for rescue StreetComplete from the demise of Mapzen.

I am opening this issue to mention that as the server is not updating at the moment (and I'm sure it is probably not trivial to do so easily or cheaply), and therefore the map is no longer up-to-date with the OSM data (and therefore StreetComplete quests).

Is it possible to restart updates or run cyclic updates (once a month? once 3 months?) or switch to an alternative background map to StreetComplete that pulls direct from osm.org rendered tiles?

Currently it is not fatal problem in areas that are not significantly changed but it will steadily get worse if nothing is changed.

westnordost commented 6 years ago

Cool!

Am 24. August 2018 08:04:46 MESZ schrieb ENT8R notifications@github.com:

I will try to do this today. Probably I will post the first result in the afternoon...

ENT8R commented 6 years ago

So, the first version is finished and can be found at https://ent8r.github.io/streetcomplete-mapstyle/ if you click at the layer button in the top right corner and select Light Style (Terrestris). The source can be found at https://github.com/ENT8R/streetcomplete-mapstyle/tree/terrestris During converting the style I found some issues regarding the functions of the tile server:

the coast lines effect does not work (maybe we have to remove the effect 😕) (code)
elements specified as stream, ditch, drain are not shown (code)
the tiles contain no information about the height of a building so a 3D feature is basically useless and this is also the reason why I set extrude to false in the code
barriers like walls and fences are not shown too even though the tiles should contain that information (code)
it seems like the tiles contain no information about trams, so this can't be displayed too (code)
steps are also not displayed at all even though they should be contained in the tiles (code)
house numbers are also not shown and I could not figure out yet whether the tiles contain that information or there is an issue in the style...
swimming pools are also not shown though this should be covered by the waterareas layer...

westnordost commented 6 years ago

Cool, thank you!

Hmm, a lot of problems, doesn't sound so good. How did you find out that certain information do not exist in the layers? Maybe they are just named differently?

Am 24. August 2018 15:45:54 MESZ schrieb ENT8R notifications@github.com:

So, the first version is finished and can be found at https://ent8r.github.io/streetcomplete-mapstyle/ if you click at the layer button in the top right corner and select Light Style (Terrestris). The source can be found at https://github.com/ENT8R/streetcomplete-mapstyle/tree/terrestris> During converting the style I found some issues regarding the functions of the tile server:>

the coast lines effect does not work (maybe we have to remove the effect 😕) (code)>

elements specified as stream, ditch, drain are not shown (code)>

the tiles contain no information about the height of a building so a 3D feature is basically useless and this is also the reason why I set extrude to false in the code>

barriers like walls and fences are not shown too even though the tiles should contain that information (code)>

it seems like the tiles contain no information about trams, so this can't be displayed too (code)>

steps are also not displayed at all even though they should be contained in the tiles (code)>

house numbers are also not shown and I could not figure out yet whether the tiles contain that information or there is an issue in the style...>

swimming pools are also not shown though this should be covered by the waterareas layer...

ENT8R commented 6 years ago

How did you find out that certain information do not exist in the layers? Maybe they are just named differently?

I looked at the source code of the demo website, the default config of imposm3 and mainly what is included in the vector tiles (the source). For example I've never seen something which says e.g. housenumber in the source of the vector tile...

westnordost commented 6 years ago

But how do you see what is included in the vector tiles? MVT/pbf is binary after all.

westnordost commented 6 years ago

Some notes:

trams, steps, swimming pools and aeroways also do not show on the terrestris demo map, looks like this information is missing. Except for steps, this is not so important though.
that the building height information is approximated from building:levels and delivered in vector tiles is actually a feature of the mapzen vector tile server, IIRC. Without this, this approximation would need to be made client-side. Though, the tiles do neither seem to contain height nor building:levels. This is too bad, after all, half the fun with vector tiles are the 3D-features.
regarding barriers, the source of the terrestris demo map does not even mention the layer osm_barrierways, perhaps it is missing altogether
the same with housenumbers, , the source of the demo map does not mention osm_housenumbers, perhaps the layer is not available
streams do show for me. For example here

New:

the roads of different classification overlap pretty ugly. I think this is because it is a Mapzen tile server feature that it delivers a z_order with every element, thus defines a "default" sort order of the elements. When this is missing, the order needs to be defined client-side (in the style). So, this is something we can fix.
there seems to be a problem with coastlines. See Manhattan (this looks quite bad), or a small part of Bremerhaven
the server seems to have problems delivering vector tiles with practically 0 users on it. Just zoomed in randomly to a city and tiles on higher zoom levels just won't load (also in their demo map not). (Tangram is much less forgiving when tiles are missing - the area will just be left blank. OpenLayers instead shows the tile from a lower zoom level blurred)

westnordost commented 6 years ago

I also sent @Akasch and @weskamm an email explaining that I want to try out the tileserver in the next beta. Though, perhaps this is a bit on a too short notice and I should wait until mid of next week, especially in light of the problems with the integration we have.

Most importantly, I don't know if this is just a hiccup right now, but the last point would be a dealbreaker. It wouldn't be usable like this. (Perhaps it is the weekly import/cache flush?)

ENT8R commented 6 years ago

streams do show for me

true, streams are shown but ditches not. (but they are not that important anyway...)

the server seems to have problems delivering vector tiles with practically 0 users on it

I also thought that it takes sometimes pretty long to load the tiles... (right now I can only zoom in until zoom level 8 to see the USA...) all other requests are failing with 400: Problem communicating with GeoServer

But how do you see what is included in the vector tiles? MVT/pbf is binary after all.

I couldn't see every detail but I made assumptions 😄...

westnordost commented 6 years ago

Oh, okay :-D

So there is a z_order, then the mentioned problem seems to stem from somewhere else.

Edit: Anyway, no sense in putting any more time in it until this 400-error is resolved.

iandees commented 6 years ago

Hi all. An update on Nextzen: we're actively working on speeding up the time it takes to build a tileset of the world so that we can get to the point of weekly updates. We're currently doing this privately but I'll see if we can kick off a build to get the Nextzen tileset more up-to-date.

rugk commented 6 years ago

BTW another maps service/tile API: https://www.mapcat.com/

westnordost commented 6 years ago

@iandees that sounds too good to be true

A short update here:

We planned to move to the terrestris tileserver. We had some problems with changing the stylesheet to be compatible with it. Also, it turns out that the terrestris tileserver is not usable for about a whole day and night each week, so this server would only be only eligible as-is in an emergency, i.e. if our current server would become inaccessible on a very short notice. But thank you, @ENT8R, for your effort to create the stylesheet so far!
The reason why our current tileserver has old tiles in many places is because it stopped updating correctly due to the database being in a partly corrupted state. To solve this problem, @Akasch is coordinating a complete reimport of the map which takes a few days up to a week, during which the tileserver is offline. To fill the gap, tiles from @iandees' nextzen tile service are served in the meantime (which are old, from Dec '17) As soon as the complete reimport finished successfully, the map is all-new again and the (bi?-)weekly update of the map should work again without problem. Disclaimer: Of course, as the currently used tileserver is a bit of a shoestring-server and only set up as an intermediate solution anyway, there is no guarantee that the problems we had with the tileserver will not repeat itself or even that the service is cancelled on a relatively short notice.

daviewales commented 5 years ago

Is there any chance we could add satellite imagery as an alternative?

I'm trying to add house numbers, but sometimes feel that satellite imagery in StreetComplete would give me a bit more confidence that I'm adding the right number to the right house. (This would also work as an interim solution to the tile-server not being updated.)

arbie65 commented 5 years ago

Thank you everyone for all the work you are doing to fix this issue. I am asking I see very out of date tiles is it safe to keep using streetcomplete or is it possible that using it with old tiles can cause conflicts on OSM?

ENT8R commented 5 years ago

is it safe to keep using streetcomplete or is it possible that using it with old tiles can cause conflicts on OSM?

It is still safe to use StreetComplete. The map tiles have no effect of the created quests. In case you want to have some more information: the quests are downloaded with the Overpass API (which is usually up-to-date with an approximately delay of 3-5 minutes...)

arbie65 commented 5 years ago

Is there anything we as non technical users of streetcomplete can do to help get this ticket closed? It's a great tool but when features are in sync with OSM as it looks daily instead of a snapshot of months ago

westnordost commented 5 years ago

Not really. A more reliable and current map would require using a commercial vector tile service like Thunderforest, Omniscale, OpenMapTiles or what other providers there are. One could tentatively ask them if they would be willing to sponsor this app by providing the service for free to StreetComplete but this is probably something that I should do myself.

mattwelke commented 5 years ago

@westnordost Have you considered hosting a tile server in the cloud and reaching out to an organization like CloudFront to see if they'd sponsor free CDN service wrapped around your tile server to minimize load on it? DigitalOcean in particular often sponsors nonprofit organizations (https://www.digitalocean.com/community/questions/discounts-or-support-for-nonprofit-organisation), which could provide the origin server.

westnordost commented 5 years ago

No, because I do not want to concern myself with administrating a live tile server, rather I spend my time on developing the app.

iandees commented 5 years ago

FYI Nextzen just finished updating the data behind our tiles. I think the date on them is December 5th or so.

westnordost commented 5 years ago

Cool! So, how long did it take? Is this now a regular update or did you manually trigger it?

iandees commented 5 years ago

how long did it take?

It takes approximately a week to rebuild the tiles.

Is this now a regular update or did you manually trigger it?

We still have to manually trigger it. We're working on funding right now, but we might be able to get to quarterly or monthly update cycles.

Susurrus commented 5 years ago

What's the process for requesting or helping with updating these tiles again? I've been making substantial edits in my area, and the quests are updating, but the map tiles are pretty out of date at this point.

Akasch commented 5 years ago

At the moment unfortunately I have not so much time. But I now have a collection of scripts to setup a tileserver serving a limited region. Together with some server components it is now possible to distribute the tileserver.

I will cleanup the scripts in the next days. Maybe put some components into a doker container for easier deployment.

A server will need something like 50GB harddisk, 2-4 CPU cores, 4 GB RAM, I think. With CPU and RAM I am not so sure at the moment. Required software is Python-virtualenv (Python2), bash and postgresql with hstor and gis extensions.

The 50 GB is for the bigger regions, there are multible smaller ones, the computing recurces are only needed at import time and for very short times while generating some tiles. So it should be possible to have two or more regions on a server if the harddisk is big enough.

At the moment I am testing it with the region Hamburg in Germany.

If there is someone who want to try hosting a region, write me, and we can try out the progress.

Julientroploin commented 5 years ago

New user of StreetComplete, I think I experience the bug described here, so I can not contribute to OSM with your app... An image of what I get while zooming : https://drive.google.com/file/d/1rArOT_YmfNMIU23lAiQPcCV5iHcZtBUo/view?usp=sharing Reading a part of the discussion (and not being an english native-speacker), I think I understood that your stuck with a server problem.

I can not help but I wanted to thank all of You for your efforts.

Akasch commented 5 years ago

@Julientroploin can you give me the location of the error, then I can try to debug the error.

Julientroploin commented 5 years ago

Thank you for your fast answer, this affect a lots of tiles near my position (La rivière-de-corps, FRANCE 48° 17′ 19″ N, 4° 01′ 14″ E ) Hope you can help to debug this!

Akasch commented 5 years ago

Ok i can see the error, but I will need 2 or 3 days to fix it.

Julientroploin commented 5 years ago

Thank you @Akasch , I'll check this in a few days.

Julientroploin commented 5 years ago

@Akasch, It work now... I suppose you fix the problem, that's great ! Right in time to contribute under a fantastic spring sun !

Akasch commented 5 years ago

@Julientroploin yes, it was faster than i thought.

NonnEmilia commented 5 years ago

@Akasch also in my zone in Italy we have an old map not showing some recent (1 or 2 months old) roads. Could you kindly take a look? Coordinates: Lat 44.76617 Lon 10.66540

Akasch commented 5 years ago

The scripts to host a regional part of the tileserver are finally mostly usable. So if you want your region to be updated you can contribute to it. The scripts can be found at: https://github.com/Map-Data/regiontileserver . You will need a Server with ~60 GB of free space, best on an SSD as the import will be faster (Hours instant of Days). Before starting the import please write me a short email so that we do not have unnecessary duplicates.

If you have questions ask me. If you do not want do do the administration yourself but sponsor a region I can do the administration on a provided server/virtual machine, contact me to discuss details.

A visualization of the hosted regions is available at https://map-data.de/#map=3/0/0&style=cinnabar&show_server . The red regions are served by http://nextzen.org with older data. The green ones are currently served by us, with data from the last few weeks. The yellow ones are normally served by us but due to maintenance currently not available so they fall back to nextzen.

westnordost commented 4 years ago

We have another ticket open for this, so I'll close this old one

Kezxo commented 4 years ago

We have another ticket open for this, so I'll close this old one

What ticket is that? I can't find any open ones linked here.

EDIT: is it #1700 ? It probably is.

rugk commented 4 years ago

Duplicate of https://github.com/westnordost/StreetComplete/issues/1700

streetcomplete / StreetComplete

Maps no longer updating on new vector tile server, out-of-date with quests #1011

Some notes: