Mapzen shutdown

[ENT8R]

Today, Mapzen sadly announced to shutdown their services at the end of January (see more here: https://mapzen.com/blog/shutdown/) They suggest to migrate to other services here: https://mapzen.com/blog/migration/ This will also affect StreetComplete as the map tiles are received from the Vector Tiles Service of Mapzen (https://github.com/westnordost/StreetComplete/blob/master/app/src/main/assets/cinnabar-style-8.0.0.yaml#L242) which means that on February, 1st, there will be no map displayed anymore in StreetComplete... I think we should search for an alternative as soon as possible...

[rugk]

WTF, why is the next OSM service shutting down? I remember there was already a problem with one, which Gnome e.g. used.

They then switched to mapbox.com.

[matkoniecz]

WTF, why is the next OSM service shutting down?

I assume that answer is standard - money (funding run out before company become profitable).

[westnordost]

One month, this is a pretty short period. I am pretty sure that I/we will not find and set up something until then. But this has a high(est) priority now. Without a map view, the app is barely usable. At least it is time enough to add an info message before their server is shutdown that the map view does not display the map and why.

I guess it is not feasible to set up an own tileserver for StreetComplete. Are there any other options than MapBox? The MapBox tileserver is only allowed to be used through the MapBox SDK. I initially decided against MapBox because they have some telemetry built-in and their terms and conditions require to have this by default on (only letting the user switch it off in the settings).

It is really a pity that Mapzen is shutting down, I considered the company a healthy competition to Mapbox.

By the way, if Mapzen as a company is shutting down, then LOST will probably also not longer be maintained and an alternative needs to be searched sooner or later.

[matkoniecz]

Without a map view, the app is barely usable. At least it is time enough to add an info message before their server is shutdown that the map view does not display the map and why.

It is a hackish band-aid but it seems that satellite images may be used for transitory period (code is at https://github.com/ENT8R/streetcomplete-mapstyle/blob/master/satellite-style.yaml ).

then LOST will probably also not longer be maintained and an alternative needs to be searched sooner or later.

tangram-es is also likely to die.

https://github.com/tangrams/tangram-es is under https://github.com/tangrams with

An open-source project sponsored by Mapzen.

with website set to https://mapzen.com/tangram

[Akasch]

I can have a look into self hosting the tileserver for Streetcomplete. I already have found out how to do most of it, it seems not overly complicated. I currently see two Problems:

1. The Database needed seems to be in the 1-2 TB Storage Range.
2. I don't know what Traffic one would have to expect.

I think that it should be possible to reduce the traffic via caching and maybe more than one server serving the cached tiles, but the Database has to live somewhere.

If it is wanted and is not to much traffic (or we can distribute the caches) I can look into it if hosting a tileserver possible

[matkoniecz]

The Database needed seems to be in the 1-2 TB Storage Range.

Maybe it can be reduced by finishing https://github.com/ENT8R/streetcomplete-mapstyle and storing only limited data used by SC?

Dropping some less used regions (Africa?) is probably not going to help much as I expect that regions with large amount of data and active mappers are the same.

reduce the traffic via caching

StreetComplete is already caching map data

[westnordost]

Tangram(-ES) is a really neat library and it would be a big loss (to the openstreetmap ecosystem) if it would die like this. But without a tileserver it can use, it makes no sense. MapBox prohibits the use of their vector tileserver using another SDK than their own and I do not know any other.

I imagine that tangram-es as an active project could be saved if someone from within the community of openstreetmap or maybe the foundation itself would slide in and set up an own vector tile server analogous to the Mapnik bitmap tiles server. If traffic is a problem, it probably is, limit usage to openstreetmap related projects. This is no guarantee that the library is actually being maintained and developed further, but if the library is continued to being used by multiple projects, it's a good start.

Given that the announcement about the company shutdown came with such short notice, I would say that it was also a surprise to all the developers who are now being laid off and probably both quite pissed now and have other worries than what may happen to their old project from work, so I don't expect any of the current maintainers to stick to the project. This will make it hard to retain the project. On that occation, I want to thank all the developers from mapzen for their devotion and great work on tangram(-es) and related projects over the last 3 years, especially @matteblair, @tallytalwar, @hjanetzek and @karimnaaji - I hope you guys find something new smoothly :-)

Has this news already reached the (talk) mailing list(s)?

[matkoniecz]

I see no posts on dev and talk.

dev current state: https://lists.openstreetmap.org/pipermail/dev/2018-January/thread.html

talk: https://lists.openstreetmap.org/pipermail/talk/ (no posts in 2018)

[rugk]

I initially decided against MapBox because they have some telemetry built-in and their terms and conditions require to have this by default on (only letting the user switch it off in the settings).

Hmm… strange. GNOME does not have this. Maybe it is only in their Android SDK?

[matkoniecz]

On that occation, I want to thank all the developers from mapzen for their devotion and great work on tangram(-es) and related projects over the last 3 years, especially @matteblair, @tallytalwar, @hjanetzek and @karimnaaji - I hope you guys find something new smoothly :-)

Also thanks from me - tangram-es and vector tiles were very pleasant to use.

[tallytalwar]

Hi There

A few things right now, will follow up on more later.

1. tangram/tangram-es: We as developers do plan to work on the project but more on a part-time basis, so the development will be a bit slow. Sorry about this. The community is welcomed to take it forward :D.
2. LOST is evolved already: https://github.com/lostzen/lost. Again development will go on but part-time and the community is more than welcomed to contribute.
3. vectortiles are interesting, we might have more info on that soon. Still, some crunching going on. But a better solution will be if someone could replicate the documented services and carry forward the legacy. Most of this is opensource and documented and tilezen team members will be happy to help if any questions arise in the process. (Edit refer: https://github.com/westnordost/StreetComplete/issues/747#issuecomment-361726992 below)

Will follow up more probably on the readme pages of the project(s).

[matkoniecz]

work on the project but more on a part-time basis, so the development will be a bit slow. Sorry about this

You really have no reason whatsoever to be sorry for continuing to work on project in your free time. It is amazing and thanks for that! If any support/development continue it would be great, but obviously nobody should feel obligated to do that.

[sylvainar]

What about using tiles directly from OSM ? That's we were doing with WikiJourney and it works pretty well.

[tallytalwar]

Tangram-es supports raster tiles really well, so if you use raster tiles directly from osm or any other service it should work. Plus all those markers (we were working on a better interface here :() stuff will work out of the box since that's all added by the client code itself.

With respect to using other vector tiles, you will have to work on stylesheet supporting the specific properties in that tile format. Once anyone of us gets some free cycles, we can support some property mapping functionality directly on tangram-es which will help you guys use amazing mapzen styles.

[tilladams]

if wanted, you can use the OSM WMS basemap-services, that my company provide. They are free to use. Find details here: https://ows.terrestris.de/dienste.html#openstreetmap-wms (german only).

@sylvainar, @tallytalwar : Using tiles directly from OSM in your application is normally "tolerated", but not really what the OSM community likes you to do (officially it is even forbidden without the permission of the administrators). If usage exceeds a certain limit, people are asked to setup their own OSM server. See a brief discussion on this here: https://operations.osmfoundation.org/policies/tiles/

(I have some direct insight here, because I am board member of FOSSGIS e.V. (www.fossgis.de), which is also the local german chapter of OSMF and that pays for some of the OSM servers ;-))

[westnordost]

Thank you all for your input.

So, here is my battle plan:

I would like to stick with Tangram-ES, if possible. Since the future of Tangram-ES remains largely unclear, I will put as least effort into the map display as possible. Any features concerning the map display will be delayed indefinitely.

Thank you, @matkoniecz, for the remark about the satellite images. These can be used for transitory period to either another vector tile server or, if none can be found, to MapBox.

@Akasch thank you for your offer, it is absolutely wanted! I can see from the Mapzen statistics that StreetComplete users are pretty consistently making about 20k tile requests per day. The size of one tile varies from mean about 2-20KB in towns, depending on how well-mapped a place is (i.e. if buildings are present etc). If you take ~10KB as a global mean, then we are at ~200MB a day, or 6GB a month. That sounds OK, doesn't it? If this server is only going to be used for StreetComplete, it could be adapted to spit out less data - only the features that StreetComplete actually displays - to save traffic. Especially with @ENT8R's new simplified custom stylesheet for StreetComplete, much less is shown I think. Getting (access to) a mapzen-based vector tile service would be the best solution for StreetComplete as it is really the least trouble.

@tilladams also thank you for your offer with the OSM WMS basemap-service! I have to say that I know very little about all those map services, this is why I very much appreciate help there. (What's WMS?) Is this a vector tile service or something Mapnik-based? If it is not vector, I would only consider this as a temporary solution. On the other hand, if it is vector, the problem is that as far as I know, only the format in which vector tiles are served and how they are accessed is standardized, but what features are part of it and how the properties are named, is not standardized. This means, that a vector style sheet made for mapzen vector tiles will not be that compatible with i.e. mapbox vector tiles. @ENT8R's new simplified custom stylesheet for StreetComplete could help very much to alleviate the problem because it is simple and thus less likely to be incompatible / easier to adapt.

[matkoniecz]

What's WMS?

AFAIK it refers to https://en.wikipedia.org/wiki/Web_Map_Service "standard protocol for serving (over the Internet) georeferenced map images".

EDIT: wikipedia is misleading, it may be text (including vector tiles) or an image:

6.6 Output formats The response to a Web Map Service request is always a computer file that is transferred over the Internet from the server to the client. The file may contain text, or the file may represent a map image.

[westnordost]

It does not list MapBox vector tile service, is MapBox not a WMS?

[matkoniecz]

It does not list MapBox vector tile service, is MapBox not a WMS?

Is it about https://ows.terrestris.de/dienste.html#openstreetmap-wms or https://en.wikipedia.org/wiki/Web_Map_Service or something else?

is MapBox not a WMS?

From reading https://www.mapbox.com/help/mapbox-arcgis-qgis/ and looking at https://www.mapbox.com/api-documentation/#retrieve-raster-tiles-from-styles it seems that mapbox is using WMTS protocol ( https://en.wikipedia.org/wiki/Web_Map_Tile_Service ) for delivering its raster maps.

[nlehuby]

Have you considered OpenMapTiles to replace Mapzen Tile services ? Their vector tiles are compatible with Tangram (but the schema is different, so you will need to adapt your scene yaml): https://openmaptiles.org/docs/website/tangram/

[matkoniecz]

Have you considered OpenMapTiles to replace Mapzen Tile services ?

note

A free OpenStreetMap vector map tileserver is available for personal non-commercial use and demonstration purposes

though with "making about 20k tile requests per day" 40$/month is not terrible (https://www.tilehosting.com/#plans)

[Akasch]

I will test over the next days what I exactly need. I think the traffic is manageable.

At first I will setup a test server with only the data from Germany, to see how everything is working. From the what I have seen until yet I think I will change the URL format a little bit for a more efficient access control on my side. After that I will try to import the whole world. If this is working I will look into reducing the data size and doing more efficient caching. Especially in the beginning I will cache rendered tiles for relative long times, I think of around 12 hours at the moment. After I have seen the load it generates a reduction is hopefully possible.

The first step should be finished in two or three days, the whole world in the middle of the next week I think.

[westnordost]

That sounds odd, are you saying that the mapzen vector tile service has inefficent caching and/or puts unnecessary data into the reply? If this is the case, perhaps you should put your enhancements upstream into their repository.

[tilladams]

Hi, "WMS" is Web Map Service", which is a standarized protocol, that allows to request map-tiles from any client (see ->http://metaspatial.net/w/index.php/OGC_WMS for a simple explanation . But you tell it, a WMS only serves tiles, no vectors. In general the advantage of using tiles (.png- or .jpg-images) is, that images are normally smaller (less traffic!) than using the vectors. I am not sure, because I do not know your code, but from my usage of the App "StreetComplete" I see no interaction with the real vectors. I think that just as background service tiles might be fine. But if vectors work fine, of course it is an opportunity.

Just another idea (without really thinking about all the consequences): We at FOSSGIs donate money to open source projects and very often this is done by paying for server capacity. Might be an idea to think about setting up a project own vector tile server. For the technical stuff, I (or guys from my company) may help (for free of course). Contact me via email if this is an opportunity.

[Akasch]

@westnordost I thought about putting the api key in the URL path instant of a GET parameter, but after some more consideration I think both forms are more or less equal.

Once the index generation is finished I have all things ready for a test on the data from Germany. As the import on the hardware I use is relatively slow (no big SSDs), and it would be a secondary use of other resources. I think it would be grate to have a (more) dedicated machine for it if it is possible @tilladams .

[Akasch]

For Testing the vector tile service is available at http://test.osm.rokita.it/{tilesize}/{layers}/{z}/{x}/{y}.{format} . As the Polygon indices are not finished now it is really slow and Polygons are missing. I hope it will finish soon. Caching is not enabled, I will do it when the Polygons are ready. Only the Data of Germany is imported.

[Akasch]

It is fully working now as far as I can see (Still only Germany as data). The Import of changes do also work.

A Online Demo can be viewed at: http://test.osm.rokita.it It would be gread if somebody can test with Conversations as Client.

At the moment the tilegeneration is a little bit slow due to IO constrains and that only one Prozess is generating the tiles, I will tweak the settings and excecution tomorow.

At the moment the Tiles are Cached for six hours to reduce the load and speed up the user experience.

@westnordost After the tests I am confident that I could operate a vector tile service for StreetComplete if you want it. But I think that for a better User experience we would need a better Server than I have at moment, especially big SSDs for the Database. I think we should email @tilladams about it. Do you want do do it or should I? We also should plan the steps towards the transition.

[westnordost]

Very cool! For StreetComplete, speed is more important that the actuality of the map as it is just the background layer.

I'd switch out the tile server address if possible latest one week before the end of Mapzen services because F-Droid usually takes up to one week to serve the new version. I'd do this in the frame of a bugfix update of version 3 rather than a new feature release, as it will not be stable until then. Sure, your server is a bit slow, but even if we might settle on a dedicated server with SSDs, your effort greatly helps to relieve the urgency of the issue.

I try to focus the time I got (for OpenStreetMap) on StreetComplete, this is why I try to spend as little time as possible on the things around it. So, if it is going to be a dedicated server, then I very much prefer to not (having) to run and maintain it myself. But of course, if someone else runs it, it is also my concern how it will be financed.

So, with the urgency gone, how about we first just switch in your server for now and wait until next month. Then, we see how the server performs and can better estimate the requirements (-> costs) of a dedicated server. If by then, other replacement services like i.e. what Steve Coast mentioned, became live, we can compare these and see about the next steps.

[Akasch]

Ok I will begin with the Import of the whole world. I can register a domain and set it up as a proxy to mapzen, so the domain in the app can be changed before the import is finished. When the Import is finished we can make the switch simply in the server configuration.

[westnordost]

As you like. If we use it from the start, we get to see the load quicker.

Am 12. Januar 2018 00:03:20 MEZ schrieb Nils Rokita notifications@github.com:

Ok I will begin with the Import of the whole world. I can register a domain and set it up as a proxy to mapzen, so the domain in the app can be changed before the import is finished. When the Import is finished we can make the switch simply in the server configuration.

-- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: https://github.com/westnordost/StreetComplete/issues/747#issuecomment-357090323

-- Diese Nachricht wurde von meinem Android-Mobiltelefon mit K-9 Mail gesendet.

[Akasch]

The Service will be available at https://tiles.map-data.de/vector/v1/{layers}/{z}/{x}/{y}.{format}?api_key={api-key} at the moment it is a Proxy for the Mapzen service at https://tile.mapzen.com/mapzen/... I will switch it over to self generated tiles when the import of the osm data is finished. As it it a Proxy to Mapzen at the moment a Mapzen Api Key is needed.

[westnordost]

Cool! You mean https://tiles.map-data.de/vector/v1/512/all/{z}/{x}/{y}.mvt, right? (Note the "512", this is part of the normal Mapzen url)

Also, is plain http also available? There is a problem with https on Android <5.0 devices. This app supports Android versions down to 4.2 at the moment. See #175

[Akasch]

As far as I have seen the 512 is optional in the Mapzen url, it should work both ways.

At the moment http is forwarding to https. Generally I would like to leave it this way, I will have a look at the bug reported there.

[rugk]

Mhh also it seems you used HTTP for all Android versions, did not you? That is not good, as I already said, – if you really need it – only disable HTTPS for the old (broken) ones.

[westnordost]

Yes because I did not care to somehow implement a switch there. OpenStreetMap data is not exactly sensitive or personal information. If anything is personal there, it is the location of the request (zoomlevel, tile x,y) as it maybe would give the user's location away. But that information is in the request URL, and thus also visible independent of which protocol is used.

[westnordost]

Actually, @Akasch nevermind the SSL issue. Apparently it was a problem with the specific root certificate used by Mapzen was not trusted in some older versions of Android. You are using LE, the same root authority as openstreetmap.org. So if LE is not trusted on some device, then, nothing would work within the app.

[HolgerJeromin]

But that information is in the request URL, and thus also visible independent of which protocol is used.

Https sends only the host not the full request unencrypted

[westnordost]

Oh, then I was misinformed.

[rugk]

Oh my… yes the URL is encrypted, only subdomain is visible. And it is exactly the location information, whcih is the problematic metadata here. And AFAIK these are not vector titles, or are they? (Just because with vector tiles an attacker may also guess the tigtle and thus the location based on the [different] sizes, but this is another story and we cannot do anything about that anyway.) So just use HTTPS… as always. Just do.

[Akasch]

The import is finally finished. After some tests, the tiles are generated mostly fast enough for StreetComplete in the meaning it is not really slower than before. Only some middle zoom tiles in citys are a slow. I render them this afternoon, and it should get better with caching over time.

After the rendering of higher zoom levels I will switch over tiles.map-data.de from a Mapzen Proxy to the self generated tiles.

[Mannshoch]

To reduce Data costs you maybe could add support for the offline tils from Osmand. (Or create an StreetComplete-Addon for Osmand)

My offline cache for Osmand is located on external sdcard.

[ENT8R]

To reduce Data costs you maybe could add support for the offline tils from Osmand. (Or create an StreetComplete-Addon for Osmand)

Tracked with #243 and #122

[rugk]

BTW before you actually switch the proxy of, can you already share some stats on the load you observe?

[westnordost]

Cool!

Definitely, we will want to look at the stats, so that we can see if that server will be enough or if we need to look for a dedicated solution - and in any case we can estimate the running costs it will generate and look for how it should be financed.

I am living close to the IKUM by the way.

[Akasch]

Yesterday I had 25551 requests, not counting cached responses (Cache duration is at 6 hours at the moment). The Database is over 1 TB at the moment I use two 512 GB SSDs in RAID 0 and one 512 HDD. The import is realistically only possible with SSD backed storage as it is really slow over wise. The pre rendering of the tiles running at the moment is slowing down the live rendering for requests, as the Database is IO waiting for the HDD as far as I can see. Generally most tasks are more bound by IO capacity than CPU Power from that I have seen so far.

[westnordost]

StreetComplete also caches, but as the map server URL changed, I guess the cache is rebuilt slowly for all the different users now.

[hjanetzek]

Hi Tobias - sorry for joining late.. I've just found that mapfit.com provides tiles with the tilezen schema. So you could ask your users to get an api key there for a fast tiles-server and to save you some server-load during the transition - though I would not recommend mapfit further - as they have zero OSM attribution... Great work with the tile-server, btw! openmaptiles would also be an option. I've started to convert some of their styles here: https://github.com/hjanetzek/tangram-scenes. (Also I think openmaptiles have a more lightweight tile generation and smaller tiles, i.e. containing less data sufficient for a base map)

[Akasch]

I have finally switched the tile Server from proxy to locally generated tiles. If there are Problems please report them. The delay was mostly to the need to shuffle the database tables around on the Storage so that the tables with most reads are on the SSDs as the whole database is not fitting on the SSDs I currently have.

I will optimize the Operations in the next hours/days and post some statistics.

[ENT8R]

openmaptiles would also be an option

I don't think so... It was already discussed earlier in this issue (https://github.com/westnordost/StreetComplete/issues/747#issuecomment-355846884) that this service won't be an option...

[hjanetzek]

one could use the openmaptiles tile-server, not necessarily use their service. The implementation is likely way more lightweight on the server resources due to using imposm3 vs osm2pgsql and rust instead of python for the tile generation. This would be my technical recommendation after having managed our universities vector tile server with osm2pgsql for almost five years. (imposm3 was ready to use when I left university and I only kept it running the last two years)