Vector Tiles on osm.org

[bhousel]

From @bhousel on May 21, 2018 19:49

I've been reviewing the Top 10 Tasks list, and noticed that several of the issues reference vector tiles as a possible way forward.

• Localized Map Rendering - https://github.com/osmlab/osm-planning/issues/9
• Clickable POIs - https://github.com/osmlab/osm-planning/issues/16

This is also something that pops up for discussion sometimes here in this repo (#1672 #1623 #1316)

Let's start a discussion about what it would look like to add a vector tile option to openstreetmap.org! Even if it's just a trial or staged rollout available only in some areas, I think it would be great to explore vector tiles as a path forward on some of these issues.

I think we all agree that if something is on the Top 10 Tasks list, it's important to enough to people that it's worth doing. It's time! ✨

Copied from original issue: openstreetmap/openstreetmap-website#1875

[bhousel]

From @tomhughes on May 21, 2018 20:3

This isn't (oddly enough) a web site issue, it's an operations issue.

Yet another discussion really isn't going to get us anywhere though - we need somebody to maintain a stylesheet and somebody with relevant expertise to tell me what toolchain to setup. The two are of course likely linked.

[tomhughes]

To explain some more the first step is to switch the default layer to vector tiles - we absolutely want to do that.

Doing that as I say means making a choice of what toolchain to use, and writing any missing bits though I think that is less likely to be needed these days. Then I can write chef recipes to deploy that toolchain.

Obviously we also a suitable stylesheet.

To be honest although I said this is an operations issue, it doesn't really become that until we have somebody with a plan - then making that plan happen becomes an operations issue and/or a chef issue.

Making a plan is what the top ten repo is for right?

[bhousel]

Doing that as I say means making a choice of what toolchain to use, and writing any missing bits though I think that is less likely to be needed these days. Then I can write chef recipes to deploy that toolchain. Obviously we also a suitable stylesheet.

Yes - we discussed a bunch today and it sounds like there are several good options of vector tile stacks to choose from.. Let's use this ticket to compare options and make a decision.

Making a plan is what the top ten repo is for right?

The Top 10 Tasks can definitely steer us in the right direction. But I wouldn't want people to start working on one of these tasks (like Clickable POIs or Multilingual Maps) and get stuck or frustrated.

I'm definitely not the person to evaluate the options, but lets tag anyone with experience in this. (off the top of my head, @gravitystorm @pnorman @systemed )

[tomhughes]

Well I suspect this is a big enough project that it will need a repo of it's own where a solution can be developed.

I suspect @gravitystorm will want to stay out of this due to his conflicts of interest. We've mostly been pinning our hopes on @pnorman for this one ;-)

[mmd-osm]

Lots of discussion on this topic already going on here: https://github.com/gravitystorm/openstreetmap-carto/issues/3201

[pnorman]

2/4 of the layers on osm.org are vector tile based - both OpenCycleMap and Transport are, like all of @gravitystorm's stuff is.

There's been some very preliminary discussion at the WMF about asking adding the WMF map style as another featured tile layer, and it's also vector tile based.

@rory has done some work on porting osm-carto to vector tiles, which has the advantage of being able to easily render similar maps without big performance impacts, and the disadvantage of a much more difficult to understand stack, more components, and worse labeling. The only advantages to it for osm.org would be multiple resolution support and lower resource usage for print.

This doesn't become an operations issue until one of two things happens

1. Someone asks the OSMF to host a style. For this to happen, that style needs to exist, and the software stack associated with it needs to be mature and sanely deployable. My professional opinion is that nothing is there yet. Auto-correct suggests deplorable, which can be the state of putting some of these in to production.

2. Someone hosting a layer asks for it to be included as a new featured tile layer. This doesn't require any OSMF resources.

My work on a new style using newer technology is bolder which is a client-side style using tegola for vector tile generation and tangram for client-side rendering.

[bhousel]

Nice @pnorman - bolder looks a lot like what we're aiming for, and that's great that Andy's thunderforest styles seem ready to go today.

Do you think we could set up a vector-tile based demonstration site? Even it's something like beta.openstreetmap.org and only renders part of the world, but it would be a place to demonstrate richer POIs and localization that people are looking for.

We'd need to swap in a client side library that replaces Leaflet, and we could use it to render the existing vector tiles from yours / andy's / WMF's styles? Apologies if I am misunderstanding the steps needed..

[tomhughes]

I think you're completely misunderstanding ;-)

Any solution that allows those kind of additions to be added to osm.org needs to be the default style and almost certainly needs to support server side rendering as well as client side.

We're a long, long way from being able to add those kind of features to osm.org and I really think you're trying to sprint when we're not even crawling yet.

Paul's proposal is certainly one way forward, but it's an extra on top of what I would like to see happening, which is a plan for moving the default OSMF hosted layer to vector tiles.

[zerebubuth]

Any solution that allows those kind of additions to be added to osm.org needs to be the default style

Would it be possible to add a client-rendered (as opposed to server-rendered) vector tile layer as a non-default layer similar to the cycle, transport or humanitarian layers? Perhaps something based on OpenMapTiles or Nextzen? (Although the latter doesn't currently meet the data freshness requirement, it could perhaps be used during style development.)

Having it as a non-default layer would allow the style to be worked on in parallel with any work around hosting the tiles or implementing server-side raster fall-back rendering.

[tomhughes]

Is that possible using leaflet? I got the impression from @bhousel that client side rendering would require use of a completely different toolkit?

My other question would be what proportion of our users such a layer would work for. I don't really have any idea of how things currently as far as client support for the necessary technology goes across the range of browsers and platforms our users use?

[zerebubuth]

Is that possible using leaflet?

I don't know about other toolkits, but Tangram integrates with Leaflet by exposing a LeafletLayer which can be used like this:

var map = L.map('map');
var layer = Tangram.leafletLayer({ scene: 'scene.yaml' });
layer.addTo(map);

I'm pretty sure that it doesn't preclude using other (raster or otherwise) layers, although there might be some lag between the layers when dragging around (i.e: markers).

what proportion of our users such a layer would work for

Tangram should work on any browser with WebGL support, and globally that's currently around 93% of users. To begin with, I think the new client-vector layer would be non-default, and would be disabled where WebGL support wasn't available.

Perhaps, before the client-vector layer could be made the default, we'd want to have a raster fall-back rendering server for those 7% (or perhaps even fewer, by the time we make the switch). However, I don't think the lack of a fall-back at the moment should prevent us from adding a non-default client-vector layer, and allowing work to be started on the style for that. We might end up using something completely different, but this looks like something we could make a start with.

[pnorman]

We'd need to swap in a client side library that replaces Leaflet, and we could use it to render the existing vector tiles from yours / andy's / WMF's styles? Apologies if I am misunderstanding the steps needed..

Vector tiles require no client-side changes. You can't tell that Andy's styles are vector tile based, nor the WMF ones, unless you know exactly what to look for.

Nice @pnorman - bolder looks a lot like what we're aiming for, and that's great that Andy's thunderforest styles seem ready to go today.

They're not just ready to go today, on osm.org they are vector tile powered, and have been for a few years.

---

Would it be possible to add a client-rendered (as opposed to server-rendered) vector tile layer as a non-default layer similar to the cycle, transport or humanitarian layers? Perhaps something based on OpenMapTiles or Nextzen?

I had urged Mapzen to suggest this for their Cinnibar style, but it never happened for an assortment of reasons, and is now a moot point.

Tangram should work on any browser with WebGL support, and globally that's currently around 93% of users.

Yep, Tangram or Mapbox GL would work for most website visitors, but many tile like iD, JOSM, QGIS, and others users need raster images, not a client-side style.

Having it as a non-default layer would allow the style to be worked on in parallel with any work around hosting the tiles or implementing server-side raster fall-back rendering.

Yes. Adding it first as an additional featured layer is the only sane way I see this happening.

---

For the two issues linked above, they could both be implemented with either vector tiles, client side rendering, or client side rendering of vector tiles. If you put client-side rendering as a pre-requisite to the two tasks linked above, then you're putting a non-software development task as a blocker for two software development tasks.

[mboeringa]

I think this is something you all need to look at:

https://www.esri.com/arcgis-blog/products/arcgis-living-atlas/mapping/new-osm-vector-basemap/

I guess this may be based of Rory's work and fork of openstreetmap-carto, or at least heavily inspired by it... I knew this was possibly coming because I talked to one of the people involved at NACIS 2017, just didn't know the current status and that they were releasing it now just for the user conference.

Do try it out on a mobile phone as well, to really appreciate the vector tile's benefits.

[kocio-pl]

Thanks for the info. Wow, on my laptop it's working really fast, much better than I would expect!

[mboeringa]

Thanks for the info. Wow, on my laptop it's working really fast, much better than I would expect!

Yes, my experience as well. And it appears client side rendering.

[imagico]

Somewhat off-topic here - this would probably not even be suitable as an additional layer for osm.org let alone as something to be hosted on OSMF infrastructure.

But it is a nice demonstration of consequent use of aggressive way_area filtering. And this also means it is the exact opposite of Rory's work which was meant to port osm-carto to a different rendering platform without changing the rendering results in a significant way.

Trying to move the discussion into a more on-topic direction - as i mentioned in

https://lists.openstreetmap.org/pipermail/talk/2018-June/080902.html

it might be a good idea to discuss and formulate a few things that are considered to be requirements for a map being rendered on OSMF infrastructure. And with that i don't mean the 'wishlist' kind of things this issue was opened for, i mean the base requirements. This issue tracker is of course not necessarily the best place to have such a discussion.

Such a document would give developers of projects that might in the future at some point want to qualify for a function on OSMF infrastructure an important guideline. Right now most developers approach things purely from a technical side and requirements regarding cartography or usability/usefulness are often - if at all - only considered as an afterthought which in many cases leads to fairly bad results.

[mboeringa]

But it is a nice demonstration of consequent use of aggressive way_area filtering. And this also means it is the exact opposite of Rory's work which was meant to port osm-carto to a different rendering platform without changing the rendering results in a significant way.

I guess that is the almost unavoidably compromise needed to have client side vector tiles work at medium to low zoom for a dataset like OSM, so if you don't rasterize them and output as raster tiles. Moving potentially millions of landuse and water features to a client at e.g. Z8, even if heavily generalized, is probably unfeasible. At those scales raster tiles excel, as you only need to send a few raster tiles.

I also realised the website's opening remark states the use of ArcGIS Pro. I intially thought this might be some (direct) conversion to JSON of the existing openstreetmap-carto style, and ESRI using their own JSON style editor to potentially fine tune and modify the result. But the use of ArcGIS Pro may actually mean this result was achieved in a way much closer to ESRI's own raster tile basemap services, with styling in ArcGIS and use of ArcGIS layer files, and then output a map document to vector tiles from within ArcGIS.

When I talked to someone from ESRI at NACIS last year, we didn't go into the details of this.

[imagico]

I guess that is the almost unavoidably compromise

No, it is not (neither unavoidable nor a compromise). But before i get into a rant - this is now definitely off-topic here so i will leave it at that.

[bhousel]

To return back to the topic - what would it take to make this the default style on openstreetmap.org? I think the Esri team has done a great job!

[mmd-osm]

what would it take to make this the default style on openstreetmap.org

Isn't this all an ESRI proprietary, closed source rendering stack? If so, there basically no way to run this on osmf hardware I'd say.

Do try it out on a mobile phone as well, to really appreciate the vector tile's benefits.

Unfortuantely, this consistently crashes on my phone after a few seconds, tried Safari and Chrome.

[bhousel]

Isn't this all an ESRI proprietary, closed source rendering stack? If so, there basically no way to run this on osmf hardware I'd say.

Ok, so then - Esri open sourcing the tools and style is what it would take? I mean, that's a good place to at least start a conversation.

[kocio-pl]

It wouldn't have to be default nor rendered on OSMF hardware, it could be just another layer just like Thunderforest ones (both using proprietary stack outside OSMF servers). Definitely worth talking with ESRI to know what's possible.

[tomhughes]

I don't think we're looking to just copy something, and a pure client side rendering is unlikely to be usable as the default style anyway. I imagine we would want to start with server side rendering of vector tiles and then experiment with moving the rendering to the client where possible.

But, to repeat myself for like the one billionth time, somebody needs to take on the job of evaluating the various available stacks and making a recommendation of what we should and what additional components (if any) need to be developed to glue them together.

Plus of course somebody needs to define both the extraction rules (ie what data to put in the vector tiles) and the rendering rules to be used with whatever stack is chosen.

[tomhughes]

@kocio-pl well we are not setup at all for client side rendering at the moment so it would be a lot more complicated then the normal addition of a new tile layer.

[kocio-pl]

I'm pretty sure that anything related to vector tiles will be non-standard, so I expect problems no matter which way we choose. I just think it's important to try, because it's the future of online maps.

[pnorman]

To return back to the topic - what would it take to make this the default style on openstreetmap.org? I think the Esri team has done a great job!

Their layer? The biggest is they'd need to have minutely updates, not every few weeks. The default layer needs fast updates because it's part of the feedback loop. For the default I'd also want a style with an established record of responding to user enhancement requests. I don't see it happening because they're aiming at a market where those aren't concerns. Going to minutely updates would probably double their hardware cost, and I can't even find a repo with their style in an editable form.

[kocio-pl]

I intially thought this might be some (direct) conversion to JSON of the existing openstreetmap-carto style, and ESRI using their own JSON style editor to potentially fine tune and modify the result.

I try to get in touch with them to talk more (@mboeringa maybe you have direct contact to them?), but found this link to the repo and it looks to me like quite different implementation of OSM Carto look and feel:

https://www.arcgis.com/sharing/rest/content/items/3e1a00aeae81496587988075fe529f71/resources/styles/root.json?f=pjson

[slibby]

I'm happy to help answer some questions from the Esri side of things and I'm glad to see the interest in the basemap work that our team has done. There are several parts to the "stack" that are worth referencing:

tl;dr: OSM minutely changesets -> PostGIS -> ArcGIS Pro -> Vector Tiles (nearly-mapbox-spec).

• Where the changesets come from and how often they're applied
• Where the data is stored in "current" form
• How the vector tiles are designed and styled
• How the vector tiles themselves are generated and then hosted

For the first half of that list, we're using a very common approach - changesets direct from OSM, minutely diffs (I believe) - these are then passed in as updates into a PostGIS database of the Planet, created and updated using a variety of commonly-used OSM-built tools. I don't know the details 100% but I believe one of my colleagues (@skmoore) is presenting on it at SOTM US in October.

Now that we have a full, minutely-updated dataset in PostGIS, the cartography was designed in ArcGIS Pro, and vector tiles were created using ArcGIS Pro. The cartographic design ends up in a style .json file that you can find linked just above this comment, and ArcGIS Pro generates a vector tile package of the tiles for the world. At this point is where the minutely updates are replaced by "less often" updates - with the goal, as our blog post said, of increasing that frequency. I will try to get my other colleagues to join in on this thread, or perhaps better -- publish a more detailed technical overview on a Github repository.

cc @jgravois @ajturner

[systemed]

We had a really productive (and very well-attended) Birds of a Feather at State of the Map on Sunday 29th, on the subject of getting vector tiles on osm.org. Huge thanks to everyone who attended - it was great to see so much enthusiasm for it.

(Some prior reading before the session: http://blog.systemed.net/post/15, https://www.openstreetmap.org/user/Zverik/diary/44506, https://twitter.com/richardf/status/1023553727389683712 .)

A few takeaways from the session:

• Universal agreement that we want vector tiles on osm.org, sooner rather than later!
• We are not expecting to replace openstreetmap-carto, at least in the medium term. osm-carto has a strong community and its own particular strengths that are not easily replicated via vector tiles. Vector tiles and (raster) osm-carto would run in parallel, much as iD and P2 did/do. That means we can iterate the schema, styles and code rather than expecting v1 to equal osm-carto.
• We are not intending to serve raster tiles from the vector stack, but solely deliver vector tiles to the client
• Aim to serve as many OSM geometries/tags as possible within the confines of sensibly-sized tiles
• Consider also serving an alternative 'lighter' schema for bandwidth-constrained scenarios, maybe as a later phase of development
• Those with experience in the field (e.g. Paul and Kate) recommended considering tegola as the server
• No strong views expressed on which renderer to use
• People are generally happy with OSM's approach to tile usage (i.e. no heavy usage, no guarantee) and a vector tile stack may offer opportunities to enforce this through tokens etc.
• Enthusiasm for a simple style editor/designer on osm.org
• Hardware is doable - OSMF has several tileservers for osm-carto; it's possible that one of those could be repurposed (or a new one identfied) for serving vector tiles
• Paul suggested that JS integration into osm.org should be less difficult than I'd anticipated

So, next steps?

To me one obvious starting point is a very rough, v0.00001 schema for osm.org vector tiles, which we can use to spin up sample deployments and start to develop against.

In due course we will probably need to move away from this operations ticket and set up a separate repo, but I hope the ops team will indulge us for a short while yet!

[tomhughes]

I'm not sure that I agree that we decided that "We are not intending to serve raster tiles from the vector stack, but solely deliver vector tiles to the client" actually.

Rather my memory is that was a strong feeling that while we may not use the capability initially we need to preserve it as an option so that we can support older clients one a vector style reaches the point where we can consider making it the default view.

[systemed]

Thanks, yep you're right.

[kocio-pl]

I have set a thread on a forum to discuss vectors:

https://forum.openstreetmap.org/viewtopic.php?id=62160

which is technically more inclusive than GitHub (it's an OSMF asset with standard mapper login) and neutral (OSM Carto has it's own vector-related tickets - https://github.com/gravitystorm/openstreetmap-carto/projects/5).

Could we move there with general vector issues and leave this ticket for implementation details for example?

[theel0ja]

This also would allow better indoor maps with floor selector instead of just piling levels on top of each other.

[kocio-pl]

@slibby I wait to hear more informations about it. Any plans about publishing repo?

[matthijsmelissen]

well we are not setup at all for client side rendering at the moment so it would be a lot more complicated then the normal addition of a new tile layer.

@tomhughes What would it take, technically, to display the ESRI vector tiles on openstreetmap.org? Assume for now that we would be allowed to use their tiles. As far as I understand, this would only require some changes to the openstreetmap-website front-end code, correct? Would this be very complex?

[tomhughes]

How would I know?

My understanding from conversations at SOTM is that there's no sane way to do it - basically everybody said that leaflet doesn't really support vector tiles in a useful way so you need to use OpenLayers or Mapbox GL but that means having different layers using different frameworks which I'm not even going to consider as a reasonable solution.

The only alternative would be to switch everything back to OpenLayers, which will be a lot of work.

So basically, as far as I know (and I have little practical experience with vector tiles) there is no quick or easy way to do it.

[matthijsmelissen]

Thanks, I understand the problem now. Does anybody know if there is any work being done on the Leaflet side to support vector tiles? Are the Leaflet people even aware that we're running into this?

[tomhughes]

I think somebody said there's a plugin for leaflet but that it's not very good.

I imagine mapbox are concentrating on Mapbox GL now rather than Leaflet.

[theel0ja]

Mapzen's vector maps were on top of Leaflet.

https://github.com/tangrams

Demo: https://tile.nextzen.org/preview.html#lat=38.9104&lng=-77.0540&z=12.2500

[tomhughes]

Like I say I'm not expert and I'm probably misrememebering everything.

Asking @pnorman would probably get you more sensible answers.

[theel0ja]

There is also Mapbox GL wrapper for Leaflet, but it doesn't have rotation, and performance isn't as good etc.

https://github.com/mapbox/mapbox-gl-leaflet

[kocio-pl]

Few weeks ago on the SotM US 2018 there was a presentation about Esri vector maps (including OpenStreetMap Vector Basemap) by Jessica Acosta Rodriguez - you can watch the video recording here:

https://2018.stateofthemap.us/program/osm-all-your-esri-base-imagery-are-belong-to-us.html

While it's nice to see this style following our changes and the reasonable speed it achieves, it's not something that we could directly use, even if it's on CC BY-SA, because it requires ArcGIS engine, which is proprietary.

Are there any other news about vector maps for showing on OSM website? Anything we could do now?

To me one obvious starting point is a very rough, v0.00001 schema for osm.org vector tiles, which we can use to spin up sample deployments and start to develop against.

@systemed Is there something happening with it?

(Sorry for posting the same message multiple times, it looks like there are some problems with GitHub comments)

[jgravois]

it's not something that we could directly use, even if it's on CC BY-SA, because it requires ArcGIS engine, which is proprietary.

Just to clarify, since (Esri) is already hosting the published tiles, there's no need to use paid software to consume them. The JSON style file that powers the cartography is also Mapbox style spec compliant and reusable.

For those that are interested in hosting OSM vector tiles themselves, it's too bad that @skmoore's talk didn't get recorded because he dug into the PostGIS backend that powers our hosted service.

https://2018.stateofthemap.us/program/breaking-down-silos-osm-and-esri-side-by-side.html

we're currently looking at parsing the minutely .xml osm diffs and loading the updates into PostGIS to increase our own publishing velocity. this means there is a lot of opportunity for collaboration outside of the proprietary ✨ stuff.

[kocio-pl]

Would it be possible to include your tiles as one of the featured layers on OSM.org?

https://wiki.openstreetmap.org/wiki/Featured_tile_layers/Guidelines_for_new_tile_layers

[tomhughes]

Only if they have server side rendering - we have no support for client side rendering at the moment so the guidelines are moot unless there are bitmap tiles we can consume.

[kocio-pl]

Sure. But this ticket is about using vector tiles to achieve Localized Map Rendering and Clickable POIs on OSM.org - no server side rendering is suitable for this. And no matter if we will create it ourselves or use some external solution, we need some changes in the code of OSM.org to support it - if not now, then later, but there's no other way.

But first I'd like to know what Esri thinks about using their tiles this way at all. Maybe their engineers could even help with the code? I don't know what kind collaboration they are ready for, that's why I ask. I look for all the possible solutions.

[tomhughes]

Right, but that's a long term project - what I was trying to say is that just getting OWG approval is not enough here as the technology doesn't exist to "just" add a layer even if we had a source.

Realistically the kind of features you're talking about are things we would want as the default layer anyway so we wouldn't want to use a third party tileset for them.

I have no objection (assuming it complies with the policy) to a layer based on ESRI tiles but I wouldn't want to add lots of extra features like localised rendering and clickable POIs that were only supported on a non-default layer.

[mboeringa]

and clickable POIs that were only supported on a non-default layer.

I think that particular functionality is much better served on OSM.org by a solution like the one Mark Zoutendijk developed for his OpenPoiMap:

http://openpoimap.org/

This doesn't depend on vector tiles, but seems to get its data straight from Overpass API.

The advantage is that it doesn't depend on a particular rendering stack or technology. Yes, sure, clicking an existing icon in a vector tile and see metadata of the POI popup may have some kind of allure, but given the fact that at most scales except Z20/21, you can only render a fraction of the existing POIs, it is also a very limited view of reality.

[kocio-pl]

Both ways - using external tiles and creating internal ones - are not short term goals, rather medium term. But it all starts with something - and that is why I'm asking @jgravois and @systemed (or anyone else who knows something) about some basic things first.

[mmd-osm]

@mboeringa : I have to disagree with your proposal. Besides usability issues, this kind of usage pattern will simply create way too much load on today's Overpass API instances, and requests will be very quickly discarded with HTTP 429 Too many queries error messages due to rate limiting.

There's a number of open pull requests and issues to significantly improve response times and CPU load for very small and short queries. Without those in place, I wouldn't even start thinking about rolling out anything resembling openpoimap to a larger audience.