search

openstreetmap / operations

OSMF Operations Working Group issue tracking
https://operations.osmfoundation.org/
99 stars 13 forks source link

Host vector tiles generated by tilemaker #565

Closed pnorman closed 6 months ago

pnorman commented 3 years ago

The OWG is planning to host vector tiles generated by tilemaker as an experiment. The goal is to get something others can work with to see where more work needs to be done

The intended architecture is

We have some resources we can potentially reuse for this, but the immediate priority is scaling up tile.osm.org with the new dublin and umea servers.

Items that need addressing are

imagico commented 3 years ago

I know you indicate this to be an experiment but still - may i ask what the strategic vision is behind this or in other words: Where do you intend this to go if the experiment should be successful (by whatever definition of successful)?

I mean is it supposed to lead to the OSMF providing a vector tile service for other parties (and if yes: how does it intend to position that relative to commercial services in that domain)? Or is it supposed to be a non-monetary support program/grant for tilemaker development (and if yes: on what basis was it decided to choose tilemaker to receive that support and not other software in the domain of map rendering/data preprocessing for map rendering)?

As you know i have been saying for quite some time that the OSMF needs to strategically invest in development of technologies suitable for scalable high quality cartography based on OSM data and usable for cooperative community projects so in principle i would see favorably any steps in that direction. But i would like to understand how this step fits into such a strategy or if not what other strategic goals it is meant to serve.

tomhughes commented 3 years ago

The immediate goal is to support the development of vector tiles for openstreetmap.org.

imagico commented 3 years ago

Trying to understand what you mean by that - so this is a feasibility study to see if tilemaker can be practically managed operationally to generate world wide vector tiles (with no requirements what that entails specified so far, presumably within the current capabilities of tilemaker) with a weekly update cycle and then recruit some volunteer style developers to develop a map style (or several - presumably using Maputnik) to produce maps targeting yet to be specified needs on openstreetmap.org.

I see a number of issues with that (in particular in the long term regarding the strategic needs for community map styles providing meaningful constructive feedback for mappers). But this is evidently not a very suitable venue to discuss such matters (although i would very much suggest you to have such a discussion with a broader reach - not many people follow this tracker and few would be inclined to comment here).

Independent of the specifics discussed here i would suggest to clearly distinguish between operational work (stuff that serves concrete and well specified immediate needs) and strategic endeavors and programs. When you state the goal to support the development of vector tiles for openstreetmap.org it is not clear if that is an operational goal (in which case it seems a bit underspecified - see above) or strategic endeavor (in which case a more long term vision what strategic goals this should serve would be highly advisable IMO).

tomhughes commented 3 years ago

What I said on the operations call yesterday (and this is my personal view) is that I hoped that providing them would break the impasse that seems to exist where people can't work on a style because there's no tiles but don't work on tiles because there's no style.

My hope would be that people will start developing a vector style we can use and that there can then be a feedback cycle between that and any necessary changes to the tile schema until we have something we feel is deployable.

At that point we can worry about what stack to use in production and how best to generate and serve vector tiles for that.

imagico commented 3 years ago

Understood. Just keep in mind that when the OSMF rolls out something like that there will be a message of endorsement (of the tools used, their capabilities and their limitations and the standards they are built upon) and a message of strategic direction (in what direction community cartography in the OSM community should develop) perceived even if no such message is intended.

bdon commented 3 years ago

(edit: not relevant to this ticket)

This is exciting! I'm new to this effort, but I'm wondering it the intent here is to offer an experimental vector layer visible on OSM.org? I predict needing some messaging to manage expectations around it not being an alternative or successor "basemap" to OSM Carto (even if that is a long term goal, it would depend on heavy development of a cartographic product like Tom described)

What would be immediately useful is if the vector layer provided both: 1) a language switcher to change the name: tag rendered by by the browser 2) integration with OSM.org User Preferences language settings to default to those language tags

Even if the vector layer displays only place= points+relations and no other geometries, it would be immediately useful as a tile-based web visualization of multilingual data already in OSM; OSM Carto only shows name, and iD only works at very high zoom levels. Even better would be if the rendered name tags were clickable links to a /node/12345/edit page.

tomhughes commented 3 years ago

Please can we keep this ticket on topic and not go off on crazy tangents.

This ticket is just about getting an experimental vector tile source deployed.

Writing a vector style is a matter for other people to be discussed in other places.

Deploying that style to openstreetmap.org is a matter for operations and/or the web site developers once it exists, which is a logn way off.

Once we have the tiles we will probably put some sort of minimal style up on a separate domain as a way of visualising them in the short term, with the hope that somebody will then start to develop a full scale style that we can then put in it's place until it's ready to consider a production deployment.

We are all well aware of the additional opportunities that vector tiles present, which is why we are trying to encourage people to work in that direction, but this is not the place to discuss wider issues.

nyurik commented 3 years ago

Tilemaker explicitly states:

But don't use tilemaker if:

  • You want the entire planet

Has this concern been addressed, or is there a workaround of sorts that might not be good for others? While we could say that the stack discussion should be postponed, the specific settings to convert download into tiles is stack-specific, so deploying a solution that is not optimal for others could in theory lead to duplicated efforts.

P.S. This is a great news regardless, thank you for working on getting MVT to OSM!

tomhughes commented 3 years ago

We literally had the author of tilemaker on the operations call on Wednesday to discuss this so yes, we have discussed it.

Firefishy commented 3 years ago

RAM estimate requirement is between 100GB and 200GB. Estimate time to generate for the full planet is around 4 days runtime on a 2x Intel Xeon X5650 using 144GB RAM system.

@systemed would be able to provide more clarity.

systemed commented 3 years ago

Yes, that's about right; hopefully a bit less time than that. Currently finishing off a set of performance optimisations which will give some harder numbers.

klokan commented 3 years ago

If I understand it well, the goal of the experimental vector tile service is to boost the OSM vector tile style development and to improve the schema for OSM vector tiles - and allow regular iteration on development by the community.

The community around @openmaptiles would be very happy to help with these efforts.

For the upcoming OMT v4 these things may be of interest to OSMF:

If OSMF is interested - we could provide a weekly updated downloadable tile package for self-hosting and/or live tile endpoint (for proxy via your CDN or direct use on your DNS), based on the latest OSM data and powered by the latest modifications of the schema in GitHub. You could also run the open-source components on-prem with Kubernetes. Just let us know.

Delay on pre-generating an entire planet is about a day - less if materializing only upper zoom levels, with real time on lower zoom levels served from an endpoint, or if only diffs are generated.

pnorman commented 3 years ago

Hosting tilemaker as an experiment does not preclude us from hosting other vector tile solutions. This is something we're planning on hosting using resources that will become available later this year. It will not be a featured tile layer on the website to start, and if it should be is a question the OWG will consider at the time. We have in the past hosted experiments that did not end up on osm.org, and looked at multiple approaches for a goal.

For using a live tile endpoint, are you thinking of that on osm.org? We had spoken to someone back in March with the call for featured layers, but communication died out. If you're interested in that, please email information about it and meeting the criteria of the featured layer policy to the OWG.

If you're thinking of instead asking the OWG to host an instance of the OMT stack, please open a new issue so this one remains about maptiler.

matkoniecz commented 3 years ago

Maybe tiles can/should be hosted on vector_tiles.experimental.openstreetmap.org or completely different domain to reduce

keep in mind that when the OSMF rolls out something like that there will be a message of endorsement

effect?

simonpoole commented 3 years ago

@matkoniecz it is rather clear that what the OSMF runs as its "standard stack" has in the past and likely will in the future have a big effect on the market for such solutions. Just consider, for example, if something else than the openmaptiles schema is chosen (a distinct possibility) for the tiles.

ZeLonewolf commented 2 years ago

My hope would be that people will start developing a vector style we can use and that there can then be a feedback cycle between that and any necessary changes to the tile schema until we have something we feel is deployable.

The openstreetmap-americana project is presently in the early stages of developing a vector style for the US mapping community, with a small team of active contributors and nearly 100 folks dialed into our project's Slack channel. The focus on our project is to develop a regional style with features of interest to the US community, most notably comprehensive highway shield rendering with concurrencies. This project is currently based on the OpenMapTiles schema. If the OWG-hosted vector server is using an OpenMapTiles configuration, we would be more than happy to collaborate for this purpose.

ZeLonewolf commented 2 years ago

I'm pleased to report we've managed to perform a one-time planet build of the OpenMapTiles schema at full zoom (z14) using planetiler, and are hosting it on AWS on a temporary basis. I'm providing a description of this test planet build, costs, and technical details in case it is useful for anyone working on cloud vector tile hosting with OpenMapTiles. I also have notes that I took during this process that I'm happy to share with anyone looking to create a similar setup.

The tile server is running here

The key advance of planetiler is the use of in-memory transformations for mbtiles generation. As a result, planetiler is able to execute OpenMapTiles builds orders of magnitude faster than the out-of-the box scripts in the OpenMapTiles distribution. The planet render was done on an AWS m5dn.8xlarge instance, which has 128GB RAM, 32 CPUs, and 2x600GB SSDs (of which we only used 1x600GB). We executed the render on a variably-priced "spot" instance which was running approximately 40 cents per hour at the time of the render. The render took 2h 0m 6s to run on this hardware, plus an additional 6 minutes to transfer the resulting 100GB planet mbtiles file to an AWS Elastic File System (EFS) volume ($10/month). Thus we were able to generate a planet mbtiles file for a bit less than $1. However, this may cost as much as $5 for the same render at peak data center times.

Because there is an OSM planet file hosted on AWS, the download of the planet pbf file takes less than three minutes, so this may result in additional download time on a different hosting provider.

Next, for the tile server we spun up a separate AWS t2.nano instance (0.5GB RAM, 1 CPU) and connected it to the EFS volume, installed docker, and ran tileserver-gl with docker's restart-always setting. The t2.nano instance runs at a cost of $0.0065 per hour (a bit less than $5/month), and costs $0.09 per GB of outbound file transfer.

While this is a low-powered setup, it demonstrates the most basic possible infrastructure that can run a planet vector tile build. An enterprise-level vector tile server would ideally also deploy a CDN and load balancer with higher powered servers in multiple availability zones.

Planetiler is nearly complete, however, it is missing an implementation of OpenMapTiles' street abbreviation functionality. In addition, for this render I turned off wikidata integration, which would have added an additional 15 minutes to the render time.

This setup could be conceptually extended to a cyclic build as follows:

  1. Download planet
  2. Update the planet file to latest, for example by running pyosmium-up-to-date
  3. Render planet, dump to file storage location, restart tile server
  4. GOTO 2
SomeoneElseOSM commented 2 years ago

Just a brief question about zoom levels. Currently the max zoom level supported by each of the osm.org maps are respectively 19, 20, 21, 21, 18 and 20. What would the max zoom level (either technical or practical) of a layer created using either tilemaker , openmaptiles, planetiler or anything else?

I appreciate that with vector tiles you've effectively got "almost infinite overzoom" - you can zoom in as much as you like; but what I'm asking here is "at what level can new features appear" - can they start to be shown at the equivalent of raster zoom 18, 19 or 20?

tomhughes commented 2 years ago

That's entirely up to the author of the style.

The actual vector tiles normally stop at z14 or so and obviously you're limited by what data your schema includes in the high zoom tiles but beyond that it's up to the style which features in the tile it renders at each zoom level.

msbarry commented 2 years ago

Planetiler currently limits to z14 but it should be pretty straightforward to increase if people want to go higher. The z14 limit was to support openmaptiles schema which only needs z14. The main reason to go further is if your style starts including more detail and the z14 tiles get too big. The biggest z14 tiles with the openmaptiles schema are about 1.7mb (before gzip compression) and there are only a handful over 1mb so there wouldn't be much benefit for that schema.

Generating more zoom levels will also make the tile render time longer, and mbtiles file bigger. The most recent run I did on planetiler on a beefy ec2 instance took 47 minutes and rendering z14 took 10 of those minutes, so z15 would probably add at least ~40 minutes and z16 would add at least another ~160 minutes.

ZeLonewolf commented 2 years ago

It's also worth pointing out that OpenMapTiles omits quite a few features of the "micro mapping" variety that are supported by osm-carto, so if the goal is a vector schema that supports everything that osm-carto does, we could be looking at very substantial mbtiles size inflation over the current 100GB full-zoom mbtiles produced from OpenMaptiles.

sfkeller commented 2 years ago

I'd suggest to look also at Shortbread https://shortbread.geofabrik.de/ , a lean Vector Tile Schema, which is CC0/FTWPL licensed.

ZeLonewolf commented 2 years ago

I'd suggest to look also at Shortbread https://shortbread.geofabrik.de/ , a lean Vector Tile Schema, which is CC0/FTWPL licensed.

The challenge at the moment is that somebody needs to develop the tilemaker .lua stylesheet or planetiler extension to support generating this or any other schema in vector tiles. Both tools are able to support OpenMapTiles out of the box, other schemas need developer time.

imagico commented 2 years ago

It would be advisable to keep in mind that most non-trivial cartography, especially at higher zoom levels/larger map scales, requires scale/zoom level dependent geometry processing. And the practical feasibility and efficiency of client side map rendering as it is common today is strongly tied to decidedly not transferring geometry data separately for the higher zoom levels.

I know this issue is not meant to be concerned with actual map design. But the question from @SomeoneElseOSM is pointing in the right direction. The bottom line is if you contemplate the question of choosing a server side data preprocessing framework for client side rendering without regards to the needs of community map design you will likely end up providing a basis exclusively for the the dead end of cartographic primitivism a la mapbox/openmaptiles.

It is no coincidence that many of the most innovative studies in high zoom level OSM based map design done more recently (like https://github.com/SupaplexOSM/strassenraumkarte-neukoelln, https://github.com/enzet/map-machine) are not based on any of the more established tools you are contemplating here.

ZeLonewolf commented 2 years ago

requires scale/zoom level dependent geometry processing

This is basic functionality in all of the tools we're discussing here, including out-of-the-box openmaptiles.

imagico commented 2 years ago

Link to zoom level specific geometry processing in openmaptiles at z>14 (or any other framework fwiw)?

ZeLonewolf commented 2 years ago

Geometry generalization is basic functionality in ALL vector tile renderers regardless of which zoom levels you choose to apply it to, whether it's the PostGIS functions used by openmaptiles or the Java Topology Suite used by planetiler, or whatever tilemaker does. It would be helpful if you could provide an example of what you mean by "scale/zoom level dependent geometry processing" because clearly that is done today.

imagico commented 2 years ago

Well - i don't want to get into what would truly be an off-topic discussion here - I was pointing to the need for zoom level specific geometry processing at the high zoom levels, you claimed this to be available routinely in openmaptiles, i was asking for links.

If you wonder about practical use cases - links i gave above show some - as does various stuff on https://github.com/imagico/osm-carto-alternative-colors. But again: This is off-topic here. My point was not discussing specific map design questions, my point was pointing out the inherent limitations of the setups discussed here so far. No one doubts that you can in principle produce a vector tile set with separate tiles for each zoom level up to 24 or higher. But the practical feasibility of all of the client side rendered vector tile systems is based on the fact that this is not done, because - as you said yourself - this would explode the volume of data to be stored and transferred quite massively.

tomhughes commented 2 years ago

Can we please keep this on topic people.

The goal here is to get a basic set of vector tiles up to encourage the creation of a project to develop a full schema and style that we can use to run a production service.

We don't need to discuss the technical minutiae of what a production quality style might look like at this point!

imagico commented 2 years ago

My aim was specifically not to discuss technical minutiae but to point out strategic issues that could well decide if what is encouraged in terms of style is going to be map design primitivism or avant-garde.

tomhughes commented 2 years ago

No problem. I'll just unsubscribe from this ticket and let you get on.

clementmas commented 2 years ago

The goal here is to get a basic set of vector tiles up to encourage the creation of a project to develop a full schema and style that we can use to run a production service.

Can anyone give us an update on the status of this proposal?

woodpeck commented 2 years ago

The challenge at the moment is that somebody needs to develop the tilemaker .lua stylesheet or planetiler extension to support generating this or any other schema in vector tiles. Both tools are able to support OpenMapTiles out of the box, other schemas need developer time.

https://github.com/geofabrik/shortbread-tilemaker.git FYI

tomhughes commented 2 years ago

To get this back on topic, some results from my experiments so far on the hardware we're planning to use for this test:

So planetiler is clearly significantly faster (and much less demanding on the machine's I/O especially) but is currently limited to the single builtin schema though that is being worked on I'm told - how much that might slow it down is unknown of course.

A weekly built test would be entirely possible with any of them, and planetiler could probably do daily if we wanted.

msbarry commented 2 years ago

@tomhughes Nice! Glad you were able to get planetiler running on your setup. My initial goal building planetiler last year was to build out the utilities needed just for the openmaptiles schema to validate that the approach was sound, with a goal of supporting more schemas in the future.

I'm working on a few more performance optimizations to make planetiler runnable on lower-end machine (<32GB ram) and utilize higher-end machines better, but after that I plan to look into how to support configurable schemas without needing to recompile the tool. We've had some initial disscussions here: https://github.com/onthegomap/planetiler/discussions/81, and I'm tracking my progress here: https://github.com/onthegomap/planetiler/projects/2. Tilemaker's lua profiles are a great option for iterating on schema definitions quickly!

ZeLonewolf commented 2 years ago

In case anyone is looking for a status update, there is significant activity around modifying planetiler to be able to configure the tile output (i.e. the data schema, mapping of OSM tags to mvt attributes) happening in the following places:

This is fairly complicated and performance-sensitive work so it's taking a bit of time to get it right, but progress is moving along quite nicely and I believe we're very close to having an initial capability that we can start testing with and incrementally improving.

sfkeller commented 2 years ago

@ZeLonewolf

In case anyone is looking for a status update, there is significant activity around modifying planetiler

  1. Is there any chance now that planetiler also works with Geofabrik's shortbread scheme as well?
  2. Could the copyright implications of the OpenMapTiles scheme when used as official OSM service be clarified?
ttomasz commented 2 years ago

I thought the goal was to start with basically any schema and change as needed. Planetiler already is mostly compatible with OpenMapTiles schema which seemed like most suitable candidate for v1 anyway.

So is the deployment actually held up by waiting for these functionalities? Feels like work on setting up the rest could continue since most of it would be useful regardless of schema.

ZeLonewolf commented 2 years ago

Is there any chance now that planetiler also works with Geofabrik's shortbread scheme as well?

The goal of modifying planetiler to be arbitrarily configurable is that it could work with any schema without having to be a Java programmer to generate it. You'd just create a YAML configuration file and planetiler could generate an mbtiles from this. It remains to be seen whether or not this approach could be used in more complex cases, but we'll cross that bridge as we come to it. I haven't done a deep dive into the shortbread schema yet but I suspect after onthegomap/planetiler#160 is merged we'd be able to produce most if not all of it. The initial goal is a "trivial" output of land, water, and roads as a demonstrar.

So is the deployment actually held up by waiting for these functionalities? Feels like work on setting up the rest could continue since most of it would be useful regardless of schema.

I don't really agree that anything is "held up" and I've been working pretty hard in my spare time on making planetiler configurable.

Do you follow the minutes of the OWG? Here are the minutes from the meeting I last attended where this effort was discussed, and it mentions the concern that the OWG has about using OpenMapTiles as their production schema. This is why there is currently interest in Shortbread or something like it which has a more open license (shortbread is CC0 as noted above in thread).

Or, are you looking for an OpenMapTiles planet vector tile server? Here you go: https://6ug7hetxl9.execute-api.us-east-2.amazonaws.com/

That server is one that I personally host for a cost of around $10 a month. It could go away any time as I'm only using it for Americana demo purposes but it's pretty straightforward to host your own vector tile server. Here's a github repo where I keep the scripts for generating that tile server on AWS (would need to modify to change server locations and so forth to replicate): https://github.com/ZeLonewolf/planetiler-scripts

So bottom line is that a lot is happening, it's just that none of it is really within the openstreetmap-carto organization. I'm personally really excited about vector tiles and what planetiler brings to the table.

ttomasz commented 2 years ago

I don't really agree that anything is "held up" and I've been working pretty hard in my spare time on making planetiler configurable.

Sorry I wasn't sure from the post if OWG is waiting for custom schemas in planetiler or not. It's great that you are working on this and this will surely be a great step for vector tile technology that will allow more people to create their own schemas/hosting. Also thanks for publishing the scripts and your server url :heart:

Do you follow the minutes of the OWG? Here are the minutes from the meeting I last attended

I read this one from march and they were the last to mention vector tiles afaik. I was under the impression that comment about OpenMapTiles was just one person's observation/opinion not official stance/decision.

Actually still I am not 100% sure what's the status on OWG side. Minutes mentioned that test setup of TileMaker would be implemented and minimal schema would be selected. Since then Tom tested all 3 generators and made a point about Planetiler being significantly faster. So was the decision made to use Planetiler instead of TileMaker? And if so is OWG waiting on the custom schema with just the most basic features instead of built in OMT schema? Or is this just a matter of lack of admin time to make the experimental setup with weekly updates (on OMT schema)?

As an aside I packaged some icons into sprites for my use when playing with vector tile maps in dev env, maybe it will be useful for someone else's tests: https://github.com/ttomasz/sample-icon-sprites

grischard commented 6 months ago

We will not be using tilemaker in the end, but tilekiln. There will be separate tickets for that.

michaelblyons commented 6 months ago

Cool. When you make those tickets, can you link them from here so watchers can re-follow?

grischard commented 6 months ago

Sure! Here's the announcement for this vector tile project: https://blog.openstreetmap.org/2024/02/11/2024-announcing-the-year-of-the-openstreetmap-vector-maps/