apache / incubator-baremaps

Create custom vector tiles from OpenStreetMap and other data sources with Postgis and Java.
baremaps.apache.org
Apache License 2.0
519 stars 62 forks source link

Update OSM documentation #574

Open tyrauber opened 1 year ago

tyrauber commented 1 year ago

Very nice project! I was hoping to get some more details about the openstreetmaps implementation. Specifically:

bchapuis commented 1 year ago

Good questions.

  • importer: Is the importer custom to this package, or using an existing library? What is the performance of that import process for the world pbf, compared to Osm2pgsql, Osmium, Osmosis and Imposm.

The pbf parser and database importer are custom to this project. The parser use the Java Stream api to deserialise the pbf in parallel. The importer uses the postgres COPY api to achieve great performance. They should be in par with what other projects are doing.

Our current performance bottleneck is related to geometry simplification (merging highways, creating simplified polygons for landuse, etc.). This currently occurs once the data has been imported in the database. For the planet, this part of the process takes about two days and we are currently trying to optimise this part of the workflow.

  • data structure: It sounds like this is a full import? That it maintains all nodes, ways and relations?

Yes, we import all node, ways, relations and tags.

  • normalization: In order to save denormalized geometries in PostGIS (e.g. linestring, polygon, multi-polygon, etc.), Baremaps creates a cache for nodes, ways and relations. [LMDB](https://symas.com/lmdb/) is used under the hood to achieve great performance. It sounds like this normalization is happening in the server and not in postgres. What is the performance impact of this normalization? Is there a cache warming period on server restart?

This part of the documentation is outdated. We removed LMDB and now rely on memory mapped files to cache the nodes and references. As the pbf files are ordered, we are creating these caches on the fly and read the pbf files only once.

  • updating: Any plans on supporting periodic updates from OSM changesets from something like osmupdate?

Yes, we do have preliminary support for osm updates but it is experimental and not extensively tested.

  • caching: It appears to me like the server is caching tiles. It would be helpful to know how that caching is happening, what are the space requirements, and how one would purge tiles from cache (especially after a changeset has been imported).

Yes, the server uses Caffeine to cache tiles in the RAM

The configuration of the cache can be tweaked with the --cache flag of the map serve command. It takes a caffeine specification as an argument.