PMTiles support

[systemed]

This adds support for generating .pmtiles archives. Format is autodetected from the output filename (i.e. .pmtiles->PMTiles, .mbtiles->MBTiles).

This does not produce the most efficient .pmtiles archives possible. Threading means tilemaker's tile output order isn't sequential, so we can't set .clustered. We also don't do anything smart with optimising directory sizes.

Memory usage and execution time will be higher than with .mbtiles. In particular, we need to keep an index of the file location (offset) of each tile we write. This is then compiled into root/leaf directories after tile generation. There's probably scope to improve this!

Some implementation details:

• File order is header, root directory, tile data, leaf directories, metadata.
• We keep an index of any tiles under 100 bytes (uncompressed), such as a typical sea tile, so we only need to write these out once.
• Tiles below z6 are written to the root directory. All other tiles go to leaf directories with a maximum size of 10000000 entries.
• For areas of <100 z6 tiles (Europe and below), we use a map for the tile offset index. For larger areas (planet), we use a vector.
• Compression is always gzip.
• There's a small amount of refactoring to move MBTiles-specific code out of tilemaker.cpp.

PMTiles Viewer seems happy with the results at county level. I've successfully generated a Great Britain .pmtiles and verified it with pmtiles verify (both with sparse and dense index) but not viewed it yet.

/cc @bdon

[bdon]

left a comment inline, to summarize:

Tiles below z6 are written to the root directory. All other tiles go to leaf directories with a maximum size of 10000000 entries.

You definitely want an all-root archive whenever possible for city-sized tilesets

Threading means tilemaker's tile output order isn't sequential, so we can't set .clustered

The only consequences here should the directories take up more bytes, and you can't use pmtiles extract on an output. For cloud storage there isn't a huge locality advantage in accessing nearby parts of the same file

[systemed]

Tiles below z6 are written to the root directory. All other tiles go to leaf directories with a maximum size of 10000000 entries.

You definitely want an all-root archive whenever possible for city-sized tilesets

Good call. Changed in https://github.com/systemed/tilemaker/pull/620/commits/03f857518603baa41333e242576781652b4550b9 - it will now use just the root directory below a certain number of entries (set to 2200 which seems to work reliably and still leave a bit of headroom).

Threading means tilemaker's tile output order isn't sequential, so we can't set .clustered

The only consequences here should the directories take up more bytes, and you can't use pmtiles extract on an output. For cloud storage there isn't a huge locality advantage in accessing nearby parts of the same file

Ah, that's good to know.