Create pmtiles directly

[dabreegster]

This is an experiment to generate pmtiles directly in Rust, not using tippecanoe. It's specialized for the use case in od2net -- a bunch of tiny LineStrings representing segments of the road network. The "importance" of the roads is given by a count property in the GeoJSON data, so this should be used to figure out which features to drop at low zooms.

CC @stuartlynn

"Correctness" to use in the od2net viewer:

• [ ] Include the actual feature properties (and maybe revisit squeezing osm_tags into a single JSON string, to improve encoding)
• [ ] Plumb along the JSON metadata field
• [ ] Decide what to do about the other layers (points representing origins and destinations)
• [ ] Use compression internally (which does tippecanoe use?)
• [ ] Enforce per-tile limits on number of features or total size, by sorting by the count property

Performance of the tool itself:

• [x] Figure out what spatial indexing we can use to filter for each tile. rstar with Web Mercator coordinates? Something specialized for WGS84? Can we directly use tile boundaries as the spatial indexing bins?
• [ ] Does processing each tile in parallel help?

Logistic:

• [ ] File bug about empty geometry encoding in mvt
• [ ] Name this properly and make a separate repo
• [ ] Get feedback from georust and others

[stuartlynn]

Hey I am going to try and take a swing at doing bbox filtering using rstar. I have used it on some projects before with good results.

[dabreegster]

Yes please! Maybe I don't understand coordinate systems well enough, but can this be done with WGS84, or do we have to project to Web Mercator or something else first?

(I won't pick this up until Monday again, no fear of merge conflicts)

[stuartlynn]

I think given that we need to convert to WebMercator anyway to generate the tiles, it makes sense to

1. Load the features
2. Transform to WebMercator
3. Build the RTree
4. Iterate through the tiles and use the RTree to select the features for each
5. Generate the tile
6. Add it to PMTiles.

One thing that might be interesting to experiment with is generating the tiles in parallel (with rayon) and storing them somewhere intermediary before constructing the full pm tiles at the end? That's an enhancement though

[dabreegster]

@stuartlynn , there's some chatter in the georust discord about possible distance metrics that could be useful: https://discord.com/channels/598002550221963289/963346313708195880/1167857249977372733

[stuartlynn]

@stuartlynn , there's some chatter in the georust discord about possible distance metrics that could be useful: https://discord.com/channels/598002550221963289/963346313708195880/1167857249977372733

Not sure that link is working for me. What were you thinking if using the different distance metrics for?

[dabreegster]

Hmm, try joining https://discord.gg/Fp2aape and checking #random around 10/28. Someone pointed out https://github.com/vipera/cheap-ruler-rs. I think web merctor + rstar makes sense to me though.

[dabreegster]

Moved to https://github.com/Urban-Analytics-Technology-Platform/lines2pmtiles