vector tiles

[ghost]

• [ ] generate geojson for bounding box queries (will need polygon detection: osmtogeojson can do this)
• [ ] map tags to layers: layer reference, test_mapping.yml
• [ ] create vector tiles from geojson layers: geojson-vt

However this works, the vector generation should be isolated so that it's possible to implement different kinds of scheduling and merging strategies with pluggable concurrency for webworkers and separate processes.

[gmaclennan]

What are you thoughts about maintaining this as an index/materialized view? As in each peer has the whole vector tile set for the whole db, and it's updated incrementally on data edits or replication? This might be hard to implement... And computationally costly perhaps? I guess the option is doing the computation on replication and edits, or doing it on requests for tiles.

[ghost]

Something that could work: if edits go into a bucket of geometry that always gets drawn over the existing map data and when the bucket fills up, the changes are written to the relevant tiles.

This would make the amortized write speed for many documents reasonable. I think it might be pretty slow to re-generate a whole tile when a single point moves, but perhaps not. These tiles could also be lazily generated when the map viewer pans over a region with buffered geometry.

A more sophisticated version of this would be to implement buckets that increase by powers of two with a cascading strategy similar to how a log structured merge tree works.

[ghost]

here's a prototype:

var togeojson = require('osmtogeojson')
var osmdb = require('osm-p2p')
var geojsonvt = require('geojson-vt')

var osm = osmdb('/tmp/osm')
if (process.argv[2] === 'create') {
  var value = JSON.parse(process.argv[3])
  osm.create(value, function (err, key, node) {
    if (err) console.error(err)
    else console.log(key)
  })
} else if (process.argv[2] === 'query') {
  var q = process.argv.slice(3).map(csplit)
  osm.query(q, function (err, elems) {
    if (err) return console.error(err)
    var geo = togeojson(wrap(elems))

    var tileIndex = geojsonvt(geo);
    console.log(tileIndex.getTile(1, 0, 0))
  })
}

function csplit (x) { return x.split(',').map(Number) }
function wrap (elems) {
  return {
    version: 0.6,
    generator: 'osm-p2p',
    elements: elems.map(function (elem) {
      if (elem.type === 'way' && elem.refs) {
        elem.nodes = elem.refs
        delete elem.refs
      }
      return elem
    })
  }
}

$ node geo.js create '{"type":"node","lat":64.5,"lon":-121.5}'
08dd4fd15fe5c3a1
$ node geo.js create '{"type":"node","lat":62.3,"lon":-119.2}'
2b53f39c7ddeec36
$ node geo.js create '{"type":"node","lat":60.6,"lon":-121.2}'
d343007221ddf9d1
$ node geo.js create '{"type":"way","refs":["08dd4fd15fe5c3a1","2b53f39c7ddeec36","d343007221ddf9d1"]}'
4745d8220fd6a50d
$ node geo.js query 60,65 -122,-118
{ features: [ { geometry: [Object], type: 2, tags: [Object] } ],
  numPoints: 3,
  numSimplified: 3,
  numFeatures: 1,
  source: 
   [ { geometry: [Object],
       type: 2,
       tags: [Object],
       min: [Object],
       max: [Object] } ],
  x: 0,
  y: 0,
  z2: 2,
  transformed: true,
  min: [ 0.16249999999999998, 0.26349642044526056 ],
  max: [ 0.16888888888888887, 0.28703564461585007 ] }

However, it's not immediately apparent how to stuff these data structures into mapbox-gl which has rather sprawling documentation and the examples all seem to revolve around API keys and tile servers.

It seems that geojson-vt is built around the idea of taking a massive geojson file as input and dicing it up into vector tiles, but osm-p2p-db has its own spatial index for cutting up the data into smaller chunks, so there is some overlap in featureset. I think a library that could output a single vector tile or svg file from geojson would be more useful and easier to implement into map tiles.

[gmaclennan]

Ok, so to stuff these into mapbox-gl-js they need to be served locally over http as protocol buffers according to vector-tile-spec and we configure mapbox-gl-js by passing a style object which defines a source as:

"osm-p2p": {
  "type": "vector",
  "tiles": [
    "http://localhost:8080/tiles/{z}/{x}/{y}.pbf"
  ],
  "maxzoom": 17
}

(NB. for mapbox-gl-native you can only pass a URL as a style, not an object, so we would need to also serve this style object locally over http. This is what @hallahan has been doing - would be interested to hear what issues he ran into)

I agree about the overlap of feature set. Looking through the geojson-vt source it seems to do 4 functions:

1. Simplify data at different zoom levels
2. Calculate tile boundaries and clipping
3. Project data into web mercator
4. Convert to protocol buffer format

The blog post is very interesting - it gains its speed by pre-processing simplification data by adding an 'importance' to each point. This we could maintain as an index in osm-p2p-db. The geojson-vt seems to be well modularized that would make it easy to use the processing steps that we need, or replicate them. Regarding the spatial index, I wonder what performance gains might be made from implementing a tile-based spatial index in osm-p2p-db?

[ghost]

implemented in https://github.com/digidem/osm-p2p-vector-tile-server