azavea / osmesa

OSMesa is an OpenStreetMap processing stack based on GeoTrellis and Apache Spark
Apache License 2.0
79 stars 26 forks source link

Add measurements + counts for landuse and natural tags #195

Closed CloudNiner closed 4 years ago

CloudNiner commented 4 years ago

For measurements, only include landuse and natural objects that are areas.

For counts, count everything.

I switched withDelta to withLinearDelta to clarify that it only computes deltas for linear features. An alternative implementation would be to have the original withDelta method compute both area and linear deltas, but then its unclear in the generated column whether the number has length or area units.

VectorPipe uses an outdated set of area keys from github.com/osmlab/id-area-keys. Counts should be more accurate if we update to the latest area keys list. @mojodna 💯 points for the comment noting where the area keys map came from including version! Here's the current diff: osmlab/id-area-keys@f645eca#diff-f36fb8b1baaccf032007cfcc70836c1d

CloudNiner commented 4 years ago

Tested on client staging environment. Here's an excerpt from one of the changesets I used to test:

id              | 954222
measurements    | {"road_km_modified": 0.0000000007130242920538876, "landuse_km2_added": 1.8527665672200757, "natural_km2_added": 3.9711439174635026}
counts          | {"other_added": 1, "landuse_added": 2, "natural_added": 1, "roads_modified": 2}
total_edits     | 6

It looks like performance creating the batch backfill is approximately the same. Previous runs took ~40 mins with the config in this repo, now they take maybe a few minutes more at most.

@jpolchlo this change is worth another look, there's been a few tweaks since you last looked.