search

azavea / osmesa

OSMesa is an OpenStreetMap processing stack based on GeoTrellis and Apache Spark
Apache License 2.0
80 stars 26 forks source link

Accompany augmented diff JSON w/ state.yaml #79

Closed mojodna closed 6 years ago

mojodna commented 6 years ago

https://github.com/azavea/augdiff-pipeline/pull/17 contains a Node app that converts Overpass augmented diffs into GeoJSON (with before/after geometries). Sequences are written to S3. Empty sequences are omitted, so the current way to retrieve the "latest" available sequence is to list the bucket/prefix (which gets slower as more data is added).

state.yaml should be created/updated after each sequence is written (or omitting) so that consumers can poll a single object to determine the current, definitive state of the stream.

mojodna commented 6 years ago

Done in https://github.com/azavea/augdiff-pipeline/pull/17/commits/6693ff3f240c12430340eb09185f7d823467c679

Example output:

last_run: 2018-08-01T21:01:16.227Z
sequence: 3095403