search

azavea / osmesa

OSMesa is an OpenStreetMap processing stack based on GeoTrellis and Apache Spark
Apache License 2.0
80 stars 26 forks source link

OSMesa Dataset for OSM records #105

Closed echeipesh closed 5 years ago

echeipesh commented 5 years ago

Currently OSMesa has streaming data sources that read augmented diffs and changsets:

osmesa.common.streaming.AugmentedDiffProvider
osmesa.common.streaming.ChangeProvider
osmesa.common.streaming.ChangesetProvider

The logic backing these can be used to implement OSMesa DataSource that will consume the same files given start and end date and produce a Dataset[OSM] DStream[OSM]

This is very useful for debugging, working in Zeppelin notebooks and creating backfill batch jobs.

Enables: https://github.com/azavea/osmesa/issues/48

mojodna commented 5 years ago

See #106 for a PR covering this.