azavea / osmesa

OSMesa is an OpenStreetMap processing stack based on GeoTrellis and Apache Spark
Apache License 2.0
80 stars 26 forks source link

Adapt replication sources for use in non-streaming contexts #106

Closed mojodna closed 5 years ago

mojodna commented 5 years ago

Groups of replication sequences can be loaded and unioned with Datasets loaded from ORC files in a static context.

These will appear as DataFrames that are schema-compatible with AugmentedDiff, Change, and Changeset. Some light massaging is necessary in order to match the schema produced when loading from ORC files (specifically, replication-based sources include a sequence column and used mixed-case column names to match the corresponding Product classes).