Import DataFrame-based code for ingesting OSM

azavea / osmesa

OSMesa is an OpenStreetMap processing stack based on GeoTrellis and Apache Spark

Apache License 2.0

80 stars 26 forks source link

Import DataFrame-based code for ingesting OSM #27

Closed jpolchlo closed 6 years ago

jpolchlo commented 6 years ago

This PR pulls in code contributed by Seth Fitzsimmons into the Ingest project as a means to replace some Vectorpipe functions that have not scaled well to planet-scale ingests. These functions have been confirmed to run on a subset of OSM, but not on the full planet history file.

jpolchlo commented 6 years ago

And sorry for the o'erhasty merge, @mojodna. Didn't think to ask for your review. Will ask in the future!

mojodna commented 6 years ago

No problem!

Looking forward to using spark-submit directly ;-)