search

azavea / osmesa

OSMesa is an OpenStreetMap processing stack based on GeoTrellis and Apache Spark
Apache License 2.0
79 stars 26 forks source link

Facilities for bulk ingest file prep #200

Closed jpolchlo closed 3 years ago

jpolchlo commented 3 years ago

When reconstructing the staging DB, it is necessary to produce an ORC file for the planet history as well as to update a changeset ORC file. The latter was covered by a make rule, but the former task was previously left to a manual process. This PR adds automation to help with this, and combines it with the changeset ORC update. Now, from ./deployment/batch/makefile, make prepare-bulk-ingest will create a cluster and add steps for both the planet history ORC conversion and the changeset ORC update. The cluster will terminate automatically after the steps terminate.

jpolchlo commented 3 years ago

Sorry. Commit log is a mess from a mistaken addition of some verboten text and the subsequent history rewrite.

jpolchlo commented 3 years ago

Closing in favor of #206 and it's clean tree.