search

azavea / osmesa

OSMesa is an OpenStreetMap processing stack based on GeoTrellis and Apache Spark
Apache License 2.0
79 stars 26 forks source link

Consolidate makefiles #212

Closed jpolchlo closed 3 years ago

jpolchlo commented 3 years ago

The purpose of this PR is manifold, but summed up as "simplify the deployment process". The aim is to consolidate as many steps as possible into make rules that can be executed reliably, without fuss.

Knock on benefits of this PR include the ability to finally retire the osmesa-stats-server by importing the necessary materialized view definitions and refresh infrastructure. This PR will also automate the process of rebuilding the changeset metadata from OSM DB extracts. This will also take a step towards eliminating the deployment/batch subdirectory, and ultimately allowing us to eliminate the distinction between batch and streaming deployments (though the batch infrastructure still provides the means to create EMR clusters for direct interaction (i.e., not pre-canned operations)).

Closes #203 Closes #189

jpolchlo commented 3 years ago

The Docker resources have been restructured. There are now two images for the two distinct tasks, and the structure has been verified in use. This is better, since adding psql to the spark container was a mess, since that image is now on an out of date debian release for which there are no longer apt profiles. Builds are faster and more maintainable for when we bump to Spark 3.