azavea / osmesa

OSMesa is an OpenStreetMap processing stack based on GeoTrellis and Apache Spark
Apache License 2.0
79 stars 26 forks source link

Staging deployment and Railway statistics #157

Closed jpolchlo closed 4 years ago

jpolchlo commented 4 years ago

We've been somewhat primitive in the way that we deploy and test different versions of OSMesa. A request arrived where we finally could not ignore the fact that a staging environment was needed, as it showed our terrible lack of sophistication in the way that we were storing and dealing with statistics. Previously, every small change to the stats would require a change in table schema, which required a complex backfill process into an existing and actively used DB table. Following which, we would need to restart a streaming update process and then infill. All because we were amending a live table that was feeding a process visible to end users.

This PR will split our deployments into staging and production environments. This considerably simplifies changes, as they can be private to a development team. Rolling out a new stat will then be accomplished by deleting the contents of any existing tables, repopulating the tables, restarting a streaming process (which can take as long as needed to catch up), and then redeploying dependent services.

This split was prompted by the need for railway statistics reporting which are also added to this PR.

The changes here are mirrored by those in azavea/osmesa-stat-server#22.

jpolchlo commented 4 years ago

Managed a successful ingest of the basic stats. These are currently populating the staging DB. Will take the time to see that we have identified any railway stats, but seems like the JSON stuff works, after slight modification (see last commit). I'll try and fire up the streaming job in the morning.

jpolchlo commented 4 years ago

Can positively report that railway stats are in the table. Also, we are now automatically adding checkpoints to the DB so that we can more smoothly start up the streaming update process

jpolchlo commented 4 years ago

This now depends on geotrellis/vectorpipe#109. I'll need to merge that PR and cut a new minor release prior to merging this contribution.

jpolchlo commented 4 years ago

Seems like the long chain of discussion here has finally played out. Merging.