Evaluate migration to event sourcing for zero downtime blue green deployments

[sopel]

This has been extracted from #321 and relates to epic #292 accordingly - I've mentioned this briefly only in a recent hangout and we skipped the discussion, because it will likely require significant architectural changes, but a recent data loss event during the currently disruptive weekly deployments suggests to at least evaluate soon whether our long term goal for zero downtime Blue Green Deployments and data backup/archiving could/should be addressed nicely by means of the Event Sourcing pattern.

I'll first update the extreme nutshell summary (there's much more to it, and as always a few issues as well):

• an explicit event sourcing approach would e.g. facilitate Apache Kafka (#273) and S3 (#272) to record the events and make them available for replay in restore operations (and possibly scaling operations too, see #313)
• depending on the use case, this pure version often doesn't scale, because replaying all events lasts way too long
• therefore it is usually desirable to start from an interim pre processed state, and that's what @dpb587 is currently experimenting with via the EBS volume reuse in fact has successfully implemented meanwhile
  • :interrobang: another option might be to facilitate the new incremental backup/restore functionality to build parts or all of the process upon (see #321)?!

The approach would be roughly as follows:

• A new cluster will be started from a well known state like the previous day's backup (or a more current EBS volume snapshot, or a resp. Elasticsearch backup) and then reprocess all source log events from that point in time until it has caught up with the running live cluster, at which point they will be switched.

Obviously there are quite some caveats in practice (e.g. requiring maintenance mode for non event sourced data like dashboards etc.), but it is a valid and successfully employed pattern to thoroughly approach the issues we are facing and should at least be considered accordingly.

If that's something considered to be relevant right now already, I could try to summarize my slightly more elaborate thoughts around the pros and cons, but that's an effort in itself, so let's first discuss our resp. mid term goals/priorities in today's hangout.

[sopel]

We have agreed to discuss this as a mid to long term option and I'll summarize my thinking around the pros and cons (resp. tradeoffs) here and/or in a blog post within the next couple of weeks.

[sopel]

Closed as Incomplete due to not being actively evaluated right here, but being the mid-term goal now regardless - going forward, this will be discussed and hopefully implemented via https://github.com/logsearch/logsearch-boshrelease/issues/73.