This depends on #320 - Elasticsearch 1.0 features a dedicated backup/restore mechanism, see Introducing Snapshot & Restore:
With most/all instances being auto deployed and scaled anyway, the current catch all approach is more of a simple stop gap measure than a reasonable and scalable backup process, which should primarily be concerned with the data.
:information_source: on top of that it might also address related scaling and resilience issues:
We are very excited about this new feature. We like to think of incremental backup as a time machine for your data. We are confident that everyone who relies on Elasticsearch as a critical component in their system and cannot afford down time for re-indexing will find the new Snapshot and Restore mechanism really helpful. [emphasis mine]
This relates to e.g. #272/#273 and esp. #313 accordingly - @dpb587, @mrdavidlaing: I've been planning to summarize my thoughts around the implicit and explicit usage (or current lack thereof) of the Event Sourcing pattern within our cluster for over a week, but didn't get around to it yet - so in an extreme nutshell (there's much more to it, and as always a few issues as well):
an explicit event sourcing approach would e.g. facilitate Apache Kafka/S3 to record the events and make them available for replay in restore and scaling operations
depending on the use case, this pure version often doesn't scale
therefore it is usually desirable to start from an interim pre processed state, and that's what @dpb587 is currently experimenting with via the EBS volume reuse in fact
:interrobang: accordingly, the new incremental backup/restore functionality might allow to achieve just that in a reliable and scalable way?!
This depends on #320 - Elasticsearch 1.0 features a dedicated backup/restore mechanism, see Introducing Snapshot & Restore:
With most/all instances being auto deployed and scaled anyway, the current catch all approach is more of a simple stop gap measure than a reasonable and scalable backup process, which should primarily be concerned with the data.
:information_source: on top of that it might also address related scaling and resilience issues:
This relates to e.g. #272/#273 and esp. #313 accordingly - @dpb587, @mrdavidlaing: I've been planning to summarize my thoughts around the implicit and explicit usage (or current lack thereof) of the Event Sourcing pattern within our cluster for over a week, but didn't get around to it yet - so in an extreme nutshell (there's much more to it, and as always a few issues as well):