search

cityindex-attic / logsearch

[unmaintained] A development environment for ELK
Apache License 2.0
24 stars 8 forks source link

Add S3 based log event persistence #272

Closed sopel closed 9 years ago

sopel commented 10 years ago

This is a generally applicable question, but has been triggered again by https://github.com/cityindex/logsearch/issues/269#issuecomment-30518105 specifically.

We have experienced loss of ingress log events already in a few scenarios, which is an artifact of our current architecture lacking a reliable message archiving tier, i.e. source messages are lost once consumed by a downstream tier.

Most message brokering solutions elsewhere are using the obvious approach to also archive every event to persistent storage before consuming it by (potential several) downstream tiers, usually S3.

This would enable both, recovery from outages without data loss as well as (selective) replay capabilities for testing etc.

sopel commented 10 years ago

Moved to Icebox due to focus on #270 (as discussed in https://github.com/cityindex/logsearch-config/issues/59).

sopel commented 9 years ago

Closed as Won't Fix due to project being retired to the CityIndex Attic.