The caching layer in logs.cloud.gov uses a single redis instance, so we'll lose logs if this instance becomes unavailable. Proposal: make the caching layer highly available.
Note: this is one of the proposed countermeasures from the retro on the recent logs.cloud.gov outage.
Options:
Use redis sentinel for redis master election and failover
Use consul for redis master election and failover
Use another service for the cache, such as rabbitmq, kafka, or kinesis
Alert on redis failure but leave architecture unchanged
Decide that dropping logs occasionally is acceptable and do nothing
The caching layer in logs.cloud.gov uses a single redis instance, so we'll lose logs if this instance becomes unavailable. Proposal: make the caching layer highly available.
Note: this is one of the proposed countermeasures from the retro on the recent logs.cloud.gov outage.
Options: