hackoregon / openelections

Open and Accountable Elections
https://openelectionsportland.org
MIT License
16 stars 1 forks source link

Your Amazon ElastiCache for Redis Cluster is Auto-failover enabled, yet not Multi AZ enabled #1056

Open jaronheard opened 4 years ago

jaronheard commented 4 years ago

[Email received from AWS]

Hello,

We are reaching out to inform you about configuration actions we recommend you take to increase the robustness of your Amazon ElastiCache for Redis clusters.

We noticed that your Amazon ElastiCache for Redis clusters enumerated below have Auto-Failover enabled but the clusters are honed in a Single Availability Zone. We recommend that you configure these clusters to use Amazon Elasticache for Redis Multi-AZ.

Multi-AZ [1] is an enhanced availability feature that ensures that the nodes of your Redis cluster are correctly placed in different AWS Availability Zones. During certain types of planned maintenance, or in the unlikely event of ElastiCache node failure or Availability Zone failure, Multi-AZ will automatically detect the failure of a primary, select a read replica, and promote it to become the new primary.

Please take the following steps in the AWS Console [2] to configure these clusters to use Multi-AZ.

1.      Log into your account. 2.      Navigate to ElastiCache and select Redis from the left pane. 3.      Find the cluster you would like to reconfigure using the list in the email and the filter search. 4.      Click on the cluster name to review the node allocation. To switch to Multi-AZ, you will need to place one of your replica nodes in a different AZ than the primary. 5.      If cluster mode enabled, click on the appropriate shard name(s). Click on the "Add node" button and select a different AZ than the AZ of the primary. Use 'Zone' column to identify AZ. 6.      If desired, you can delete one of the existing replicas that share the same AZ as the primary. 7.      Click on the chevron (blue) next to the cluster name at the top to go back to the cluster view. 8.      Click on the checkbox next to the cluster name. 9.      Click on the "Actions" button and select "Modify". 10.     Check the Multi-AZ.

For more details please see the Multi-AZ FAQs [3].

Your clusters that can be configured with Multi-AZ for enhanced availability are listed below:

Cluster name | Region

openelections-staging | us-west-2 openelections-production | us-west-2

If you have questions or concerns, please contact AWS Support [4].

[1] https://aws.amazon.com/marketplace/solutions/infrastructure-software/high-availability [2] https://console.aws.amazon.com/console/home [3] https://aws.amazon.com/elasticache/faqs/#redis-multi-az [4] https://aws.amazon.com/support

Sincerely, Amazon Web Services

Amazon Web Services, Inc. is a subsidiary of Amazon.com, Inc. Amazon.com is a registered trademark of Amazon.com, Inc. This message was produced and distributed by Amazon Web Services Inc., 410 Terry Ave. North, Seattle, WA 98109-5210


Reference: https://phd.aws.amazon.com/phd/home#/event-log?Event%20ARN=arn:aws:health:global::event/ELASTICACHE/AWS_ELASTICACHE_OPERATIONAL_NOTIFICATION/AWS_ELASTICACHE_OPERATIONAL_NOTIFICATION_359aa14e-83c1-493e-9ce8-05d36141994e&eventID=arn:aws:health:global::event/ELASTICACHE/AWS_ELASTICACHE_OPERATIONAL_NOTIFICATION/AWS_ELASTICACHE_OPERATIONAL_NOTIFICATION_359aa14e-83c1-493e-9ce8-05d36141994e&eventTab=details&layout=vertical

jaronheard commented 4 years ago

@andrewbiang888 is this relevant to #1058 ?

andrewbiang888 commented 4 years ago

I think if we were seeing downtime it would be. I just spent more time with the current errors and I take back what I said. I don't think it is Redis related (at least not right now 🤪)