search

hackoregon / civic-devops

Master collection point for issues, procedures, and code to manage the HackOregon Civic platform
MIT License
11 stars 4 forks source link

Cloudformation stack creation fails due to insufficient Elastic IPs #243

Closed MikeTheCanuck closed 5 years ago

MikeTheCanuck commented 5 years ago

When creating a new stack based on our CloudFormation stack templates, I have repeatedly run into situations where the stack creation fails and rolls back, claiming some combination of failures to do with the VPC and its resources, e.g.:

Screen Shot 2019-06-23 at 11 25 13
MikeTheCanuck commented 5 years ago

I've learned that this problem has a viable (if frustrating) explanation:

When digging through the Stacks in CloudFormation, I finally noticed the Deleted filter (whose stacks are hidden from the default filter) and was able to find a single, repeated type of error from this morning's repeated failures:

Screen Shot 2019-06-23 at 12 16 49
MikeTheCanuck commented 5 years ago

Well, it appears that we're allocating "too many" Elastic IPs in our account/region: https://github.com/hashicorp/terraform/issues/6018#issuecomment-214907400

And this will be a problem, since we've already got 4 of the 5 allowable addresses allocated:

Screen Shot 2019-06-23 at 12 37 41

This is where the limit is documented: https://docs.aws.amazon.com/vpc/latest/userguide/amazon-vpc-limits.html#vpc-limits-eips

MikeTheCanuck commented 5 years ago

So for us to continue developing a test-vs-staging-vs-production infrastructure (plus all the one-off deploys from other teams), we're likely to need to request an increase in the Elastic IPs limit on our account - otherwise, only one test stack can be deployed at a time, making it very hard for multiple engineers to work simultaneously.

Requesting an increase to these limits has one somewhat-mystifying question we need to resolve: https://console.aws.amazon.com/support/home#/case/create?issueType=service-limit-increase&limitType=vpc

Are we currently rate-limited on "EC2-Classic Elastic IP Addresses" or "New VPC Elastic IP Addresses"?

MikeTheCanuck commented 5 years ago

Here's how to request an increase: https://stackoverflow.com/questions/24275080/aws-eip-limit-increase

DingoEatingFuzz commented 5 years ago

I just mentioned this in #239, but I'll say it here also.

I don't think we need separate VPCs in order to have isolated test/staging/production environments.

Budget aside, as long as we have separate ECS clusters and ALBs, then we have isolation.

Including budgets, it's important to be aware that a single NAT gateway costs ~$30/mo.

danieldn commented 5 years ago

Will merge this with #239