Shared NAT gateway across availability zones

[bvosk]

Consumers of this module recently received the following email from AWS.

Hello,

We have observed that your Amazon VPC resources are using a shared NAT Gateway across multiple Availability Zones (AZ). To ensure high availability and minimize inter-AZ data transfer costs, we recommend utilizing separate NAT Gateways in each AZ and routing traffic locally within the same AZ.

Each NAT Gateway operates within a designated AZ and is built with redundancy in that zone only. As a result, if the NAT Gateway or AZ experiences failure, resources utilizing that NAT Gateway in other AZs will also be impacted. Additionally, routing traffic from one AZ to a NAT Gateway in a different AZ incurs additional inter-AZ data transfer charges.

The main route table of your VPC, which holds a route pointing to the NAT Gateway, is associated to subnets in multiple AZs.

We advise that you keep this route table associated only to the subnets that are in the same AZ as the NAT Gateway in its route entry. For subnets in other AZs, please associate separate route tables with routes to existing or new NAT Gateways in the same AZ. Please note, we recommend choosing a maintenance window for architecture changes in your Amazon VPC.

We should take their advice and provision a NAT gateway in each availability zone.

[bvosk]

I brainstormed this with @dumptruckman. Here is the plan.

• Rework the existing public_private_subnet_pairs variable to nest subnet pairs under their corresponding availability zone
  • This will allow us to provision resources per availability zone in addition to per subnet pair. That should make these changes possible/easier
  • This work will be a breaking change anyway, so breaking the interface is fair game
• Provision public utility subnet per availability zone. We'll put a NAT gateway in each of these
  • The bastion host will remain in one of these utility subnets. Changing that is out of scope.
• Rework the private route tables accordingly

Here is a before/after of the networking architecture to aid discussion.

Before

After