brexhq / substation

Substation is a toolkit for routing, normalizing, and enriching security event and audit logs.
https://substation.readme.io
MIT License
330 stars 21 forks source link

build: networking module for custom VPCs #101

Closed ashishxpatel closed 1 year ago

ashishxpatel commented 1 year ago

Description

Include a networking module for a custom VPC that can be created for specific substation resources.

Motivation and Context

In order to allow end users to specify a specific VPC that resources should be deployed in rather than the default.

How Has This Been Tested?

We'll test in our dev envs.

Types of changes

Checklist:

jshlbrd commented 1 year ago

The contributions I made to this PR were based on testing and trying to figure out what the best user experience is. Here are the changes with some reasons why:

The point about VPC endpoints is important because NAT gateways are expensive. The cost of data processed in a NAT gateway is $0.045/GB versus $0.01/GB in a VPC endpoint (and endpoints become more affordable beyond 1 PB). At Brex we're processing upwards of 10 TB per day, so the cost savings with endpoints is significant. Overall I can't recommend that anyone use a NAT-enabled VPC unless absolutely necessary due to the cost. Once we add support for VPC endpoints, the cost will be much more in line with how we've built the rest of the project (meant to keep costs low).