This terraform module implements a Linux VM acting as IP router between two VPC networks. The primary use case is an alternative to VPC peering and VPN tunneling for east-west connectivity.
Functionality:
When upgrading to a new versions of this module, follow the process described in UPGRADE.md to avoid downtime.
The core functionality is implemented in the 50_compute/ nested module. This module is intended to be easily reused in your environment.
See examples/compute/ for a complete example which ties together the following resources.
See examples/networksetup for code to create VPC networks and other dependencies necessary to evaluate the solution.
The module operates similar to GKE's model of one instance group per availability zone. Each VPC network has one Internal TCP/UDP Load Balancer forwarding rule. Each ILB distributes traffic across multiple instances in multiple zones within a single region.
Version 3.1.0 and later of this module pins the OS image used for multinic
instances to a specific value. This ensures the same image is used as
instances scale in, scale out, and are auto-healed. In addition, multiple
runs of terraform will use the same image specified by the image_name input
value.
See Deploying the Latest Image for additional information.
ILB addresses should be in the same subnet as the associated network interface so that additional forwarding rules can be added and removed at runtime without having to reconfigure each multinic instance.
Routed VPC networks must be attached to nic0 and nic1 presently. Additional VPC networks may be attached, but they are not configured for policy routing. A future enhancement may support an arbitrary number of attached networks.
Take an instance out of rotation with systemctl stop hc-traffic.service.
Start the auto-healing process with systemctl stop hc-health.service.
Exercise a kernel panic with systemctl start kpanic.service. This is useful
for evaluating failure modes and recovery behavior.
Stopping hc-traffic.service causes new connections to use healthy instances,
if available. Existing connections flowing through this instance continue to
do so.
See Balancing mode for more information.
Stopping hc-health.service causes the instance group to auto-heal
the instance. Existing connections flowing through this instance being flowing
through another healthy instance.
Established TCP connections remain established during the process.
The process takes ~30 to ~40 seconds with the following health check configuration.
check_interval_sec = 10
timeout_sec = 5
healthy_threshold = 2
unhealthy_threshold = 3Auto-healed instances receive the same name as the instance they replace, but have a unique instance ID number.
Triggering a kernel panic with systemctl start kpanic.service exercises
auto-healing behavior. Existing connections pause for ~45 seconds with
check_interval_sec=10 and unhealthy_threshold=3, then recover without
disconnect.
Information regarding startup and shutdown events are logged to the projects
Global resource under the multinic entry. For example:
gcloud logging read logName="projects/${PROJECT_ID}/logs/multinic"There are two types of health checks used in this solution:
The MIG auto-healing health checks will come into nic0.
The Load Balancing health checks will come into the nic attached to the network associated with the ILB forwarding rule being checked.
Helper scripts are included in the scripts directory.
rolling_updateUse this script after applying changes to the instance template used by the managed instance group. The helper script performs a [Rolling Update][rolling-update] which replaces each instance in the vpc-link group with a new instance.
panicTriggers a kernel panic on one of the MIG instances. This is intended to exercise the behavior of auto-healing, and impact to application network flows in unplanned situations.
Linux Policy Routing is configured with the following behavior:
There are two additional routing tables named nic0 and nic1. The tables are identical except for the default route:
Traffic with a source address in the subnet attached to nic1 uses the nic1
routing table. Similarly, traffic with a source address in the subnet attaced
to nic0 uses the nic0 table. This source traffic includes ILB addresses. See
[requirements][#Requirements].
Policy routing is configured on each instance using the
policy-routing.service unit file, which executes /usr/bin/policy-routing.
The startup script is responsible for enabling ip forwarding, configuring policy routing in the instance, and starting the health check endpoints.
Log into the instance, then run:
sudo DEBUG=1 google_metadata_script_runner --script-type startup --debugThe shutdown script is responsible for signaling the load balancer should take
the instance out of rotation. It does this by stopping the hc-traffic
service, then sleeping. The sleep is intended to maintain service levels until
the load balancer health check takes the instance out of service.
sudo DEBUG=1 google_metadata_script_runner --script-type shutdown --debugDue to VPC Network Limits in GCP, the number of link instances in the Managed Instance Group will determine the total bandwidth available between the VPCs. These are different than quotas in that they cannot be changed. To test these limits, metrics are provided using iPerf2.
| Item | Limit | Notes |
|---|---|---|
| Maximum ingress data rate | Depends on machine type | GCP does not artificially cap VM instance inbound or ingresstraffic. VMs are allowed to receive as much traffic as resources and network conditions allow. For purposes of capacity planning, you should assume that each VM instance can handle no more than 10 Gbps of external Internet traffic. This value is an approximation, is not covered by an SLA, and is subject to change. Adding Alias IP addresses or multiple network interfaces to a VM does not increase its ingress capacity. |
| Maximum egress data rate | Depends on the machine type of the VM:
|
Egress traffic is the total outgoing bandwidth shared among all network interfaces of a VM. It includes data transfer to persistent disks connected to the VM. |
iperf -c <INTERNAL IP ADDRESS> -P 100 -t 60
-c Run as client
-P Run multiple clients (If it’s run with one client, all traffic will go through 1 VM)
-t seconds to run test (Allow a large number to average out results)
iperf -s
-s Run as Server
Test 1
3 Link VMs 8vCPU each - Potential Egress 16 Gbps Each 2 Client VMs 16 vCPU each - Multi-Stream Clients 2 Server VMs 16 vCPU Each
Client 1 to Server 1 - 24.4 Gbps Client 2 to Server 2 - 22.4 Gbps Simultaneously
Potential Bandwidth - 48 Gbps Actual Bandwidth - 46.8 Gbps
Test 2
1 Link VM 16vCPU - Potential Egress 32 Gbps 1 Client VM 16vCPU - Multi-Stream Clients 1 Server VM 16vCPU
Potential Bandwidth - 32 Gbps Actual Bandwidth - 30.2 Gbps
Test 3
1 Link VM 8vCPU - Potential Egress 16 Gbps 1 Client VM 16vCPU Multi-Stream Client 1 Server VM 16vCPU
Potential Bandwidth - 16 Gbps Actual Bandwidth - 14.3 Gbps
Test 4
1 Link VM 8vCPU - Potential Egress 16 Gbps 1 Client VM 16 vCPU Single-Stream Client 1 Server VM 16 vCPU
Potential Bandwidth - 16 Gbps Actual Bandwidth - 13.4 Gbps
Red Hat Enterprise Linux Network Performance Tuning Guide provides detailed information on tuning network interfaces. It focuses on TCP, which is not relevant to the stateless IP routing nature of the vpc-link router instances, but it is full of useful information, like detecting dropped packets.