search

PaloAltoNetworks / lab-aws-gwlb-vmseries

Materials for PS Regional Training AWS lab
MIT License
24 stars 30 forks source link

readme

1. Warning

This repository and lab guide are intended to be used with a specific QwikLabs scenario, and some steps are specific to Qwiklabs. It contains deployment code and a lab guide for learning GWLB traffic flows with VM-Series. Some configurations and resources are intentionally omitted to be left as troubleshooting exercises.

Do not use this for a production deployment or an easy demo environment!

You can find the regularly maintained terraform modules and example deployments for your production deployments at pan.dev. These modules are also published on the HashiCorp Registry.

The Combined Design example closely matches the design from this learning lab.

2. VM-Series on AWS Gateway Load Balancer Lab

Manual Last Updated: 2024-07-18
Lab Last Tested: 2024-07-11

2.1. Overview

This lab will involve deploying Palo Alto Networks VM-Series in AWS with a Gateway Load Balancer (GWLB) topology. You will first deploy a Panorama and then the rest of the AWS resources and VM-Series with bootstrapping. It is a challenge lab that will require you to troubleshoot and fix some issues.

2.2. Lab Guide Syntax conventions

ℹ Items with info icon are additional context or details around actions performed in the lab

❓ Items with question mark icon are questions that should be answered on the quiz

Use the corresponding quiz that relates to the questions you will see inside this guide to test your knowledge!

2.3. Table of Contents

3. Lab Topology

GWLB Topology

3.1. Flow Diagrams

Reference these diagrams for a visual of traffic flows through this topology.

3.1.1. Outbound Traffic Flows

3.1.2. Inbound Traffic Flows

4. Lab Steps

4.1. Initialize Lab

4.1.1. Find SSH Key Pair Name

ℹ Any EC2 Instance must be associated with a SSH key pair, which is the default method of initial interactive login to EC2 instances. With successful bootstrapping, there should not be any need to connect to the VM-Series instances directly with this key, but it is usually good to keep this key securely stored for any emergency backdoor access.

For this lab, we will use the key pair automatically generated by Qwiklabs. The key will also be used for connecting to the test web server instances.

4.2. Update IAM Policies

ℹ Qwiklabs has an explicit Deny for some actions we need to use for this lab. However, we have permission to remove this policy. Take a look at the other Deny statements while you are here.

ℹ It is important to be familiar with IAM concepts for VM-Series deployments. Several features (such as bootstrap, custom metrics, cloudwatch logs, HA, and VM Monitoring) require IAM permissions. You also need to consider IAM permissions in order to deploy with IaC or if using lambda for custom automation.

4.3. Check Marketplace Subscriptions

ℹ Before you can launch VM-Series or Panorama images in an account, the account must first have accepted the Marketplace License agreement for that product.

ℹ The QwikLabs accounts should already be subscribed to these offers, but we will need to verify and correct if required.

4.4. Launch CloudShell

ℹ This lab will use Cloudshell for access to AWS CLI and as a runtime environment to provision your lab resources in AWS using Terraform. Cloudshell will have the same IAM role as your authenticated user and has some utilities (git, aws cli, etc) pre-installed. It is only available in limited regions currently.

Anything saved in the home directory /home/cloudshell-user will remain persistent if you close and relaunch CloudShell

4.5. Search Available VM-Series Images (AMIs)

ℹ We will use us-west-2 for example of using this search and answering the questions, but your actual deployment for this lab may be in a different region.

aws ec2 describe-images --filters "Name=owner-alias,Values=aws-marketplace" --filters Name=name,Values=PA-VM-AWS-11* Name=product-code,Values=6njl1pau431dv1qxipg63mvah --region us-west-2
aws ec2 describe-images --filters "Name=owner-alias,Values=aws-marketplace" --filters Name=name,Values=PA-VM-AWS-11* Name=product-code,Values=6njl1pau431dv1qxipg63mvah --region us-west-2 --query 'Images[].[ImageId,Name]'

❓ What is the BYOL Marketplace AMI ID for 10.1.8 in the us-east-1 region?

❓ What are some options if there is no AMI available for your targeted version?

ℹ This terraform deployment will look up the AMI ID to use for the deployment based on the variable fw_version. New AMIs are not always published for each minor release. Therefore, it is a good idea to verify what version AMI most closely matches your target version.

ℹ product-code is a global value that correlates with Palo Alto Networks marketplace offerings This is global and the same across all regions.

  "byol"  = "6njl1pau431dv1qxipg63mvah"
  "payg1" = "e9yfvyj3uag5uo5j2hjikv74n"
  "payg2" = "hd44w1chf26uv4p52cdynb2o"

byol will be the most common. You will need to obtain Software Firewall Flex Credits to license these after deployment.

The other image types are "Pay as you go" and come pre-licensed and are billed by AWS. These will generally be more costly to run over long periods than BYOL. PAYG is good for certain scenarios such as autoscaling or when you need to add capacity for a short period.

payg1 is VM-Series Next-Gen Virtual Firewall w/Advanced Threat Prevention (PAYG) in the marketplace and only has advanced threat subscription.

payg2 is VM-Series Next-Gen Virtual Firewall w/ Advanced Security Subs (PAYG) in the marketplace and has additional security subscriptions enabled (Adv URL, Adv Wildfire, DNS, GlobalProtect)

You can also use the EC2 web console to search for available images based on product code or AMI name.

ℹ The name tag of the image should be standard and can be used for the filter. For example PA-VM-AWS-10.1*, PA-VM-AWS-9.1.3*, PA-VM-AWS-10*. This is the same logic the terraform will use to lookup the AMI based on the fw_version variable.

ℹ Not needed for this lab, but when deploying VM-Series from EC2 console, it will default to the latest version. You can instead go to the AWS Marketplace to subscribe to the offering and select previous versions to deploy the desired AMI

4.6. Clone Repo and Install Terraform

rm -rf ~/bin && rm -rf ~/lab-aws-gwlb-vmseries/
cd ~ && git clone https://github.com/PaloAltoNetworks/lab-aws-gwlb-vmseries.git && chmod +x ~/lab-aws-gwlb-vmseries/terraform/install_terraform.sh && ~/lab-aws-gwlb-vmseries/terraform/install_terraform.sh && source ~/.bashrc

alt text

ℹ Terraform projects often have version constraints in the code to protect against potentially breaking syntax changes when new version is released. For this project, the version constraint is:

terraform {
 required_version = ">=0.12.29, <2.0"
}

Terraform is distributed as a single binary so isn't usually managed by OS package managers. It simply needs to be downloaded and put into a system $PATH location. For Cloudshell, we are using the /home/cloud-shell-user/bin/ so it will be persistent if the sessions times out.

4.7. Deploy Panorama and TGW Infrastructure with Terraform

ℹ One of the complications with fully automating VM-Series deployments is the need to have a Panorama already running and configured for the VM-Series to bootstrap to. There isn't a built in bootstrap method for Panorama itself. There are ways to orchestrate it, but it is complex. In addition the Panorama takes a long time to initialize. So typically you need to bring up the Panorama as a separate terraform deployment before launching the VM-Series.

cd ~/lab-aws-gwlb-vmseries/terraform/panorama
terraform init
terraform apply

ℹ You can also come back to this directory in CloudShell later and run terraform output to view this information

❓ What AWS resources were created from this terraform execution?

4.8. Prepare Panorama

ℹ The Panorama was deployed from a partially prepped image that is licensed and has a baseline Template/Device Group. However, additional steps will be completed here to prepare the Panorama for logging and bootstrapping.

ℹ Auto Deactive is handy for automating the cleanup of devices that have been terminated, for example in an autoscaling VM-Series deployment. Caution should be used as there is a possibility for false positives. For example, if there is an unrelated connectivity issue, Panorama could perceive that the devices are no longer active and initiate the deactivation based on the timer. This plugin can also be used for manually deactivating devices, or creating an event-driven workflow to make an API call to Panorama for deactivation. For any deactivation functions to work, the Panorama must be configured with a Licensing API Key that is generated from the Customer Support Portal.

ℹ This feature was added in 10.2 and is very useful for automated deployments, especially autoscaling. It ensures that as devices bootstrap, the Panorama will push down the current dynamic content before committing the configuration.

4.9. Update Deployment Values in tfvars for VM-series

Most of the bootstrap parameters and environment-specific variable values have already been prepped in the Terraform code for this deployment. You will only need to update the auth-key value for your Panorama.

cd ~/lab-aws-gwlb-vmseries/terraform/vmseries
auth-key        = "_AQ__xxxxxxxxxxxxxxxxxxx"
Expand Me For Specific Steps --- - ( Option 1 ) Use vi to update values in `student.auto.tfvars` ``` vi student.auto.tfvars ``` --- - ( Option 2 ) If you don't like vi, you can install nano editor: ``` sudo yum install -y nano nano student.auto.tfvars ``` --- 
cat ~/lab-aws-gwlb-vmseries/terraform/vmseries/student.auto.tfvars

ℹ This deployment is using a basic bootstrapping that does not require S3 buckets. Any parameters normally specified in init-cfg can now be passed directly to the instance via UserData. Prerequisite is the image you are deploying has plugin 2.0.1+ installed

❓ What are some bootstrap options that won't be possible with this basic bootstrap method?

ℹ If you have time left after the rest of the lab activities, later steps will return to do some more digging into the terraform code.

4.10. Apply Terraform

cd ~/lab-aws-gwlb-vmseries/terraform/vmseries
terraform init
terraform apply

ℹ You can also come back to this directory in CloudShell later and run terraform output to view this information

4.11. Inspect deployed resources

All resources are now created in AWS, but it will be around 10 minutes until VM-Series are fully initialized and bootstrapped.

In the meantime, let's go look at what you built!

❓ What are some tradeoffs of using the user-data method for bootstrap vs S3 bucket?

❓ What needs to happen if you have a typo or missed a value for bootstrap when you deployed?

4.11.1. Get VM-Series instance screenshot

ℹ This can be useful to get a view of the console during launch. It is not interactive and must be manually refreshed, but you can at least see some output related to the bootstrap process or to troubleshoot if the VM-Series isn't booting properly or is in maintenance mode.

4.11.2. Check VM-Series instance details

❓ What is the instance type?

❓ How many interfaces are associated to the VM-Series?

❓ Which interface is the default ENI for the instance?

❓ Which interface has a public IP associated?

❓ Check the security group associated with the "data" interface. What is allowed inbound? What is the logic of this SG?

❓ Review the Instance Profile (IAM Role) the VM-Series launched with. What actions does it allow?

❓ What are some other use-cases where you need to allow additional IAM permissions for the VM-Series instance profile?

4.11.3. Check cloudwatch bootstrap logs

ℹ It is normal for the VMs to briefly lose connectivity to Panorama after first joining.

ℹ This feature is only implemented for AWS currently.

❓ What is required to enable these logs during the boot process?

4.12. Verify Bootstrap in Panorama

ℹ For this lab, the Panorama and VM-Series mgmt are publicly accessible. For production deployments, management should not be exposed to inbound Internet traffic as a general practice. If public inbound management access is required, make sure to use other controls (MFA, AWS security groups, PAN-OS permitted-ip lists).

❓ Why are NAT policies not needed for GWLB model?

4.13. Access VM-Series Management

vmseries_eips = {
  "vmseries01-mgmt" = "54.71.121.124"
  "vmseries02-mgmt" = "44.237.145.237"
}

❓ Why don't you have to use an SSH key pair to authenticate to these VM-Series?

4.14. Check bootstrap logs

ℹ It is common to have issues when initially setting up an environment for bootstrapping. Thus it is good to know how to troubleshoot. When using the new basic bootstrapping with user-data, there is less potential for problems.

Some things to keep in mind:

  • Default AWS ENI needs access to reach S3 bucket as well as path to Internet for licensing
    • S3 access can be via internet or with a VPC endpoint
    • IAM Instance Profile will need permissions to access S3 bucket
  • When using interface swap, the subnet for the second ENI will also need a path to S3 and Internet
    • Interface swap can be done with user-data or in init-cfg parameters.
    • Generally better to do via user-data
  • Template Stack and Device Group names must match exactly or they will never join Panorama (no indication of this in Panorama logs)
  • If there are any issues with licensing, VM-Series will not join Panorama (no indication of this in Panorama logs)
show system bootstrap status
debug logview component bts_details

ℹ If you have Cloudwatch logs enabled, you can see most of this status without SSH session to VM-Series.

4.15. Fix GWLB Health Probes

❓ What Protocol and Port is the Target Group Health Probe configured to use?

❓ What Protocol and Port is the GWLB listening on and forwarding to the VM-Series instances?

ℹ You can check traffic logs either in Panorama or local device

ℹ Reboot your VM-Series if you do not see any traffic logs!

  • Check Traffic Logs for Health Probes
  • In Panorama UI -> Monitor -> Traffic
  • Analyze the traffic logs for the port 80 traffic
  • Enable Columns to view Bytes Sent and Bytes Received
  • Notice that the sessions matching allow policy for student-gwlb-any policy but aging out

❓ Why are there two different source addresses in the traffic logs for these GWLB Health Probes?

Expand Me For Specific Steps > ℹ We can see in the traffic logs that the health probes are being received. So we know they are being permitted by the AWS Security Group. They are being permitted by catch-all security policy but there is no return traffic (Notice 0 Bytes Received in the traffic logs). This indicates the VM-Series dataplane interface is not listening - Create and add Interface Management profile to eth1/1 - In Panorama select Network Tab -> Template `tpl-aws-gwlb-lab` - Create an Interface Management Profile - Name: `gwlb-health-probe` - Services: HTTP - Permitted IP addresses: `10.100.0.16/28`, `10.100.1.16/28` - Select Interfaces -> ethernet1/1 -> Advanced -> Management Profile `gwlb-health-probe` - Create a specific security policy for these health probes to keep logging clean - In Panorama select Policies Tab -> Device Group `AWS-GWLB-LAB` - Create a new security policy - Name: `gwlb-health-probe` - Source Zone: `gwlb` - Source Addresses: `10.100.0.16/28`, `10.100.1.16/28` (Can use predefined address objects) - Dest Zone: `gwlb` - Dest Addresses: `10.100.0.16/28`, `10.100.1.16/28` (Can use predefined address objects) - Application: `Any` - Serivce: `service-http` - Make sure the new policy is before the existing catch-all `student-gwlb-any` policy - Commit and Push your changes

❓ Why is the application still detected as incomplete?

4.16. Inbound Traffic Flows to App Spoke VPCs

ℹ For the GWLB Distributed model, Inbound traffic for public services comes directly into the App Spoke Internet Gateway. The Ingress VPC route table directs traffic to the GWLB Endpoints in the App Spoke VPC.

Application owners can provision their external facing resources in their VPC (EIP, Public NLB / ALB, etc), but all traffic will be forwarded to Security VPC (via GWLB endpoint) for inspection prior to reaching the resource.

This inbound traffic flow uses AWS Private Link technology and does not involve the Transit Gateway at all.

4.16.1. Update Spoke1 App VPC networking for Inbound inspection with GWLB

Expand For Specific Steps Starting left to right on the diagram... - VPC Dashboard -> Filter by VPC -> `spoke1-app-vpc` - Route Tables -> `spoke1-vpc-igw-edge` -> Routes Tab (bottom panel) - Add Route (spoke1-vpc-alb1 subnet CIDR to spoke1-gwlbe1) - CIDR: 10.200.0.16/28 - Target: Gateway Load Balancer Endpoint (ID of spoke1-vpc-inbound-gwlb-endpoint1 GWLBE) - Add Route (spoke1-vpc-alb2 subnet CIDR to spoke2-gwlbe2) - CIDR: 10.200.1.16/28 - Target: Gateway Load Balancer Endpoint (ID of spoke1-vpc-inbound-gwlb-endpoint2 GWLBE) - Save Routes --- - Route Tables -> `spoke1-vpc-gwlbe1` -> Routes Tab (bottom panel) - Add Route (default route to spoke1-IGW) - CIDR: 0.0.0.0/0 - Target: Internet Gateway (only one per VPC) - Save Routes --- - Route Tables -> `spoke1-vpc-gwlbe2` -> Routes Tab (bottom panel) - Add Route (default route to spoke1-IGW) - CIDR: 0.0.0.0/0 - Target: Internet Gateway (only one per VPC) - Save Routes --- - Route Tables -> `spoke1-vpc-alb1` -> Routes Tab (bottom panel) - Add Route (default route to spoke1-gwlbe1) - CIDR: 0.0.0.0/0 - Target: Gateway Load Balancer Endpoint (ID of spoke1-vpc-inbound-gwlb-endpoint1 GWLBE) - Save Routes --- - Route Tables -> `spoke1-vpc-alb2` -> Routes Tab (bottom panel) - Add Route (default route to spoke1-gwlbe2) - CIDR: 0.0.0.0/0 - Target: Gateway Load Balancer Endpoint (ID of spoke1-vpc-inbound-gwlb-endpoint2 GWLBE) - Save Routes ---

4.16.2. Update Spoke2 App VPC networking for Inbound inspection with GWLB

Expand For Specific Steps Only `spoke2-vpc-igw-edge` Route Table is missing routes for App2 Spoke Starting left to right on the diagram... - VPC Dashboard -> Filter by VPC -> `spoke2-app-vpc` - Route Tables -> `spoke2-vpc-igw-edge` -> Routes Tab (bottom panel) - Add Route (spoke2-vpc-alb1 subnet CIDR to spoke2-gwlbe1) - CIDR: 10.250.0.16/28 - Target: Gateway Load Balancer Endpoint (ID of spoke2-vpc-inbound-gwlb-endpoint1 GWLBE) - Add Route (spoke2-vpc-alb2 subnet CIDR to spoke2-gwlbe2) - CIDR: 10.250.1.16/28 - Target: Gateway Load Balancer Endpoint (ID of spoke2-vpc-inbound-gwlb-endpoint2 GWLBE) - Save Routes

4.16.3. Test Inbound Traffic to Spoke Web Apps

Generate some HTTP traffic to the web apps using the DNS name of the Public NLB that is in front of the web compute instances. URL will be the FQDN of the NLBs from the terraform output

ℹ The inbound routing should now be in place, but you will not get a response yet as the instances are not yet running a web server.

ℹ The web instances are configured to update and install the web server automatically with a user-data script, but they first must have a working outbound path to the Internet to retrieve packages.

4.16.4. Test Outbound Traffic from Spoke1 Instance

Access the spoke web servers console using the AWS Systems Manager connect

ping 8.8.8.8

curl http://ifconfig.me

ℹ Session Manager relies on a package being installed in the OS that makes an outbound connection to the AWS SSM service. We do not have outbound internet currently, but there is a private endpoint for the SSM service configured. It is not possible to use SSM for CLI access to VM-Series, as it does not have the package installed

4.16.5. Check Inbound Traffic Logs

❓ Why does VM-Series see the private NLB addresses as the destination instead of the public address?

4.16.6. Check Outbound Traffic Logs

ℹ Since outbound traffic was not working earlier, let's check to see if it is making it to VM-Series.

ℹ We now have visibility and control for inbound connectivity but instances do not yet have a path outbound.

4.17. Outbound and East / West (OBEW) Traffic Flows

4.17.1. Update Spoke1 VPC for OB/EW routing with TGW

Expand For Specific Steps - VPC Dashboard -> Filter by VPC -> `spoke1-app-vpc` - Route Tables -> `spoke1-vpc-web1` -> Routes Tab (bottom panel) - Add Route (default route to TGW) - CIDR: 0.0.0.0/0 - Target: Transit Gateway (only one available) - Save Routes --- - Route Tables -> `spoke1-vpc-web2` -> Routes Tab (bottom panel) - Add Route (default route to TGW) - CIDR: 0.0.0.0/0 - Target: Transit Gateway (only one available) - Save Routes

4.17.2. Update Spoke2 VPC for OB/EW routing with TGW

Expand For Specific Steps - Nothing is missing for `spoke2-app-vpc`! Web1 and Web2 Route Tables already have routes to TGW.

4.17.3. Update Transit Gateway (TGW) Route Tables

ℹ For GWLB Centralized Outbound, the TGW routing for Outbound and EastWest (OB/EW) traffic is the same as previous TGW models. Spokes send all traffic to TGW. Spoke TGW RT directs all traffic to Security VPC. Security TGW RT has routes to reach all spoke VPCs for return traffic.

For OB/EW flows, the GWLB doesn't come into play until traffic comes into the Security VPC from TGW before being forwarded to a GWLB endpoint.

Expand For Specific Steps - VPC Dashboard -> Transit Gateway Route Tables -> Select `from-spoke-vpcs` - Check `Associations` tab and verify the two spoke App VPCs are associated - Check Routes tab and notice there is no default route - Create Static Route (Default to security VPC) - CIDR: 0.0.0.0/0 - Attachment: Security VPC (Name Tag = security-vpc) - VPC Dashboard -> Transit Gateway Route Tables -> Select `from-security-vpc` - Check `Associations` tab and verify the security VPC is associated - Check `Routes` tab and notice there are no existing routes to reach the spoke VPCs - Select Propagations Tab -> Create Propagation - Select attachment with Name Tag `spoke1-vpc` - Repeat for attachment with Name Tag `spoke2-vpc` - Return to `Routes` tab and verify the table now has routes to reach the App VPCs (may need to refresh)

❓ What needs to be done on the TGW route tables in order to bring additional Spoke VPCs online for OB/EW traffic?

4.17.4. Update Security VPC networking for OB/EW with GWLB

ℹ The routing and traffic flows can be tricky to grasp, especially when designing for multiple availability zones. For this lab, we are using separate endpoints for Outbound VS EastWest, plus separate endpoints per AZ. Take your time and understand the traffic flows as you configure the routing.

Multi-AZ GWLB generally requires unique route tables per subnet in both AZs in order to direct traffic in that AZ toward the resources (GWLB endpoint, NAT GW) in the same AZ.

Expand For Specific Steps - VPC Dashboard -> Filter by VPC in left menu -> Select `security-vpc` - Now when checking other sections in this dashboard (route tables, subnets, etc) will be filtered to Security VPC - Select Route Tables section - Look through the route tables. Most have no routes! - Identify Security VPC in the Topology Diagram and start from left-to-right **TGW Attach Route Tables** > ℹ Here we are defining what will happen for any traffic coming in to the Security VPC from the TGW. > If traffic is destined for private summary range, we will send it to the EastWest GWLB Endpoints > > If traffic is destined for anything else (default route), we will send it to the Outbound GWLB Endpoint > > For AZ resilience, we also make sure that whatever AZ traffic comes in from TGW it is forwarded to the GWLB endpoint in the same AZ > > Any traffic sent to these endpoints will go directly to GWLB / VM-Series as a "bump-in-the-wire" and return to the corresponding endpoint - Select Route Table `security-vpc-tgw-attach1` and edit routes - 0.0.0.0/0 -> Gateway Load Balancer Endpoint `outbound1` - Select vpce Endpoint ID `outbound1` from terraform output - 10.0.0.0/8 -> Gateway Load Balancer Endpoint - Select vpce Endpoint ID `eastwest1` from terraform output - Select Route Table `ps-lab-tgw-attach2` and edit routes - 0.0.0.0/0 -> Gateway Load Balancer Endpoint `outbound2` - Select vpce Endpoint ID `outbound2` from terraform output - 10.0.0.0/8 -> Gateway Load Balancer Endpoint - Select vpce Endpoint ID `eastwest2` from terraform output --- **GWLB Endpoint East/West Route Tables** > ℹ Here we are defining what will happen for East / West traffic returning from the GWLB / VM-Series. > This means traffic has already passed policy/inspection and we now define the direct path to reach the destination. > > For East / West traffic, this means sending all internal traffic to the TGW where it will then be forwarded directly back to the spoke VPC. > > Note that Backhaul traffic (VPN / DirectConnect attachments to TGW) usually follows this same pattern. - Select Route Table `security-vpc-gwlbe-eastwest-1` and edit routes - 10.0.0.0/8 -> Transit Gateway -> Select TGW ID - Select Route Table `security-vpc-gwlbe-eastwest-2` and edit routes - 10.0.0.0/8 -> Transit Gateway -> Select TGW ID > For this traffic, routes are identical for both AZs, so doesn't strictly require separate route tables. We maintain the separation only for consistency and clarity. --- **GWLB Endpoint Outbound Route Tables** > ℹ Here we are defining what will happen for Outbound traffic returning from the GWLB / VM-Series. > This means traffic has already passed policy/inspection and we now define the direct path to reach the destination. > > For Outbound traffic, this means sending all traffic toward the NAT GW in the corresponding AZ. > > We also need to consider the return traffic from the Internet / NAT GW which uses the same GWLB Endpoint. For this, we define a path to reach the Spoke VPC summary directly via TGW - Select Route Table `security-vpc-gwlbe-outbound-1` and edit routes - 0.0.0.0/0 -> NAT Gateway -> Select NAT GW ID for AZ1 - 10.0.0.0/8 -> Transit Gateway -> Select TGW ID - Select Route Table `security-vpc-gwlbe-outbound-2` and edit routes - 0.0.0.0/0 -> NAT Gateway -> Select NAT GW ID for AZ2 - 10.0.0.0/8 -> Transit Gateway -> Select TGW ID **NAT Gateway Route Tables** > ℹ Here we are defining what will happen for return traffic from the Internet for traffic that was initiated outbound. > From the perspective of the NAT GW routing, we direct the Spoke VPC summary back to the Outbound Endpoint of the respective AZ so the return traffic will be statefully forwarded to GWLB / VM-Series. > > Note: The NAT GW subnet is already provisioned with a default route pointed to IGW - Select Route Table `security-vpc-natgw1` and edit routes - 10.0.0.0/8 -> Gateway Load Balancer Endpoint `outbound1` - Select vpce Endpoint ID `outbound1` from terraform output - Select Route Table `security-vpc-natgw2` and edit routes - 10.0.0.0/8 -> Gateway Load Balancer Endpoint `outbound2` - Select vpce Endpoint ID `outbound2` from terraform output

4.18. Test Traffic Flows

At this point, all routing should be in place for GWLB topology. Now we will verify traffic flows and check the logs.

ℹ Note that web instances in Spoke VPCs are configured to update and install the web server automatically, now that you have provided an outbound path, this should have been completed.

4.18.1. Test Outbound Traffic from Spoke1 Instances

ping 8.8.8.8

curl http://ifconfig.me

ℹ ifconfig.me is a service that just returns your client's public IP (like google "what is my ip" or ipmonkey.com)

❓ What AWS resources have the public IPs you are egressing from?

4.18.2. Test Inbound Web Traffic to Spoke1 and Spoke2 Apps

Expand For Specific Steps If your NLB is not responding and you see traffic in the Panorama logs, it is possible the script to install the web server didn't execute - From the Session Manager, verify if you can connect locally - curl http://localhost - If not, check the user data of the instance to see the bash script that was configured and run it manually - If the web service is responding locally, there is likely an issue in the spoke VPC routing

ℹ Local IP and VM Name in the response will show you which VM you are connected to behind the NLB. Session persistence may keep you pinned to specific instances

4.18.3. Test E/W Traffic from Spoke1 Instance to Spoke2 Instance

ping 10.250.0.x

curl http://10.250.0.x

ℹ Backhaul traffic flows (VPN or Direct Connect Attachments to TGW) will follow this same general traffic flow as E/W between VPCs.

4.19. GWLBE / Sub-Interface associations

Now we have verified inbound, outbound, and east/west traffic flows. We have full visibility of this traffic but as you can see in the logs, everything is wide open!

Since all traffic to GWLB comes in and out of VM-Series on the same interface and zone, it is tricky to create and manage effective security policies specific to traffic flow directions.

We will now fix this using GWLB sub-interface associations.

ℹ Since each GLWB endpoint can be associated with a specific sub-interface, each endpoint can have a separate zone.

ℹ This does not change the overall concept that all traffic from GWLB is an encapsulated bump in the wire and will ingress and egress the same sub-interface. In the default behavior, there is no routing between zones.

4.19.1. Configure Zones in Panorama

4.19.2. Configure Sub-Interfaces in Panorama

ℹ No IP configurations are actually used by these sub-interfaces, but they should be set to use DCHP to function properly

ℹ Unique VLAN Tags must be specified but are not actually used for GWLB GENEVE traffic. The VLAN concept is being repurposed to identify sub-interfaces based on the endpoint IDs.

4.19.3. Create associations from GWLB Endpoints

ℹ Now since we have sub-interfaces and zones, we can associate specific endpoints to sub-interfaces.

ℹ For dual-az deployments, there will be two endpoints for each traffic flow. Both will be associated to the same sub-interface.

request plugins vm_series aws gwlb associate interface ethernet1/1.10 vpc-endpoint ${outbound1}
request plugins vm_series aws gwlb associate interface ethernet1/1.10 vpc-endpoint ${outbound2}
request plugins vm_series aws gwlb associate interface ethernet1/1.11 vpc-endpoint ${east-west1}
request plugins vm_series aws gwlb associate interface ethernet1/1.11 vpc-endpoint ${east-west2}
request plugins vm_series aws gwlb associate interface ethernet1/1.12 vpc-endpoint ${spoke1-inbound1}
request plugins vm_series aws gwlb associate interface ethernet1/1.12 vpc-endpoint ${spoke1-inbound2}
request plugins vm_series aws gwlb associate interface ethernet1/1.13 vpc-endpoint ${spoke2-inbound1}
request plugins vm_series aws gwlb associate interface ethernet1/1.13 vpc-endpoint ${spoke2-inbound2}

show plugins vm_series aws gwlb

ℹ Typically, the GWLB endpoint associations will be set as bootstrap parameters. These are plugin request commands and not stored in XML config and cannot be pushed from Panorama.

4.19.4. Create Zone-Based policies for sub-interfaces

ℹ Now that each endpoint is associated with a specific zone, we can have more logic with our security policies. Important to understand that although we have associated each endpoint to separate sub-interfaces, all of the traffic flows are still intra-zone. VM-Series will return traffic back to the same endpoint that it came in from.

ℹ If everything is configured correctly, only the Health Probes should be hitting the gwlb zone on the eth1 interface. The health probes are sent from the GWLB directly to the VM-Series ENI, not through an endpoint or GENEVE.

4.20. Overlay Routing

Overlay Routing enables the VM-Series to strip off the GENEVE encapsulation and use standard routing behavior to determine the next hop. Most commonly this is used for outbound Internet traffic. When the return traffic is received by VM-Series, it will be re-encapsulated and sent to the same endpoint where the session originated.

Overlay allows the ability to use familiar inside -> outside zone-based policies. With overlay enabled, you typically do not need separate endpoints (and sub-interface associations) for outbound and eastwest.

We will update our existing infrastructure to use overlay routing. On the updated diagram, you will see the changes on the right side where the NAT Gateways are replaced and instead, an additional public interface and EIPs are attached to the VM-series.

GWLB Topology

4.20.1. Create Public Interfaces for VM-Series

4.20.2. Associate the EIPs to the Public Interfaces

4.20.3. Configure Networking and Policies in Panorama

ℹ Overlay routing presents a problem when using multiple AZs. Since the next hop is different for each AZ, we can no longer use identical configurations for all VMs. We can accept the gateway from DHCP for the public interface, but need to create a static route for return traffic to the inside. There are several methods to handle this, for this manual deployment using device-specific variables is arguably the cleanest approach.

eth1-2-overlay

variable-next-hop

nat-policy

4.20.4. Enable Overlay mode

request plugins vm_series aws gwlb overlay-routing enable yes

request restart system

4.20.5. Test Outbound Traffic

yum update

curl http://ifconfig.me

❓ What AWS resources now have the public IPs you are egressing from?

4.21. Review Lab Quiz Questions

Submit your answers for the Lab Questions found throughout this guide.

4.22. Finished

Congratulations!