Terraform Highly Available ICP Deployment on AWS

This Terraform configurations uses the AWS provider to provision virtual machines on AWS to prepare VMs and deploy IBM Cloud Private on them. This Terraform template automates best practices learned from installing ICP on AWS at numerous client sites in production.

This template (on the master branch) provisions a highly-available cluster with ICP 3.1.2 Enterprise Edition. The template uses the terraform-module-icp-deploy sub module to install the cluster once the infrastructure has been prepared.

• Infrastructure Architecture
• Terraform Automation
• Installation Procedure
• Community Edition
• Cluster access
• AWS Cloud Provider

Infrastructure Architecture

The following diagram outlines the infrastructure architecture.

In a single availability zone, we divide the network into a public subnet which is directly connected to the internet, and a private subnet that can reach the internet through the NAT gateway:

Terraform Automation

Prerequisites

1. To use Terraform automation, download the Terraform binaries here.

   On MacOS, you can acquire it using homebrew using this command:

   brew install terraform

2. Create an S3 bucket in the same region that the ICP cluster will be created and upload the ICP binaries. Make note of the bucket name. You can use the AWS CLI to do this.

   For ICP 3.1.2-EE, you will need to copy the following:

   • the ICP binary package tarball (ibm-cloud-private-x86_64-3.1.2.tar)
   • ICP Docker package (icp-docker-18.03.1_x86_64)

3. Create a file, terraform.tfvars containing the values for the following:

See Terraform documentation for the format of this file.

5. If using a user-provided TLS certificate containing a custom DNS name, copy icp-auth.crt and icp-auth.key to this directory before installation to the cfc-certs directory. See documentation for more details. The certificate should contain the user_provided_cert_dns as a common name, and the DNS entry corresponding should be a CNAME pointing at the created ELB DNS entry for the master console.

6. Provide AWS credentials using environment variables:

   export AWS_ACCESS_KEY_ID=AKIAADGHASKDHGAKSDHGKASDHGK
   export AWS_SECRET_ACCESS_KEY=BAzxcvq^.asdgaljlajdfl235bads

7. Initialize Terraform using this command. This will download all dependent modules.

   terraform init

Run the Terraform Automation

Run this command to see what would be created in the AWS account:

terraform plan

To move forward and create the objects, use the following command:

terraform apply

The following diagram illustrates the process:

1. Terraform creates the infrastructure objects including EC2 instances.
2. The scripts in the scripts directory will be uploaded to an S3 bucket
3. The EC2 instances are configured with cloud-init to retrieve the scripts from the S3 bucket on startup. The bootstrap.sh script is executed silently on every node and bootstraps each node (install docker, prepare storage, etc).
4. A configuration file (terraform.tfvars) is generated from the outputs of the infrastructure for the terraform-module-icp-deploy module and copied to the S3 bucket.
5. The start_install.sh script is run on the first ICP master host, which clones the github module, downloads the terraform.tfvars file from the S3 bucket, and runs terraform apply in a docker container that triggers the rest of the ICP installation.

If no bastion host is provisioned, the installation runs silently on the boot master (i.e. icp-master01) using cloud-init until it completes; otherwise the installation will continue synchronously using the bastion host's public IP (by setting the number of bastion nodes in terraform.tfvars to 1).

bastion = {
 nodes = "1"
}

The installation output will be written to /tmp/icp_logs/start_install.log on the first master. Bastion hosts are not required for normal operation of the cluster, but may be desired if you want to synchronously wait for the installation to complete, such as in execution from a devops pipeline.

When the installation completes, the /opt/ibm/cluster directory on the boot master (i.e. icp-master01) is backed up to S3 in a bucket named icpbackup-<clusterid>, which can be used in master recovery in case one of the master nodes fails. It is recommended after every time terraform apply is performed, to commit the terraform.tfstate into a backend so that the state is stored in source control.

When installation completes, if a user-provided certificate is used, create a CNAME entry in your DNS provider from the DNS entry to the ELB DNS URL output at the end of the terraform process.

Terraform objects

The Terraform automation creates the following objects.

EC2 Instances

these are tagged with the cluster id for Kubernetes-AWS integration

(the instance types, base AMIs and counts can be configured in variables.tf)

Elastic Network Interfaces

For recovery, master and proxy nodes have Network Interfaces created and attached to them as the first network device. The private IP address is bound to the network interface, so when the interface is attached to a newly created instance, the IP address is preserved. This is useful for Master Recovery, which is covered in below.

IAM Configuration

An IAM role is created in AWS and attached to each EC2 instance. See AWS Cloud provider documentation for minimum permissions to enable the AWS cloud provider.

(The Kubernetes AWS Cloud Provider needs access to read information from the AWS API about the instance (i.e. which subnet it's in, the private DNS name, whether nodes that have been removed, etc), and to create LoadBalancers and EBS Volumes on demand.)

Additionally, we add S3FullAccess policy so that the IAM role can get installation images out of an S3 bucket and back up the /opt/ibm/cluster directory to an S3 bucket after installation.

VPC

• VPC with an internet gateway
• All ICP nodes are placed in private subnets, each with their own NAT Gateway.
  • outbound Internet access from private subnet through NAT Gateway
• Private EC2 and S3 API endpoints are created in the VPC

Subnets

• two for each AZ (one public, one private)
  • (CIDR for VPC and subnets can be configured in terraform.tfvars, see variables.tf for values)
  • (these are tagged with the cluster id for Kubernetes ELB integration)

Security Group

Note that the below are the defaults, and each security group can have its whitelist be configured in terraform.tfvars.

• icp-bastion
  • allow 22 from 0.0.0.0/0
• icp-default
  • allow ALL traffic from itself (all nodes are in this security group)
  • (this is tagged with the cluster id for Kubernetes ELB integration)
• icp-proxy
  • allow from 0.0.0.0/0 on port 80
  • allow from 0.0.0.0/0 on port 443
• icp-master
  • allow from 0.0.0.0/0 on port 8500 (image registry)
  • allow from 0.0.0.0/0 on port 8600 (image registry)
  • allow from 0.0.0.0/0 on port 8001 (kube api)
  • allow from 0.0.0.0/0 on port 8443 (master UI)
  • allow from internal on port 9443 (Auth service)

Load Balancer

Note that the Network Load Balancer (NLB) used in the terraform template does not have explicit security groups like Application Load Balancers so the security groups are instead placed on the instances themselves.

• Network LoadBalancer for ICP console
  • listen on 8443, forward to master nodes port 8443 (ICP Console)
  • listen on 8001, forward to master nodes port 8001 (Kubernetes API)
  • listen on 8500, forward to master nodes port 8500 (Image registry)
  • listen on 8600, forward to master nodes port 8600 (Image registry)
  • listen on 9443, forward to master nodes port 9443 (Auth service)
• Network LoadBalancer for ICP Ingress resources
  • listen on port 80, forward to proxy nodes port 80 (http)
  • listen on port 443, forward to proxy nodes port 443 (https)

Route53 DNS Zone

A private DNS Zone is created in Route 53. The domain name can be configured in variables.tf; by default it is <clusterid>.icpcluster.icp. The domain search suffixes are added to resolv.conf, but due to a bug in cloud-init, resolv.conf is overwritten by NetworkManager in RHEL. It should be resolved in a future release of cloud-init.

S3 Bucket

An S3 Bucket for Configuration is created and the /opt/ibm/cluster is uploaded after the cluster is installed. This is not deleted after an terraform destroy.

Auto-scaling group (BETA)

An auto-scaling group for the ICP worker nodes is created containing the same configuration as the ICP worker nodes, if enable_autoscaling is set to true. Scaling up and down is triggered manually.

Lambda function (BETA)

A Lambda function is created that responds to the auto-scaling events, if enable_autoscaling is set to true. The function is zipped and uploaded to an S3 bucket, along with client certificates used to talk to the Kubernetes API. The function creates Kubernetes jobs in ICP that add and remove worker nodes from the cluster using the icp-inception image. The function runs from within the VPC.

Installation Procedure

The installer automates the install procedure described here.

ICP Installation Parameters on AWS

Suggested ICP Installation parameters specific on AWS:

---
calico_tunnel_mtu: 8981
cloud_provider: aws
kubelet_nodename: fqdn

Because AWS enables Jumbo frames (MTU 9001), the Calico IP-in-IP tunnel is configured to take advantage of the larger MTU.

The cloud_provider parameter allows Kubernetes to take advantage of some Kubernetes-AWS integration with dynamic ELB creation and dynamic EBS creation for persistent volumes. When the AWS cloud_provider is used, all node names use the private FQDN retrieved from from the AWS metadata service and nodes are tagged with the correct region and availability zone. Kubernetes will stripe deployments across availability zones. See the below section for more details.

The Terraform automation generates cluster_CA_domain, cluster_lb_address, and proxy_lb_address corresponding to the DNS names for the master and proxy ELB.

Note the other parameters in the icp-deploy.tf module. The config files are stored in /opt/ibm/cluster/config.yaml on the boot-master.

Installing IBM Cloud Private Community Edition

The following parameters are required settings to install IBM Cloud Private Community Edition. These values are the preferred values for any conflicting paramters in the terraform.tfvars file, as specified above in the Prerequisites section. These settings have been validated on IBM Cloud Private 3.1.0 Community Edition.

image_location = ""

icp_inception_image = "ibmcom/icp-inception-amd64:3.1.2"

bastion = {
 nodes = "1"
}

master = {
  nodes = "1"          # required to be '1' to install CE
  type = "m4.2xlarge"  # or m4.4xlarge if 'management' nodes=0
  disk = "300"
}

management = {
  nodes = "1"          # or optionally 0 if you want to run all platform services on 'master'
  type = "m4.xlarge"
  disk = "300"
}

va = {
  nodes = "0"
}

proxy = {
  nodes = "1"          # required to be '1' to install CE
  disk = "150"
}

worker = {
  disk = "150"
}

Cluster access

ICP Console access

The ICP console can be accessed at https://<cluster_lb_address>:8443. See documentation.

ICP Private Image Registry access

The registry is available at https://<cluster_lb_address>:8500. See documentation for how to configure Docker to access the registry.

ICP Kubernetes API access

The Kubernetes API can be reached at https://<cluster_lb_address>:8001. To obtain a token, see the documentation or this blog post,

ICP Ingress Controller

Ingress Resources can be created and exposed using the proxy node endpoints at http://<proxy_lb_address>:80 or https://<proxy_lb_address>:443

AWS Cloud Provider

The AWS Cloud provider provides Kubernetes integration with Elastic Load Balancer and Elastic Block Store. See documentation on LoadBalancer and Volume