search

kz8s / tack

Terraform module for creating Kubernetes cluster running on Container Linux by CoreOS in an AWS VPC
MIT License
719 stars 145 forks source link
aws-terraform aws-vpc coreos coreos-cluster docker etcd-cluster infrastructure-as-code kubernetes kubernetes-coreos-terraform

readme

terraform-aws-coreos-kubernetes

Circle CI

Opinionated Terraform module for creating a Highly Available Kubernetes cluster running on Container Linux by CoreOS (any channel) in an AWS Virtual Private Cloud VPC. With prerequisites installed make all will simply spin up a default cluster; and, since it is based on Terraform, customization is much easier than CloudFormation.

The default configuration includes Kubernetes add-ons: DNS, Dashboard and UI.

tl;dr

# prereqs
$ brew update && brew install awscli cfssl jq kubernetes-cli terraform

# build artifacts and deploy cluster
$ make all

# nodes
$ kubectl get nodes

# addons
$ kubectl get pods --namespace=kube-system

# verify dns - run after addons have fully loaded
$ kubectl exec busybox -- nslookup kubernetes

# open dashboard
$ make dashboard

# obliterate the cluster and all artifacts
$ make clean

Component and Tool Versions

component / tool version
Container Linux by CoreOS 1409.7.0, 1465.3.0, 1492.1.0
kubernetes 1.7.4
flanneld 0.7.1
docker 1.12.6
etcd 3.1.6
rkt 1.25.0
terraform 0.10.0
cfssl 1.2.0
aws-cli aws-cli/1.11.129 Python/2.7.10 Darwin/16.7.0 botocore/1.5.92
jq 1.5

Features

AWS

Container Linux by CoreOS

Kubernetes

Terraform

Prerequisites

Quick install prerequisites on Mac OS X with Homebrew:

$ brew update && brew install awscli cfssl jq kubernetes-cli terraform

Launch Cluster

make all will create:

make all

To open dashboard:

make dashboard

To display instance information:

make instances

To display status:

make status

To destroy, remove and generally undo everything:

make clean

make all and make clean should be idempotent - should an error occur simply try running the command again and things should recover from that point.

How Tack works

Tack Phases

Tack works in three phases:

  1. Pre-Terraform
  2. Terraform
  3. Post-Terraform

Pre-Terraform

The purpose of this phase is to prep the environment for Terraform execution. Some tasks are hard or messy to do in Terraform - a little prep work can go a long way here. Determining the Container Linux by CoreOS AMI for a given region, channel and VM Type for instance is easy enough to do with a simple shell script.

Terraform

Terraform does the heavy lifting of resource creation and sequencing. Tack uses local modules to partition the work in a logical way. Although it is of course possible to do all of the Terraform work in a single .tf file or collection of .tf files, it becomes unwieldy quickly and impossible to debug. Breaking the work into local modules makes the flow much easier to follow and provides the basis for composing variable solutions down the track - for example converting the worker Auto Scaling Group to use spot instances.

Post-Terraform

Once the infrastructure has been configured and instantiated it will take some time for it to settle. Waiting for the 'master' ELB to become healthy is an example of this.

Components

Like many great tools, tack has started out as a collection of scripts, makefiles and other tools. As tack matures and patterns crystalize it will evolve to a Terraform plugin and perhaps a Go-based cli tool for 'init-ing' new cluster configurations. The tooling will compose Terraform modules into a solution based on user preferences - think npm init or better yet yeoman.

TLS Certificates

curl --cacert /etc/kubernetes/ssl/ca.pem --cert /etc/kubernetes/ssl/k8s-etcd.pem --key /etc/kubernetes/ssl/k8s-etcd-key.pem https://etcd.test.kz8s:2379/health
openssl x509 -text -noout -in /etc/kubernetes/ssl/ca.pem
openssl x509 -text -noout -in /etc/kubernetes/ssl/k8s-etcd.pem

ElasticSearch and Kibana

To access Elasticseach and Kibana first start kubectl proxy.

$ kubectl proxy
Starting to serve on localhost:8001

FAQs

Advanced Features and Configuration

Using an Existing VPC

If you have an existing VPC you'd like to deploy a cluster into, there is an option for this with tack.

Constraints

Enabling Existing VPC Support

Testing Existing VPC Support from Scratch

In order to test existing VPC support, we need to generate a VPC and then try the overrides with it. After that we can clean it all up. These instructions are meant for someone wanting to ensure that the tack existing VPC code works properly.

Additional Configuration

Inspiration

Other Terraform Projects

References