Ensure kubernetes deployment is automated

[jmcarp]

In order to have a repeatable, auditable configuration the Kubernetes deployment needs to be created by automation.

Items:

• [x] IAM users for k8s masters / workers are created automatically
• [ ] Security group for k8s masters (only allow ingress on 6443 and ephemeral ports) are created automatically
• [ ] ELB for k8s masters is created automatically

[jmcarp]

Note: the kubernetes source helpfully includes example IAM policies at https://github.com/kubernetes/kubernetes/blob/master/cluster/aws/templates/iam/kubernetes-minion-policy.json and https://github.com/kubernetes/kubernetes/blob/master/cluster/aws/templates/iam/kubernetes-master-policy.json.

[mogul]

Can you go through the grooming process on this one? It's hard to see what the outcome's expected to be and why.

[datn]

1) I see k8s_master and k8s_minion being created in cg-provision:terraform/modules/kubernetes/iam.tf

2) In cg-provision:terraform/modules/kubernetes/sg_elb.tf I see security groups and port creation conforming to 6443 inbound and anything outbound... not sure what the ephemeral port requirement is.

3) Pretty sure cg-provision:terraform/modules/kubernetes/sg_elb.tf creates ELBs for k8s masters, but I need some more info to understand fully.

[cnelson]

1) I notice those policies don't exactly match what's in the example polices @jmcarp linked. For example. https://github.com/18F/cg-provision/blob/master/terraform/modules/kubernetes/iam.tf#L64 I don't see CreateTags in the example Minion policy, and in the master policy explicitly listing it is probably unnecessary since we grant ec2:*

2) LGTM 🌮

[jmcarp]

We need CreateTags until bosh director knows how to do it, at which point we can get rid of https://github.com/18F/kubernetes-release/blob/master/jobs/kubernetes-minion/templates/pre-start.erb.

[rogeruiz]

A good way to verify this is to deploy this to production in an automated way.

[cnelson]

k8s is in production now. Are we good to go on this?