instance group with on-demand and spot

[pahud]

kops 1.12.0開始支持instance group with mixed instance types，詳見上游官方文件 https://github.com/kubernetes/kops/blob/master/docs/instance_groups.md#creating-a-instance-group-of-mixed-instances-types-aws-only

這個配置方式將會使instance group後面的autoscaling group使用Launch Template來起ondemand+spot跨多個instance types與多個AZ的配置。關於這部分的AWS功能可以查看這裡 https://aws.amazon.com/blogs/aws/new-ec2-auto-scaling-groups-with-multiple-instance-types-purchase-options/

操作流程

參考kops-cn說明，make create-cluster與make edit-cluster完成

手動執行kops修改nodes instance group

$ export KOPS_STATE_STORE=s3://pahud-kops-state-store-zhy
$ export AWS_REGION=cn-northwest-1
$ export AWS_DEFAULT_REGION=$AWS_REGION
$ kops edit ig --name=cluster.zhy.k8s.local nodes

For example

apiVersion: kops/v1alpha2
kind: InstanceGroup
metadata:
  creationTimestamp: 2019-06-04T10:19:12Z
  labels:
    kops.k8s.io/cluster: cluster.zhy.k8s.local
  name: nodes
spec:
  image: ami-0cb93c9d844de0c18
  machineType: c5.large
  maxSize: 10
  minSize: 10
  mixedInstancesPolicy:
    instances:
    - m4.large
    - c4.xlarge
    - c4.large
    onDemandAboveBase: 0
    onDemandBase: 2
    spotInstancePools: 9
  nodeLabels:
    kops.k8s.io/instancegroup: nodes
  role: Node
  subnets:
  - cn-northwest-1a
  - cn-northwest-1b
  - cn-northwest-1c

最後

$ make update-cluster

查看EC2 autoscaling console

[pahud]

这个模式目前看起来还有一些问题， node无法ready，还需要深入查一下原因 可能跟这个问题有关 https://github.com/kubernetes/kops/issues/7091

[walkley]

kops的问题kubernetes/kops#7091 已经被close了，对应的pr是kubernetes/kops#7370

我刚才用最新的kops 1.13.0 + k8s v1.13.10测试，worker node正常工作，扩容spot fleet也正常工作

[pahud]

太棒了 @walkley 那我先close這個issue了