kops_instance_group does not accept the documented mixed_instances_policy arguments

[ddelange]

Goal

Start a simple kops cluster with 100% spot workers

Setup

Following MixedInstancesPolicySpec, I drafted the following instance groups for Master and Node in a specific availability zone:

resource "kops_instance_group" "masters" {
  # one master per availability zone, and one subnet per availability zone -> one master per subnet (ref https://kops.sigs.k8s.io/tutorial/working-with-instancegroups/)
  for_each     = data.aws_subnet.all
  cluster_name = kops_cluster.cluster.name
  name         = "master-${each.value.availability_zone}"
  role         = "Master"
  min_size     = 1
  max_size     = 1
  machine_type = "t3a.small"  # FIXME might need to go back to t3a.medium
  subnets      = [each.value.id]
}

resource "kops_instance_group" "nodes" {
  for_each     = data.aws_subnet.all  # one node instance group per availability zone
  cluster_name = kops_cluster.cluster.id
  name         = "node-${each.value.availability_zone}"
  role         = "Node"
  min_size     = 0
  max_size     = 2
  machine_type = "t3a.small"  # why is this required when using mixed_instances_policy, will it be overwritten or still somehow used?
  subnets      = [each.value.id]
  mixed_instances_policy {
    instances = ["t3a.small", "t3.small"]
    # ref https://docs.aws.amazon.com/autoscaling/ec2/userguide/ec2-auto-scaling-mixed-instances-groups.html#allocation-strategies
    on_demand_allocation_strategy = "lowest-price"
    # ref https://docs.aws.amazon.com/autoscaling/ec2/userguide/ec2-auto-scaling-mixed-instances-groups.html#instances-distribution
    on_demand_base = 0
    on_demand_above_base = 0
    # ref https://docs.aws.amazon.com/autoscaling/ec2/userguide/ec2-auto-scaling-mixed-instances-groups.html#spot-best-practices
    spot_allocation_strategy = "lowest-price"
    spot_instance_pools = 2
  }
}

Result

When running terraform plan, the following errors occur:

│ Error: Unsupported argument
│
│   on workload.tf line 117, in resource "kops_instance_group" "nodes":
│  117:     on_demand_base = 0
│
│ An argument named "on_demand_base" is not expected here. Did you mean to define a block of type "on_demand_base"?
╵
╷
│ Error: Unsupported argument
│
│   on workload.tf line 118, in resource "kops_instance_group" "nodes":
│  118:     on_demand_above_base = 0
│
│ An argument named "on_demand_above_base" is not expected here. Did you mean to define a block of type "on_demand_above_base"?

Question

Can you advise on how to set up the node instance group with a mixed instances policy?

Versions

$ terraform version
Terraform v1.0.9
on darwin_amd64
+ provider registry.terraform.io/banzaicloud/k8s v0.8.2
+ provider registry.terraform.io/eddycharly/kops v1.21.2-alpha.2
+ provider registry.terraform.io/hashicorp/aws v3.58.0
...

[ddelange]

Commenting out the two erroneous arguments leads to a successful plan:

  # kops_instance_group.nodes["subnet-87b73aed"] will be created
  + resource "kops_instance_group" "nodes" {
      + cluster_name = (known after apply)
      + id           = (known after apply)
      + image        = (known after apply)
      + machine_type = "t3a.small"
      + max_size     = 2
      + min_size     = 0
      + name         = "node-eu-central-1a"
      + revision     = 1
      + role         = "Node"
      + subnets      = [
          + "subnet-87b73aed",
        ]

      + mixed_instances_policy {
          + instances                     = [
              + "t3a.small",
              + "t3.small",
            ]
          + on_demand_allocation_strategy = "lowest-price"
          + spot_allocation_strategy      = "lowest-price"
          + spot_instance_pools           = 2
        }
    }

[eddycharly]

Hey, thanks for trying it out. The issue here is that null and 0 do not mean the same thing and terraform has some limitations on this. It should be set like this:

      on_demand_base {
        value = 0
      }
      on_demand_above_base {
        value = 0
      }

[eddycharly]

There have been a couple of issues related to this #330 #256 for example. Hope this helps.

[ddelange]

Thanks for the quick reply, already read something about nullable arguments in the docs, should have rung a bell...

Trying out first thing tomorrow!