search

zifeo / terraform-openstack-rke2

Easily deploy a high-availability RKE2 Kubernetes cluster on OpenStack providers like Infomaniak.
https://registry.terraform.io/modules/zifeo/rke2/openstack/latest
Mozilla Public License 2.0
29 stars 18 forks source link

Reducing node count on agent nodes leads to "inconsistent final plan" #7

Closed UncleSamSwiss closed 1 year ago

UncleSamSwiss commented 1 year ago

I had the following configuration:

locals {
  config = <<EOF
# https://docs.rke2.io/install/install_options/server_config/

etcd-snapshot-schedule-cron: "0 */6 * * *"
etcd-snapshot-retention: 20

# control-plane-resource-requests: kube-apiserver-cpu=75m,kube-apiserver-memory=128M,kube-scheduler-cpu=75m,kube-scheduler-memory=128M,kube-controller-manager-cpu=75m,kube-controller-manager-memory=128M,kube-proxy-cpu=75m,kube-proxy-memory=128M,etcd-cpu=75m,etcd-memory=128M,cloud-controller-manager-cpu=75m,cloud-controller-manager-memory=128M
  EOF
}

module "rke2" {
  source  = "zifeo/rke2/openstack"
  version = "1.1.0"

  # must be true for single-server cluster or only on first run for HA cluster
  bootstrap           = true
  name                = "i8k-test"
  ssh_public_key_file = "${path.module}/../../id_rsa.pub"
  floating_pool       = "ext-floating1"

  # should be restricted to a secure bastion
  rules_ssh_cidr = "0.0.0.0/0"
  rules_k8s_cidr = "0.0.0.0/0"

  # auto load manifest form a folder (https://docs.rke2.io/advanced#auto-deploying-manifests)
  manifests_folder = "./manifests"

  servers = [
    for i in range(1, 4) : {
      name = "server-${format("%03d", i)}"

      flavor_name      = "a2-ram4-disk0"
      image_name       = "Ubuntu 22.04 LTS Jammy Jellyfish"
      system_user      = "ubuntu"
      boot_volume_size = 4

      rke2_version     = "v1.25.5+rke2r2"
      rke2_volume_size = 6
      # https://docs.rke2.io/install/install_options/install_options/#configuration-file
      rke2_config = local.config
    }
  ]

  agents = [{
    name        = "pool"
    nodes_count = 3

    flavor_name      = "a4-ram8-disk0"
    image_name       = "Ubuntu 20.04 LTS Focal Fossa"
    system_user      = "ubuntu"
    boot_volume_size = 8

    rke2_version     = "v1.25.5+rke2r2"
    rke2_volume_size = 16
    }
  ]

  # enable automatically `kubectl delete node AGENT-NAME` after an agent change
  ff_autoremove_agent = true
  # rewrite kubeconfig
  ff_write_kubeconfig = true
  # deploy etcd backup
  ff_native_backup = true

  identity_endpoint     = var.openstack_auth_url
  object_store_endpoint = "s3.pub1.infomaniak.cloud"
}

Then I changed the nodes_count for the agent pool (and removed the bootstrap flag):

@@ -14,7 +14,7 @@ module "rke2" {
   version = "1.1.0"

   # must be true for single-server cluster or only on first run for HA cluster
-  bootstrap           = true
+  #bootstrap           = true
   name                = "i8k-test"
   ssh_public_key_file = "${path.module}/../../id_rsa.pub"
   floating_pool       = "ext-floating1"
@@ -44,7 +44,7 @@ module "rke2" {

   agents = [{
     name        = "pool"
-    nodes_count = 3
+    nodes_count = 2

     flavor_name      = "a4-ram8-disk0"
     image_name       = "Ubuntu 20.04 LTS Focal Fossa"

This lead to an inconsistent plan:

Error: Provider produced inconsistent final plan
│ 
│ When expanding the plan for module.rke2.openstack_lb_members_v2.k8s to include new values learned so far during apply, provider "registry.terraform.io/terraform-provider-openstack/openstack" produced
│ an invalid new value for .member: planned set element cty.ObjectVal(map[string]cty.Value{"address":cty.StringVal("192.168.42.34"), "admin_state_up":cty.True, "backup":cty.NullVal(cty.Bool),
│ "id":cty.StringVal("f96cb672-594c-4b5f-a965-eb928edcb2da"), "monitor_address":cty.NullVal(cty.String), "monitor_port":cty.NullVal(cty.Number), "name":cty.StringVal("i8k-test-server-003-1"),
│ "protocol_port":cty.NumberIntVal(6443), "subnet_id":cty.NullVal(cty.String), "weight":cty.NumberIntVal(1)}) does not correlate with any element in actual.
│ 
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.
╵
╷
│ Error: Provider produced inconsistent final plan
│ 
│ When expanding the plan for module.rke2.openstack_lb_members_v2.k8s to include new values learned so far during apply, provider "registry.terraform.io/terraform-provider-openstack/openstack" produced
│ an invalid new value for .member: planned set element cty.ObjectVal(map[string]cty.Value{"address":cty.StringVal("192.168.42.79"), "admin_state_up":cty.True, "backup":cty.NullVal(cty.Bool),
│ "id":cty.StringVal("991fc6ae-ebc8-4908-a7c5-9305b2e4e06f"), "monitor_address":cty.NullVal(cty.String), "monitor_port":cty.NullVal(cty.Number), "name":cty.StringVal("i8k-test-server-002-1"),
│ "protocol_port":cty.NumberIntVal(6443), "subnet_id":cty.NullVal(cty.String), "weight":cty.NumberIntVal(1)}) does not correlate with any element in actual.
│ 
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.
╵
╷
│ Error: Provider produced inconsistent final plan
│ 
│ When expanding the plan for module.rke2.openstack_lb_members_v2.k8s to include new values learned so far during apply, provider "registry.terraform.io/terraform-provider-openstack/openstack" produced
│ an invalid new value for .member: planned set element cty.ObjectVal(map[string]cty.Value{"address":cty.UnknownVal(cty.String), "admin_state_up":cty.True, "backup":cty.NullVal(cty.Bool),
│ "id":cty.UnknownVal(cty.String), "monitor_address":cty.StringVal(""), "monitor_port":cty.NullVal(cty.Number), "name":cty.UnknownVal(cty.String), "protocol_port":cty.NumberIntVal(6443),
│ "subnet_id":cty.StringVal(""), "weight":cty.NumberIntVal(1)}) does not correlate with any element in actual.
│ 
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.
╵
╷
│ Error: Provider produced inconsistent final plan
│ 
│ When expanding the plan for module.rke2.openstack_lb_members_v2.rke2 to include new values learned so far during apply, provider "registry.terraform.io/terraform-provider-openstack/openstack"
│ produced an invalid new value for .member: planned set element cty.ObjectVal(map[string]cty.Value{"address":cty.StringVal("192.168.42.34"), "admin_state_up":cty.True, "backup":cty.NullVal(cty.Bool),
│ "id":cty.StringVal("c1724279-7626-424b-a785-7f16db169463"), "monitor_address":cty.NullVal(cty.String), "monitor_port":cty.NumberIntVal(6443), "name":cty.StringVal("i8k-test-server-003-1"),
│ "protocol_port":cty.NumberIntVal(9345), "subnet_id":cty.NullVal(cty.String), "weight":cty.NumberIntVal(1)}) does not correlate with any element in actual.
│ 
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.
╵
╷
│ Error: Provider produced inconsistent final plan
│ 
│ When expanding the plan for module.rke2.openstack_lb_members_v2.rke2 to include new values learned so far during apply, provider "registry.terraform.io/terraform-provider-openstack/openstack"
│ produced an invalid new value for .member: planned set element cty.ObjectVal(map[string]cty.Value{"address":cty.StringVal("192.168.42.79"), "admin_state_up":cty.True, "backup":cty.NullVal(cty.Bool),
│ "id":cty.StringVal("ab5fd8bb-943b-49c8-9de6-a6f0c8de7b28"), "monitor_address":cty.NullVal(cty.String), "monitor_port":cty.NumberIntVal(6443), "name":cty.StringVal("i8k-test-server-002-1"),
│ "protocol_port":cty.NumberIntVal(9345), "subnet_id":cty.NullVal(cty.String), "weight":cty.NumberIntVal(1)}) does not correlate with any element in actual.
│ 
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.
╵
╷
│ Error: Provider produced inconsistent final plan
│ 
│ When expanding the plan for module.rke2.openstack_lb_members_v2.rke2 to include new values learned so far during apply, provider "registry.terraform.io/terraform-provider-openstack/openstack"
│ produced an invalid new value for .member: planned set element cty.ObjectVal(map[string]cty.Value{"address":cty.UnknownVal(cty.String), "admin_state_up":cty.True, "backup":cty.NullVal(cty.Bool),
│ "id":cty.UnknownVal(cty.String), "monitor_address":cty.StringVal(""), "monitor_port":cty.NumberIntVal(6443), "name":cty.UnknownVal(cty.String), "protocol_port":cty.NumberIntVal(9345),
│ "subnet_id":cty.StringVal(""), "weight":cty.NumberIntVal(1)}) does not correlate with any element in actual.
│ 
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.
╵
╷
│ Error: Provider produced inconsistent final plan
│ 
│ When expanding the plan for module.rke2.openstack_lb_members_v2.ssh[0] to include new values learned so far during apply, provider "registry.terraform.io/terraform-provider-openstack/openstack"
│ produced an invalid new value for .member: planned set element cty.ObjectVal(map[string]cty.Value{"address":cty.StringVal("192.168.42.34"), "admin_state_up":cty.True, "backup":cty.NullVal(cty.Bool),
│ "id":cty.StringVal("c10a15d6-afa5-4d53-a668-073121f27262"), "monitor_address":cty.NullVal(cty.String), "monitor_port":cty.NullVal(cty.Number), "name":cty.StringVal("i8k-test-server-003-1"),
│ "protocol_port":cty.NumberIntVal(22), "subnet_id":cty.NullVal(cty.String), "weight":cty.NumberIntVal(1)}) does not correlate with any element in actual.
│ 
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.
╵
╷
│ Error: Provider produced inconsistent final plan
│ 
│ When expanding the plan for module.rke2.openstack_lb_members_v2.ssh[0] to include new values learned so far during apply, provider "registry.terraform.io/terraform-provider-openstack/openstack"
│ produced an invalid new value for .member: planned set element cty.ObjectVal(map[string]cty.Value{"address":cty.StringVal("192.168.42.79"), "admin_state_up":cty.True, "backup":cty.NullVal(cty.Bool),
│ "id":cty.StringVal("27d67330-2cc0-466e-98c4-0a6f338843eb"), "monitor_address":cty.NullVal(cty.String), "monitor_port":cty.NullVal(cty.Number), "name":cty.StringVal("i8k-test-server-002-1"),
│ "protocol_port":cty.NumberIntVal(22), "subnet_id":cty.NullVal(cty.String), "weight":cty.NumberIntVal(1)}) does not correlate with any element in actual.
│ 
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.
╵
╷
│ Error: Provider produced inconsistent final plan
│ 
│ When expanding the plan for module.rke2.openstack_lb_members_v2.ssh[0] to include new values learned so far during apply, provider "registry.terraform.io/terraform-provider-openstack/openstack"
│ produced an invalid new value for .member: planned set element cty.ObjectVal(map[string]cty.Value{"address":cty.UnknownVal(cty.String), "admin_state_up":cty.True, "backup":cty.NullVal(cty.Bool),
│ "id":cty.UnknownVal(cty.String), "monitor_address":cty.StringVal(""), "monitor_port":cty.NullVal(cty.Number), "name":cty.UnknownVal(cty.String), "protocol_port":cty.NumberIntVal(22),
│ "subnet_id":cty.StringVal(""), "weight":cty.NumberIntVal(1)}) does not correlate with any element in actual.
│ 
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.
╵
UncleSamSwiss commented 1 year ago

This could of course be a bug in the provider, but I somehow doubt it.

But, I should perhaps note that I have a service in the cluster that is of type LoadBalancer which of course created a new OpenStack load balancer.

zifeo commented 1 year ago

@UncleSamSwiss This is definitely a bug of the OpenStack provider. However I never manage to get it reproduced consistently and never had an actual effect (I suspect this is when reading the state after an apply). When it happens, you can apply a second time and everything should be fine.