[BUG] Adding one more worker node gives waiting for cluster to be updated error

[KostLinux]

Rancher Server Setup

• Rancher version: 2.7.4
• Installation option (Docker install/Helm Chart): RKE1
  • If Helm Chart, Kubernetes Cluster and version (RKE1, RKE2, k3s, EKS, etc):
• Proxy/Cert Details: Sectigo

Information about the Cluster

• Kubernetes version: v1.25.9
• Cluster Type (Local/Downstream):
  • If downstream, what type of cluster? (Custom/Imported or specify provider for Hosted/Infrastructure Provider):

    I've created terraform module that creates cluster in rancher. When adding worker to node list, apply step gets stuck at rancher2 provider with following error [ERROR] vertex "module.kubernetes_clusters.rancher2_cluster.rke" error: [ERROR] waiting for cluster (c-kxv62) to be updated: unexpected state 'unavailable', wanted target 'active, provisioning, pending'. last error: %!s(<nil>)

    Googling error didn't give any answers.

    To Reproduce

    Create rancher2 cluster via provider ->

    resource "rancher2_cluster" "rke" {
      name        = var.cluster_name
      description = var.cluster_description
      depends_on = [ vsphere_virtual_machine.master_nodes, vsphere_virtual_machine.storage_nodes, vsphere_virtual_machine.worker_nodes ]
      enable_network_policy = true
      rke_config {
        kubernetes_version = var.kubernetes_version
        enable_cri_dockerd = true
        private_registries {
          url = "MY_PRIVATE_REPO"
          is_default = true
        }
        dynamic nodes {
            for_each = concat(var.master_nodes, var.worker_nodes, var.storage_nodes)
            content {
              role    = nodes.value["role"] == "control-plane" ? ["controlplane", "etcd"] : ["worker"]
              address = nodes.value["vm_net_ip"]
              user    = var.ssh_user
              ssh_key = var.private_ssh_key          
            }
    
        }
        ingress {
          default_backend = true
          provider        = "nginx"
        }
        network {
          plugin = "canal"
        }
        services {
          etcd {
            backup_config {
                interval_hours = 4
                retention = 2
                timeout = 300
            }
            creation = "12h"
            extra_args = {
                quota-backend-bytes = "7500000000"
            }
          }
          kube_api {
            service_cluster_ip_range = "10.3.0.0/16"
            service_node_port_range = "30000-32767"
            extra_args = {
                audit-log-path = "/var/log/kube-audit.log"
            }
            audit_log {
              enabled = true
            }
          }
          kube_controller {
            cluster_cidr             = "10.2.0.0/16"
            service_cluster_ip_range = "10.3.0.0/16"
            extra_args = {
                cluster-signing-cert-file = "/etc/kubernetes/ssl/kube-ca.pem"
                cluster-signing-key-file = "/etc/kubernetes/ssl/kube-ca-key.pem"
            }
          }
          kubelet {
            cluster_dns_server = "10.3.0.10"
            extra_args = {
              max-pods       = 250
              cloud-provider = "external"
            }
            fail_swap_on = false
            extra_binds = [
              "/usr/libexec/kubernetes/kubelet-plugins:/usr/libexec/kubernetes/kubelet-plugins",
              "/lib/modules:/lib/modules:rshared"
            ]        
          }
        }
    
        upgrade_strategy {
          drain                        = true
          max_unavailable_controlplane = "1"
          max_unavailable_worker       = "20%"
    
          drain_input {
            delete_local_data  = true
            force              = true
            grace_period       = -1
            ignore_daemon_sets = true
            timeout            = 120
          }
        }
    
      }
    
      cluster_auth_endpoint {
        enabled = false
      }
    }

    Actual Result

    It gives unknown error that i can't even debug normally...

    [ERROR] vertex "module.kubernetes_clusters.rancher2_cluster.rke" error: [ERROR] waiting for cluster (c-kxv62) to be updated: unexpected state 'unavailable', wanted target 'active, provisioning, pending'. last error: %!s(<nil>)

    Expected Result

    It will register another worker to cluster

    Screenshots

    Additional context

    Module works in these steps:

    • Create vSphere VM's (master_nodes, worker_nodes, storage_nodes)
    • Configure & Create Rancher2 cluster (you can look for code above)
    • Run Ansible playbook that registers node according to role
    • Configure rancher2 catalog for helm repository

[gorantornqvist-sr]

Same when upgrading RKE kubernetes version.

Terraform will perform the following actions:

  # module.cluster.rancher2_cluster.cluster will be updated in-place
  ~ resource "rancher2_cluster" "cluster" {
        id                                                         = "c-wp5kb"
        name                                                       = "lab01"
        # (17 unchanged attributes hidden)

      ~ rke_config {
          ~ kubernetes_version    = "v1.26.11-rancher2-1" -> "v1.27.11-rancher1-1"
            # (5 unchanged attributes hidden)

            # (9 unchanged blocks hidden)
        }

        # (1 unchanged block hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.
module.cluster.rancher2_cluster.cluster: Modifying... [id=c-wp5kb]
module.cluster.rancher2_cluster.cluster: Still modifying... [id=c-wp5kb, 10s elapsed]
module.cluster.rancher2_cluster.cluster: Still modifying... [id=c-wp5kb, 20s elapsed]
module.cluster.rancher2_cluster.cluster: Still modifying... [id=c-wp5kb, 30s elapsed]
module.cluster.rancher2_cluster.cluster: Still modifying... [id=c-wp5kb, 1m0s elapsed]
module.cluster.rancher2_cluster.cluster: Still modifying... [id=c-wp5kb, 1m10s elapsed]
module.cluster.rancher2_cluster.cluster: Still modifying... [id=c-wp5kb, 1m20s elapsed]
module.cluster.rancher2_cluster.cluster: Still modifying... [id=c-wp5kb, 1m30s elapsed]
module.cluster.rancher2_cluster.cluster: Still modifying... [id=c-wp5kb, 1m40s elapsed]
module.cluster.rancher2_cluster.cluster: Still modifying... [id=c-wp5kb, 1m50s elapsed]
module.cluster.rancher2_cluster.cluster: Still modifying... [id=c-wp5kb, 2m0s elapsed]
module.cluster.rancher2_cluster.cluster: Still modifying... [id=c-wp5kb, 2m10s elapsed]
module.cluster.rancher2_cluster.cluster: Still modifying... [id=c-wp5kb, 2m20s elapsed]
module.cluster.rancher2_cluster.cluster: Still modifying... [id=c-wp5kb, 2m30s elapsed]
module.cluster.rancher2_cluster.cluster: Still modifying... [id=c-wp5kb, 2m40s elapsed]
module.cluster.rancher2_cluster.cluster: Still modifying... [id=c-wp5kb, 2m50s elapsed]
module.cluster.rancher2_cluster.cluster: Still modifying... [id=c-wp5kb, 3m0s elapsed]
module.cluster.rancher2_cluster.cluster: Still modifying... [id=c-wp5kb, 3m10s elapsed]
module.cluster.rancher2_cluster.cluster: Still modifying... [id=c-wp5kb, 3m20s elapsed]
module.cluster.rancher2_cluster.cluster: Still modifying... [id=c-wp5kb, 3m30s elapsed]
module.cluster.rancher2_cluster.cluster: Still modifying... [id=c-wp5kb, 3m40s elapsed]
module.cluster.rancher2_cluster.cluster: Still modifying... [id=c-wp5kb, 3m50s elapsed]
module.cluster.rancher2_cluster.cluster: Still modifying... [id=c-wp5kb, 4m0s elapsed]
module.cluster.rancher2_cluster.cluster: Still modifying... [id=c-wp5kb, 4m10s elapsed]
module.cluster.rancher2_cluster.cluster: Still modifying... [id=c-wp5kb, 4m20s elapsed]
module.cluster.rancher2_cluster.cluster: Still modifying... [id=c-wp5kb, 4m30s elapsed]
module.cluster.rancher2_cluster.cluster: Still modifying... [id=c-wp5kb, 4m40s elapsed]
module.cluster.rancher2_cluster.cluster: Still modifying... [id=c-wp5kb, 4m50s elapsed]
module.cluster.rancher2_cluster.cluster: Still modifying... [id=c-wp5kb, 5m0s elapsed]
module.cluster.rancher2_cluster.cluster: Still modifying... [id=c-wp5kb, 5m10s elapsed]
module.cluster.rancher2_cluster.cluster: Still modifying... [id=c-wp5kb, 5m20s elapsed]
╷
Error: [0m Error: [ERROR] waiting for cluster (c-wp5kb) to be updated: unexpected state 'error', wanted target 'active, provisioning, pending'. last error: %!s(<nil>)