[Bug]: Pool Recreates everytime

[serafdev]

Terraform Version

Terraform v1.3.6 on linux_amd64

Linode Provider Version

version = "1.30.0

Effected Terraform Resources

linode_lke_cluster

Terraform Config Files

❯ cat lke.tf 
resource "linode_lke_cluster" "apps" {
  label       = "apps"
  k8s_version = "1.25"
  region      = "us-central"
  tags        = ["apps"]
  pool {
    type  = "g6-standard-2"
    count = 3

    autoscaler {
      min = 3
      max = 10
    }
  }

  lifecycle {
    ignore_changes = [
      pool.0.count
    ]
  }
}

data "linode_lke_cluster" "apps" {
  id = linode_lke_cluster.apps.id
}

resource "local_file" "kubeconfig" {
  content  = base64decode(data.linode_lke_cluster.apps.kubeconfig)
  filename = ".kubeconfig_apps"
}

Debug Output

linode_lke_cluster.apps: Refreshing state... [id=94328]
local_file.kubeconfig: Refreshing state... [id=4b5e023697dab09c7be8e97806d7350203ddf184]
helm_release.ingress_nginx: Refreshing state... [id=ingress-nginx]

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  ~ update in-place
-/+ destroy and then create replacement
 <= read (data resources)

Terraform will perform the following actions:

  # data.linode_lke_cluster.apps will be read during apply
  # (depends on a resource or a module with changes pending)
 <= data "linode_lke_cluster" "apps" {
      + api_endpoints = (known after apply)
      + control_plane = (known after apply)
      + dashboard_url = (known after apply)
      + id            = 94328
      + k8s_version   = (known after apply)
      + kubeconfig    = (sensitive value)
      + label         = (known after apply)
      + pools         = (known after apply)
      + region        = (known after apply)
      + status        = (known after apply)
      + tags          = (known after apply)
    }

  # linode_lke_cluster.apps will be updated in-place
  ~ resource "linode_lke_cluster" "apps" {
        id            = "94328"
        tags          = [
            "apps",
        ]
        # (7 unchanged attributes hidden)

      ~ pool {
            id    = 0
          + type  = "g6-standard-2"
            # (2 unchanged attributes hidden)

          + autoscaler {
              + max = 10
              + min = 3
            }
        }
      - pool {
          - count = 4 -> null
          - id    = 142633 -> null
          - nodes = [
              - {
                  - id          = "142633-63f6963d0f54"
                  - instance_id = 42959431
                  - status      = "ready"
                },
              - {
                  - id          = "142633-63f6963d1425"
                  - instance_id = 42959439
                  - status      = "ready"
                },
              - {
                  - id          = "142633-63f6963d18b3"
                  - instance_id = 42959442
                  - status      = "ready"
                },
              - {
                  - id          = "142633-640e1b22b723"
                  - instance_id = 43789456
                  - status      = "ready"
                },
            ] -> null
          - type  = "g6-standard-2" -> null

          - autoscaler {
              - max = 10 -> null
              - min = 3 -> null
            }
        }

        # (1 unchanged block hidden)
    }

  # local_file.kubeconfig must be replaced
-/+ resource "local_file" "kubeconfig" {
      # Warning: this attribute value will no longer be marked as sensitive
      # after applying this change.
      ~ content              = (sensitive value) -> (known after apply) # forces replacement
      ~ id                   = "4b5e023697dab09c7be8e97806d7350203ddf184" -> (known after apply)
        # (3 unchanged attributes hidden)
    }

Plan: 1 to add, 1 to change, 1 to destroy.
local_file.kubeconfig: Destroying... [id=4b5e023697dab09c7be8e97806d7350203ddf184]
local_file.kubeconfig: Destruction complete after 0s
linode_lke_cluster.apps: Modifying... [id=94328]
linode_lke_cluster.apps: Modifications complete after 0s [id=94328]
data.linode_lke_cluster.apps: Reading...
data.linode_lke_cluster.apps: Read complete after 0s
local_file.kubeconfig: Creating...
local_file.kubeconfig: Creation complete after 0s [id=4b5e023697dab09c7be8e97806d7350203ddf184]

Apply complete! Resources: 1 added, 1 changed, 1 destroyed.

Panic Output

No response

Expected Behavior

Should not be modifying the cluster's pool or anything in the lke cluster

Actual Behavior

• Modifies the lke cluster by deleteing->recreating pool (seems to not affect actual infra)
• recreates the kubeconfig file

Steps to Reproduce

• Add linode_lke_cluster terraform resource
• Add the .kubeconfig output to a local file
• Run terraform apply and wait for cluster to be created
• Re-run terraform apply without modifying anything

[lgarber-akamai]

Thanks for the bug report!

I was able to reproduce this issue locally by applying, resizing the node pool up by one node, and applying again. Additionally, this issue only seems to occur when the ignore_changes lifecycle rule is specified.

Fortunately, it looks like applying does not destroy or recreate the existing node pool as you mentioned, but the inaccurate diff can definitely lead to some user confusion.

I've added this issue to our internal tracker and will let you know when we have any updates 🙂

[github-actions[bot]]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days

[tchernobog]

Still actual.

[github-actions[bot]]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days

[Aransh]

still actual

[github-actions[bot]]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days

[Aransh]

still actual 🙃

[displague]

I created issue https://github.com/linode/terraform-provider-linode/issues/1076 with an enhancement path that could help with this issue.

[lgarber-akamai]

Closing this issue out since it should have been resolved in https://github.com/linode/terraform-provider-linode/pull/1260. If this bug is still relevant feel free to @ me and it can be reopened 🙂

[Aransh]

@lgarber-akamai I am still seeing this issue

[lgarber-akamai]

@Aransh Thank you for the info!

I'm not entirely sure if this is related but could you try dropping the count field from the node pool and ignore_changes block from the resource? These were previously required as a workaround but are no longer necessary for autoscaled pools now that count has been made optional.

If that doesn't work, could you share your (sanitized) Terraform state with us as well as any other relevant information you have?

[Aransh]

Hi @lgarber-akamai here's some additional details: I have already removed the count field and ignore_changes prior.

This is my current cluster in tfstate (sanitized of all ids/urls):

# module.csi_cluster.linode_lke_cluster.csi-k8s-cluster:
resource "linode_lke_cluster" "csi-k8s-cluster" {
    api_endpoints = [
        "https://xxxx.us-iad-2.linodelke.net:443",
        "https://xxxx.us-iad-2.linodelke.net:6443",
        "https://xxxx:443",
        "https://xxxx:6443",
    ]
    dashboard_url = "https://xxxx.dashboard.us-iad-2.linodelke.net"
    id            = "xxxx"
    k8s_version   = "1.27"
    kubeconfig    = (sensitive value)
    label         = "xxxx"
    region        = "us-iad"
    status        = "ready"
    tags          = [
        "dev",
    ]

    control_plane {
        high_availability = true
    }

    pool {
        count = 98
        id    = xxxx
        nodes = [
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
        ]
        type  = "g6-dedicated-32"

        autoscaler {
            max = 100
            min = 2
        }
    }
    pool {
        count = 40
        id    = 228786
        nodes = [
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "not_ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
            {
                id          = "xxxx"
                instance_id = xxxx
                status      = "ready"
            },
        ]
        type  = "g6-dedicated-32"

        autoscaler {
            max = 100
            min = 2
        }
    }
}

This is the relevant part of my module to create the cluster:

resource "linode_lke_cluster" "csi-k8s-cluster" {
  k8s_version = "1.27"
  label       = "xxxx"
  region      = "us-iad"
  tags        = ["dev"]
  dynamic "pool" {
    for_each = var.pools
    content {
      type  = pool.value["type"]
      autoscaler {
        min = pool.value["min"]
        max = pool.value["max"]
      }
    }
  }

  control_plane {
    high_availability = true
  }
}

And these are the relevant tfvars (note 2 pools because max is 100 nodes):

pools       = [
  {
    type  = "g6-dedicated-32"
    min   = 2
    max   = 100
  },
  {
    type  = "g6-dedicated-32"
    min   = 2
    max   = 100
  }
]

Running apply on this configuration gives me this plan:

  # module.csi_cluster.linode_lke_cluster.csi-k8s-cluster has changed
  ~ resource "linode_lke_cluster" "csi-k8s-cluster" {
        id            = "xxxx"
        tags          = [
            "dev",
        ]
        # (7 unchanged attributes hidden)

      ~ pool {
          ~ count = 40 -> 39
            id    = xxxx
          ~ nodes = [
                # (26 unchanged elements hidden)
                {
                    id          = "xxxx"
                    instance_id = xxxx
                    status      = "ready"
                },
              - {
                  - id          = "xxxx"
                  - instance_id = xxxx
                  - status      = "not_ready"
                },
                {
                    id          = "xxxx"
                    instance_id = xxxx
                    status      = "ready"
                },
                # (11 unchanged elements hidden)
            ]
            # (1 unchanged attribute hidden)

            # (1 unchanged block hidden)
        }

        # (2 unchanged blocks hidden)
    }

As you can see the changes here are for the count field (which I do not provide), and for the list of nodes (which I do not manage). Also worth mentioning I recently tried to state rm and import this cluster to solve this issue, it did solve it (plan was empty), but as soon as nodes were replaced, same issue.

[lgarber-akamai]

@Aransh Thank you for the detailed info and sorry to hear this has been a continued issue. I'll see if I can reproduce this on my end using your configuration.

[lgarber-akamai]

Unfortunately, I haven't had any luck reproducing this issue on the latest version of the provider yet. We still have this issue on our radar so I'll update this issue if we make any progress :+1:

[github-actions[bot]]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days

[Aransh]

Still experiencing this

[github-actions[bot]]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days

[Aransh]

Still happening

[jcallahan-akamai]

@Aransh what version of the Linode TF provider are you using?

[Aransh]

@jcallahan-akamai 2.23.0

[jcallahan-akamai]

Hi @Aransh, unfortunately we are stumped by this one. We have tried to reproduce this bug several times with no success. I'm going to close this issue, but if you are able to produce a minimal, reproducible example, feel free to open it back up.