[BUG] ignore_changes does not work

papanito commented 1 year ago

Rancher Server Setup

Rancher version: 2.7.4
Installation option (Docker install/Helm Chart):
- If Helm Chart, Kubernetes Cluster and version (RKE1, RKE2, k3s, EKS, etc):
Proxy/Cert Details: -

Information about the Cluster

Kubernetes version: 1.24.13

Cluster Type (Local/Downstream): Downstream (Imported)

I manually changed Pod Security Admission Configuration Template in the UI. Since then tf plan fails as mentioned in https://github.com/rancher/terraform-provider-rancher2/issues/908

╷
│ Error: rke_config.0.services.0.kube_api.0.admission_configuration.plugins: '' expected type 'string', got unconvertible type '[]interface {}', value: '[map[configuration:map[apiVersion:pod-security.admission.config.k8s.io/v1beta1 defaults:map[audit:restricted audit-version:latest enforce:restricted enforce-version:latest warn:restricted warn-version:latest] exemptions:map[namespaces:[ingress-nginx kube-system cattle-system cattle-epinio-system cattle-fleet-system longhorn-system cattle-neuvector-system cattle-monitoring-system rancher-alerting-drivers cis-operator-system cattle-csp-adapter-system cattle-externalip-system cattle-gatekeeper-system istio-system cattle-istio-system cattle-logging-system cattle-windows-gmsa-system cattle-sriov-system cattle-ui-plugin-system tigera-operator metallb-system ]] kind:PodSecurityConfiguration] name:PodSecurity path:]]'
│
│   with rancher2_cluster.cluster,
│   on cluster.tf line 2, in resource "rancher2_cluster" "cluster":
│    2: resource "rancher2_cluster" "cluster" {
│
╵

So I add the following to be able to ignore the changes

  lifecycle {
    ignore_changes = [
      rke_config.0.services.0.kube_api.0.admission_configuration
    ]
  }

However I still get the error.

I also downgraded the provider to e.g. 2.0.0 with the same result

I also tried this notation, without success

  lifecycle {
    ignore_changes = [
      rke_config[0].services[0].kube_api[0].admission_configuration
    ]
  }

To Reproduce

Have a terraform code which manages resource "rancher2_cluster" "cluster" with the ignore_changes added
Enable "Pod Security Admission Configuration Template" in the Rancher UI for the downstream cluster
Run tf plan

Actual Result

Error appears as mentioned in the bug description

Expected Result

The changes I expect are ignored and I get a successful plan

Screenshots

N/A

Additional context

## Configure rancher2 RKE Cluster
resource "rancher2_cluster" "cluster" {
  name                      = "playground"
  rke_config {
    kubernetes_version = "v1.24.13-rancher2-2"
    addon_job_timeout  = xxx
    network {
      plugin = "canal"
    }
    enable_cri_dockerd = true
    services {
      etcd {
        gid = xxxx
        uid = xxxx
      }
      kube_api {
        always_pull_images  = true
        extra_args = {
          "feature-gates" = "CronJobTimeZone=true"
        }
        audit_log {
          enabled = true
        }
        event_rate_limit {
          enabled = true
        }
      }

      kube_controller {
        extra_args = {
          "bind-address" = "127.0.0.1"
          "feature-gates" = "CronJobTimeZone=true"
        }
        extra_binds = [
          "/etc/localtime:/etc/localtime",
        ]
        extra_env = []
      }

      kubelet {
        extra_args = {
          "feature-gates"           = "CronJobTimeZone=true"
          "protect-kernel-defaults" = "true"
          "tls-cipher-suites"       = "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256"
        }
        generate_serving_certificate = true
      }

      kubeproxy {
        extra_args = {
          "feature-gates" = "CronJobTimeZone=true"
        }
        extra_binds = []
        extra_env   = []
      }
      scheduler {
        extra_args = {
          "bind-address" = "127.0.0.1"
          "feature-gates" = "CronJobTimeZone=true"
        }
        extra_binds = []
        extra_env   = []
      }
    }
    upgrade_strategy {
      max_unavailable_worker = "2"
    }
  }

  lifecycle {
    ignore_changes = [
      rke_config[0].services[0].kube_api[0].admission_configuration
    ]
  }
}

Output of console

> rancher2_cluster.cluster.rke_config.0.services.0.kube_api.0
{
  "admission_configuration" = tomap({})
  "always_pull_images" = true
  "audit_log" = tolist([
    {
      "configuration" = tolist([])
      "enabled" = true
    },
  ])
  "event_rate_limit" = tolist([
    {
      "configuration" = ""
      "enabled" = true
    },
  ])
  "extra_args" = tomap({
    "feature-gates" = "CronJobTimeZone=true"
  })
  "extra_binds" = tolist([])
  "extra_env" = tolist([])
  "image" = ""
  "pod_security_policy" = false
  "secrets_encryption_config" = tolist([])
  "service_cluster_ip_range" = ""
  "service_node_port_range" = "xxxxx-xxxxx"
}

git-ival commented 11 months ago

@papanito Can you please review the discussion in https://github.com/rancher/terraform-provider-rancher2/pull/1117 and try the suggested workaround on the latest version of the provider and verify if this is sufficient for your usecase? Thanks

papanito commented 11 months ago

The recommended workaround at https://github.com/rancher/rancher/issues/12821#issuecomment-391381651 or https://github.com/rancher/rancher/issues/12821#issuecomment-516421250 respectively, seems to work for us. So basically we add a placeholder

    annotations = {
      "metallb.universe.tf/ip-allocated-from-pool" = "placeholder"
      "field.cattle.io/publicEndpoints" = "placeholder"
    }

then, ignore_changes works

  lifecycle {
    prevent_destroy = true 
    ignore_changes = [
      metadata[0].annotations["field.cattle.io/publicEndpoints"],
      metadata[0].annotations["metallb.universe.tf/ip-allocated-from-pool"]  
    ]
  }

rancher / terraform-provider-rancher2