rancher / terraform-provider-rancher2

Terraform Rancher2 provider
https://www.terraform.io/docs/providers/rancher2/
Mozilla Public License 2.0
260 stars 226 forks source link

Resources Using Data Resource Props Are Recreated On Every Plan/Apply #460

Closed lots0logs closed 4 years ago

lots0logs commented 4 years ago

The project_id argument for the two resources is sourced from a rancher2_project data resource. It worked fine with previous versions of the provider. This is a recent regression.

Terraform v0.13.2
Configuring remote state backend...
Initializing Terraform configuration...
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.

data.terraform_remote_state.rancher: Refreshing state...
data.digitalocean_loadbalancer.lb: Refreshing state... [id=0c19ff62-0c1c-4110-915f-06ebed626224]
digitalocean_tag.do_tag: Refreshing state... [id=k8s-cluster-stage]
digitalocean_spaces_bucket.cluster_backup_storage: Refreshing state... [id=et-backup-k8s-cluster-stage]
data.cloudflare_ip_ranges.cloudflare: Refreshing state... [id=3818245435]
digitalocean_firewall.firewall: Refreshing state... [id=c6ebecab-7280-4b98-baac-cf2312998062]
module.k8s_cluster.random_password.node_password: Refreshing state... [id=none]
module.k8s_cluster.rancher2_catalog.ingress_nginx: Refreshing state... [id=ingress-nginx]
module.k8s_cluster.rancher2_cloud_credential.do_credentials[0]: Refreshing state... [id=cattle-global-data:cc-n95q8]
module.k8s_cluster.rancher2_cluster.cluster: Refreshing state... [id=c-ggvrb]
module.k8s_cluster.rancher2_node_template.cluster_node_template_do[0]: Refreshing state... [id=cattle-global-nt:nt-dhzhr]
module.k8s_cluster.rancher2_node_pool.node_pool_cluster_admin: Refreshing state... [id=c-ggvrb:k8s-cluster-stage--control]
module.k8s_cluster.rancher2_node_pool.node_pool_cluster_worker: Refreshing state... [id=c-ggvrb:k8s-cluster-stage--worker]
module.k8s_cluster.rancher2_cluster_sync.cluster: Refreshing state... [id=c-ggvrb]
module.k8s_cluster.kubernetes_config_map.nginx_template: Refreshing state... [id=ingress-nginx/nginx-template]
module.k8s_cluster.data.rancher2_project.system: Refreshing state... [id=c-ggvrb:p-kdrr6]
module.k8s_cluster.rancher2_catalog.cert_manager: Refreshing state... [id=jetstack]
module.k8s_cluster.rancher2_catalog.helm_hub: Refreshing state... [id=helm-hub]
rancher2_catalog.elegantthemes: Refreshing state... [id=elegantthemes]
module.k8s_cluster.kubernetes_service_account.cert_manager_crd: Refreshing state... [id=cert-manager/cert-manager-crd]
module.k8s_cluster.rancher2_namespace.ingress_nginx: Refreshing state... [id=ingress-nginx]
module.k8s_cluster.rancher2_namespace.cert_manager_ns: Refreshing state... [id=cert-manager]
module.k8s_cluster.kubernetes_cluster_role_binding.cert_manager_crd_admin: Refreshing state... [id=cert-manager-crd-admin]
module.k8s_cluster.rancher2_secret.cloudflare_token: Refreshing state... [id=cert-manager:cloudflare-api-token]
module.k8s_cluster.rancher2_app.cert_manager: Refreshing state... [id=p-kdrr6:cert-manager]
module.k8s_cluster.kubernetes_job.install_certmanager_crds: Refreshing state... [id=cert-manager/install-certmanager-crds]
module.k8s_cluster.rancher2_app.ingress_nginx: Refreshing state... [id=p-kdrr6:ingress-nginx]
module.k8s_cluster.kubernetes_job.install_certmanager_issuer: Refreshing state... [id=cert-manager/install-certmanager-issuer]

------------------------------------------------------------------------

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  ~ update in-place
-/+ destroy and then create replacement
 <= read (data resources)

Terraform will perform the following actions:

  # module.k8s_cluster.data.rancher2_project.system will be read during apply
  # (config refers to values not yet known)
 <= data "rancher2_project" "system"  {
      ~ annotations                     = {
          - "authz.management.cattle.io/creator-role-bindings"                   = jsonencode(
                {
                  - created  = [
                      - "project-owner",
                    ]
                  - required = [
                      - "project-owner",
                    ]
                }
            )
          - "field.cattle.io/systemImageVersion"                                 = jsonencode(
                {
                  - alerting = "system-library-rancher-monitoring-0.1.2"
                  - logging  = "system-library-rancher-logging-0.2.1"
                  - pipeline = "6acb8e1"
                }
            )
          - "lifecycle.cattle.io/create.mgmt-project-rbac-remove"                = "true"
          - "lifecycle.cattle.io/create.project-namespace-auth_c-ggvrb"          = "true"
          - "lifecycle.cattle.io/create.project-precan-alert-controller_c-ggvrb" = "true"
        } -> (known after apply)
        cluster_id                      = "c-ggvrb"
      + container_resource_limit        = (known after apply)
      ~ description                     = "System project created for the cluster" -> (known after apply)
      + enable_project_monitoring       = (known after apply)
      ~ id                              = "c-ggvrb:p-kdrr6" -> (known after apply)
      ~ labels                          = {
          - "authz.management.cattle.io/system-project" = "true"
          - "cattle.io/creator"                         = "norman"
        } -> (known after apply)
        name                            = "System"
      + pod_security_policy_template_id = (known after apply)
      + resource_quota                  = (known after apply)
      ~ uuid                            = "c549b97b-06d0-4151-908d-d6df15136849" -> (known after apply)
    }

  # module.k8s_cluster.rancher2_app.cert_manager must be replaced
-/+ resource "rancher2_app" "cert_manager" {
      ~ annotations      = {
          - "lifecycle.cattle.io/create.helm-controller_c-ggvrb" = "true"
        } -> (known after apply)
        catalog_name     = "jetstack"
      + description      = (known after apply)
      ~ external_id      = "catalog://?catalog=jetstack&template=cert-manager&version=v1.0.1" -> (known after apply)
        force_upgrade    = false
      ~ id               = "p-kdrr6:cert-manager" -> (known after apply)
      ~ labels           = {
          - "cattle.io/creator" = "norman"
        } -> (known after apply)
        name             = "cert-manager"
      ~ project_id       = "c-ggvrb:p-kdrr6" -> (known after apply) # forces replacement
      ~ revision_id      = "apprevision-wm6b6" -> (known after apply)
        target_namespace = "cert-manager"
        template_name    = "cert-manager"
        template_version = "v1.0.1"
        wait             = true
    }

  # module.k8s_cluster.rancher2_app.ingress_nginx must be replaced
-/+ resource "rancher2_app" "ingress_nginx" {
      ~ annotations      = {
          - "lifecycle.cattle.io/create.helm-controller_c-ggvrb" = "true"
        } -> (known after apply)
        catalog_name     = "ingress-nginx"
      + description      = (known after apply)
      ~ external_id      = "catalog://?catalog=ingress-nginx&template=ingress-nginx&version=2.16.0" -> (known after apply)
        force_upgrade    = false
      ~ id               = "p-kdrr6:ingress-nginx" -> (known after apply)
      ~ labels           = {
          - "cattle.io/creator" = "norman"
        } -> (known after apply)
        name             = "ingress-nginx"
      ~ project_id       = "c-ggvrb:p-kdrr6" -> (known after apply) # forces replacement
      ~ revision_id      = "apprevision-8mxf8" -> (known after apply)
        target_namespace = "ingress-nginx"
        template_name    = "ingress-nginx"
        template_version = "2.16.0"
        values_yaml      = ""
        wait             = true
    }

  # module.k8s_cluster.rancher2_cluster.cluster will be updated in-place
  ~ resource "rancher2_cluster" "cluster" {
        annotations                = {
            "authz.management.cattle.io/creator-role-bindings"            = jsonencode(
                {
                    created  = [
                        "cluster-owner",
                    ]
                    required = [
                        "cluster-owner",
                    ]
                }
            )
            "field.cattle.io/overwriteAppAnswers"                         = jsonencode(
                {
                    answers = {
                        grafana.persistence.enabled         = "true"
                        grafana.persistence.size            = "10Gi"
                        grafana.persistence.storageClass    = "do-block-storage"
                        prometheus.persistence.enabled      = "true"
                        prometheus.persistence.size         = "10Gi"
                        prometheus.persistence.storageClass = "do-block-storage"
                        prometheus.retention                = "168h"
                    }
                }
            )
            "lifecycle.cattle.io/create.cluster-agent-controller-cleanup" = "true"
            "lifecycle.cattle.io/create.cluster-provisioner-controller"   = "true"
            "lifecycle.cattle.io/create.cluster-scoped-gc"                = "true"
            "lifecycle.cattle.io/create.mgmt-cluster-rbac-remove"         = "true"
            "provisioner.cattle.io/ke-driver-update"                      = "updated"
        }
        cluster_registration_token = [
            {
                annotations          = {}
                cluster_id           = "c-ggvrb"
                command              = ""
                id                   = "c-ggvrb:system"
                insecure_command     = ""
                labels               = {
                    "cattle.io/creator" = "norman"
                }
                manifest_url         = ""
                name                 = "system"
                node_command         = ""
                token                = ""
                windows_node_command = """
            },
        ]
        default_project_id         = "c-ggvrb:p-4h49x"
        description                = "Staging Cluster"
        docker_root_dir            = "/var/lib/docker"
        driver                     = "rancherKubernetesEngine"
        enable_cluster_alerting    = false
        enable_cluster_monitoring  = true
        enable_network_policy      = false
        id                         = "c-ggvrb"
        istio_enabled              = false
        kube_config                = (sensitive value)
        labels                     = {
            "cattle.io/creator" = "norman"
        }
        name                       = "k8s-cluster-stage"
        system_project_id          = "c-ggvrb:p-kdrr6"
        windows_prefered_cluster   = false

        cluster_auth_endpoint {
            enabled = false
        }

        cluster_monitoring_input {
            answers = {
                "grafana.persistence.enabled"         = "true"
                "grafana.persistence.size"            = "10Gi"
                "grafana.persistence.storageClass"    = "do-block-storage"
                "prometheus.persistence.enabled"      = "true"
                "prometheus.persistence.size"         = "10Gi"
                "prometheus.persistence.storageClass" = "do-block-storage"
                "prometheus.retention"                = "168h"
            }
        }

      ~ rke_config {
            addon_job_timeout     = 90
            addons                = <<~EOT
                ---
                apiVersion: v1
                kind: Secret
                metadata:
                  name: digitalocean
                  namespace: kube-system
                stringData:
                  access-token: ""
            EOT
            addons_include        = [
                "https://raw.githubusercontent.com/digitalocean/digitalocean-cloud-controller-manager/master/releases/v0.1.26.yml",
                "https://raw.githubusercontent.com/digitalocean/csi-digitalocean/master/deploy/kubernetes/releases/csi-digitalocean-v2.0.0/crds.yaml",
                "https://raw.githubusercontent.com/digitalocean/csi-digitalocean/master/deploy/kubernetes/releases/csi-digitalocean-v2.0.0/driver.yaml",
                "https://raw.githubusercontent.com/digitalocean/csi-digitalocean/master/deploy/kubernetes/releases/csi-digitalocean-v2.0.0/snapshot-controller.yaml",
            ]
            ignore_docker_version = true
            kubernetes_version    = "v1.18.6-rancher1-1"
            ssh_agent_auth        = false

            authentication {
                sans     = []
                strategy = "x509"
            }

            authorization {
                mode    = "rbac"
                options = {}
            }

            bastion_host {
                ssh_agent_auth = false
            }

          ~ cloud_provider {
              ~ custom_cloud_provider = <<~EOT
                    ---
                    apiVersion: v1
                    kind: Secret
                    metadata:
                      name: digitalocean
                      namespace: kube-system
                    stringData:
                      access-token: ""
                EOT
                name                  = "custom"
            }

            dns {
                node_selector        = {}
                provider             = "coredns"
                reverse_cidrs        = []
                upstream_nameservers = [
                    "1.1.1.1",
                    "8.8.4.4",
                ]
            }

            ingress {
                extra_args    = {}
                node_selector = {}
                options       = {}
                provider      = "none"
            }

            monitoring {
                node_selector = {}
                options       = {}
                provider      = "metrics-server"
                replicas      = 1
            }

            network {
                mtu     = 0
                options = {}
                plugin  = "canal"
            }

            services {
                etcd {
                    creation      = "12h"
                    external_urls = []
                    extra_args    = {
                        "election-timeout"   = "5000"
                        "heartbeat-interval" = "500"
                    }
                    extra_binds   = []
                    extra_env     = []
                    gid           = 0
                    retention     = "72h"
                    snapshot      = false
                    uid           = 0

                    backup_config {
                        enabled        = true
                        interval_hours = 12
                        retention      = 6
                        safe_timestamp = false

                        s3_backup_config {
                            access_key  = (sensitive value)
                            bucket_name = ""
                            endpoint    = "sfo2.digitaloceanspaces.com"
                            region      = "sfo2"
                            secret_key  = (sensitive value)
                        }
                    }
                }

                kube_api {
                    admission_configuration = {}
                    always_pull_images      = false
                    extra_args              = {
                        "cloud-provider"                  = ""
                        "feature-gates"                   = "VolumeSnapshotDataSource=true,CSIDriverRegistry=true"
                        "kubelet-preferred-address-types" = "InternalIP,ExternalIP,Hostname"
                    }
                    extra_binds             = []
                    extra_env               = []
                    pod_security_policy     = false
                    service_node_port_range = "30000-32767"
                }

                kube_controller {
                    extra_args  = {
                        "cloud-provider" = ""
                    }
                    extra_binds = []
                    extra_env   = []
                }

                kubelet {
                    extra_args                   = {
                        "cloud-provider" = "external"
                        "feature-gates"  = "VolumeSnapshotDataSource=true,CSIDriverRegistry=true"
                    }
                    extra_binds                  = []
                    extra_env                    = []
                    fail_swap_on                 = false
                    generate_serving_certificate = false
                }

                kubeproxy {}

                scheduler {}
            }

            upgrade_strategy {
                drain                        = false
                max_unavailable_controlplane = "1"
                max_unavailable_worker       = "10%"

                drain_input {
                    delete_local_data  = false
                    force              = false
                    grace_period       = -1
                    ignore_daemon_sets = true
                    timeout            = 120
                }
            }
        }
    }

  # module.k8s_cluster.rancher2_secret.cloudflare_token must be replaced
-/+ resource "rancher2_secret" "cloudflare_token" {
      ~ annotations  = {} -> (known after apply)
        data         = (sensitive value)
      ~ id           = "cert-manager:cloudflare-api-token" -> (known after apply)
      ~ labels       = {
          - "cattle.io/creator" = "norman"
        } -> (known after apply)
        name         = "cloudflare-api-token"
        namespace_id = "cert-manager"
      ~ project_id   = "c-ggvrb:p-kdrr6" -> (known after apply) # forces replacement
    }

Plan: 3 to add, 2 to change, 3 to destroy.

cc: @rawmind0

rawmind0 commented 4 years ago

This doesn't seem a regression. Depending how the datasource and its dependencies are defined , this may be the expected datasource behaviour. Terraform advices that datasources are updated on refresh stage (before apply) unless using "known after apply" values on them. Your ff plan is saying module.k8s_cluster.data.rancher2_project.system will be read during apply (config refers to values not yet known).

Is the datasource using any "known after apply" value?? Have you tried to tf refresh the datasource before plan/apply?? Have you applied the plan and the diff is being showed again?? For some reason, your defined datasource seems to be using any "known after apply" value, and tf is updating it on apply stage instead of refresh stage, forcing the update of dependant resources.

I've made some tests with same tf version, using a similar config,

data "rancher2_project" "system" {
  name = "System"
  cluster_id = <id>
}
resource "rancher2_secret" "test" {
  name = "test"
  project_id = data.rancher2_project.system.id
  data = {
    address = "test"
  }
}

The provider is working fine from previous v1.10.0 to current provider v1.10.3. Also tested provider upgrade, not generating any diff. As tf log shows, datasource is updated on refresh stage, before apply due to datasource values are known.

# terraform init 
Initializing the backend...

Initializing provider plugins...
- Using previously-installed terraform-providers/rancher2 v1.10.0

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

# terraform apply
data.rancher2_project.system: Refreshing state...

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # rancher2_secret.test will be created
  + resource "rancher2_secret" "test" {
      + annotations = (known after apply)
      + data        = (sensitive value)
      + id          = (known after apply)
      + labels      = (known after apply)
      + name        = "test"
      + project_id  = "<id>"
    }

Plan: 1 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

rancher2_secret.test: Creating...
rancher2_secret.test: Creation complete after 6s [id=<id>]

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

# terraform init
Initializing the backend...

Initializing provider plugins...
- Finding terraform-providers/rancher2 versions matching "1.10.2"...
- Installing terraform-providers/rancher2 v1.10.2...
- Installed terraform-providers/rancher2 v1.10.2 (signed by HashiCorp)

Warning: Additional provider information from registry

The remote registry returned warnings for
registry.terraform.io/terraform-providers/rancher2:
- For users on Terraform 0.13 or greater, this provider has moved to
rancher/rancher2. Please update your source in required_providers.

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

# terraform apply 

data.rancher2_project.system: Refreshing state... [id=<id>]
rancher2_secret.test: Refreshing state... [id=<id>]

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

#terraform init

Initializing the backend...

Initializing provider plugins...
- Finding terraform-providers/rancher2 versions matching "1.10.3"...
- Installing terraform-providers/rancher2 v1.10.3...
- Installed terraform-providers/rancher2 v1.10.3 (signed by HashiCorp)

Warning: Additional provider information from registry

The remote registry returned warnings for
registry.terraform.io/terraform-providers/rancher2:
- For users on Terraform 0.13 or greater, this provider has moved to
rancher/rancher2. Please update your source in required_providers.

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

# terraform apply
data.rancher2_project.system: Refreshing state... [id=<id>]
rancher2_secret.test: Refreshing state... [id=<id>]

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

BTW, as you are using system project, have you consider to use rancher2_cluster.cluster.system_project_id attribute instead of datasource??

lots0logs commented 4 years ago

I will look at this again on Monday and let you know. Thanks!

lots0logs commented 4 years ago

I think my conclusion is incorrect. IF there is a bug its not what I thought. I will open a new issue if needed once I get to the bottom of it.

lots0logs commented 4 years ago

@rawmind0 I have a hunch as to what could be the problem.

Here is the data source in question:

data "rancher2_project" "system" {
    cluster_id = rancher2_cluster_sync.cluster.cluster_id
    name       = "System"
}

You are right that it is by design that Terraform is behaving this way since the value of the cluster_id argument comes from the output of another resource. I wonder if its really necessary to require providing the cluster id though? Could that data provider simply take the name argument similar to how the rancher2_cluster data provider works?

rawmind0 commented 4 years ago

You are right that it is by design that Terraform is behaving this way since the value of the cluster_id argument comes from the output of another resource. I wonder if its really necessary to require providing the cluster id though? Could that data provider simply take the name argument similar to how the rancher2_cluster data provider works?

Nope, due to project is scoped within cluster. As mentioned, have you consider to use rancher2_cluster.cluster.system_project_id attribute instead of datasource??

lots0logs commented 4 years ago

As mentioned, have you consider to use rancher2_cluster.cluster.system_project_id attribute instead of datasource??

Yes, thank you for that btw!