forseti install is not idempotent

[Jean-Mercier]

Hello,

When we launch terraform multiple time there is always modification because he find a md5 modification

Plan: 1 to add, 2 to change, 1 to destroy.

Terraform will perform the following actions:

  # module.forseti.data.http.server_config_contents will be read during apply
  # (config refers to values not yet known)
 <= data "http" "server_config_contents"  {
      + body            = (known after apply)
      + id              = (known after apply)
      + request_headers = {
          + "Content-MD5" = "BaeHfxvS5zbUjhoUCs7WbQ=="
        }
      + url             = "https://storage.googleapis.com/forseti-server-a9d259e3/configs/forseti_conf_server.yaml?GoogleAccessId=myaccount&Expires=1594300780&Signature=*******"
    }

  # module.forseti.data.kubernetes_service.forseti_server will be read during apply
  # (config refers to values not yet known)
 <= data "kubernetes_service" "forseti_server"  {
      + id                    = (known after apply)
      + load_balancer_ingress = (known after apply)
      + spec                  = (known after apply)

      + metadata {
          + generation       = (known after apply)
          + name             = "forseti-server"
          + namespace        = "forseti-a9d259e3"
          + resource_version = (known after apply)
          + self_link        = (known after apply)
          + uid              = (known after apply)
        }
    }

  # module.forseti.helm_release.forseti-security will be updated in-place
  ~ resource "helm_release" "forseti-security" {
        chart            = "forseti-security"
        disable_webhooks = false
        force_update     = false
        id               = "forseti"
        metadata         = [
            {
                chart     = "forseti-security"
                name      = "forseti"
                namespace = "forseti-a9d259e3"
                revision  = 2
                values    = <<~EOT
                    configValidator:
                      enabled: false
                      image: gcr.io/forseti-containers/config-validator
                      imageTag: "572e207"
                      networkPolicy:
                        enabled: true
                      policyLibrary:
                        bucket: forseti-server-a9d259e3
                        gitSync:
                          enabled: false
                          image: gcr.io/google-containers/git-sync
                          imageTag: v3.1.2
                          privateSSHKey: ""
                          wait: 30
                        repositoryBranch: master
                        repositoryURL: ""
                      workloadIdentity: forseti-client-gcp-a9d259e3@My-project.iam.gserviceaccount.com
                    database:
                      connectionName: My-project:europe-west1:forseti-server-db-a9d259e3
                      name: forseti_security
                      password: ********==
                      username: *******
                    networkPolicy:
                      enabled: true
                      ingressCidr:
                      - 10.1.0.0/20
                    nodeSelectors:
                    - cloud.google.com/gke-nodepool=default-node-pool
                    orchestrator:
                      image: gcr.io/forseti-containers/forseti
                      imageTag: v2.25.1
                      runFrequency: 51 */2 * * *
                      workloadIdentity: forseti-client-gcp-a9d259e3@My-project.iam.gserviceaccount.com
                    production: true
                    server:
                      cloudProfilerEnabled: false
                      config:
                        bucket: forseti-server-a9d259e3
                        contents: **************++
                      image: gcr.io/forseti-containers/forseti
                      imageTag: v2.25.1
                      loadBalancer: internal
                      logLevel: info
                      workloadIdentity: forseti-server-gcp-a9d259e3@My-project.iam.gserviceaccount.com
                EOT
                version   = "2.2.0"
            },
        ]
        name             = "forseti"
        namespace        = "forseti-a9d259e3"
        recreate_pods    = true
        repository       = "https://forseti-security-charts.storage.googleapis.com/release/"
        reuse            = false
        reuse_values     = false
        status           = "DEPLOYED"
        timeout          = 300
        values           = [
            "nodeSelectors: ['cloud.google.com/gke-nodepool=default-node-pool']",
        ]
        verify           = false
        version          = "2.2.0"
        wait             = true

        set {
            name  = "configValidator.enabled"
            value = "false"
        }
        set {
            name  = "configValidator.image"
            value = "gcr.io/forseti-containers/config-validator"
        }
        set {
            name  = "configValidator.imageTag"
            value = "572e207"
        }
        set {
            name  = "configValidator.networkPolicy.enabled"
            value = "true"
        }
        set {
            name  = "configValidator.policyLibrary.bucket"
            value = "forseti-server-a9d259e3"
        }
        set {
            name  = "configValidator.policyLibrary.gitSync.enabled"
            value = "false"
        }
        set {
            name  = "configValidator.policyLibrary.gitSync.image"
            value = "gcr.io/google-containers/git-sync"
        }
        set {
            name  = "configValidator.policyLibrary.gitSync.imageTag"
            value = "v3.1.2"
        }
        set {
            name  = "configValidator.policyLibrary.gitSync.wait"
            value = "30"
        }
        set {
            name  = "configValidator.policyLibrary.repositoryBranch"
            value = "master"
        }
        set {
            name = "configValidator.policyLibrary.repositoryURL"
        }
        set {
            name  = "configValidator.workloadIdentity"
            value = "forseti-client-gcp-a9d259e3@My-project.iam.gserviceaccount.com"
        }
        set {
            name  = "database.connectionName"
            value = "My-project:europe-west1:forseti-server-db-a9d259e3"
        }
        set {
            name  = "database.name"
            value = "forseti_security"
        }
        set {
            name  = "database.username"
            value = "*******"
        }
        set {
            name  = "networkPolicy.enabled"
            value = "true"
        }
        set {
            name  = "networkPolicy.ingressCidr"
            value = "{10.1.0.0/20}"
        }
        set {
            name  = "orchestrator.image"
            value = "gcr.io/forseti-containers/forseti"
        }
        set {
            name  = "orchestrator.imageTag"
            value = "v2.25.1"
        }
        set {
            name  = "orchestrator.runFrequency"
            value = "51 */2 * * *"
        }
        set {
            name  = "orchestrator.workloadIdentity"
            value = "forseti-client-gcp-a9d259e3@My-project.iam.gserviceaccount.com"
        }
        set {
            name  = "production"
            value = "true"
        }
        set {
            name  = "server.cloudProfilerEnabled"
            value = "false"
        }
        set {
            name  = "server.config.bucket"
            value = "forseti-server-a9d259e3"
        }
        set {
            name  = "server.image"
            value = "gcr.io/forseti-containers/forseti"
        }
        set {
            name  = "server.imageTag"
            value = "v2.25.1"
        }
        set {
            name  = "server.loadBalancer"
            value = "internal"
        }
        set {
            name  = "server.logLevel"
            value = "info"
        }
        set {
            name  = "server.workloadIdentity"
            value = "forseti-server-gcp-a9d259e3@My-project.iam.gserviceaccount.com"
        }

        set_sensitive {
            name = "configValidator.policyLibrary.gitSync.privateSSHKey"
        }
        set_sensitive {
            name  = "database.password"
            value = (sensitive value)
        }

      - set_string {
          - name  = "server.config.contents" -> null
          - value = "++*******" -> null
        }
      + set_string {
          + name  = "server.config.contents"
          + value = (known after apply)
        }
    }

  # module.gke.google_container_cluster.primary will be updated in-place
  ~ resource "google_container_cluster" "primary" {
        additional_zones            = []
        cluster_ipv4_cidr           = "10.2.0.0/20"
        default_max_pods_per_node   = 110
        enable_binary_authorization = false
        enable_intranode_visibility = false
        enable_kubernetes_alpha     = false
        enable_legacy_abac          = false
        enable_shielded_nodes       = false
        enable_tpu                  = false
        endpoint                    = "********"
        id                          = "projects/My-project/locations/europe-west1/clusters/forseti-cluster"
        initial_node_count          = 0
        instance_group_urls         = [
            "https://www.googleapis.com/compute/beta/projects/My-project/zones/europe-west1-c/instanceGroups/gke-forseti-cluster-default-node-pool-f00ae016-grp",
            "https://www.googleapis.com/compute/beta/projects/My-project/zones/europe-west1-b/instanceGroups/gke-forseti-cluster-default-node-pool-017cbf54-grp",
            "https://www.googleapis.com/compute/beta/projects/My-project/zones/europe-west1-d/instanceGroups/gke-forseti-cluster-default-node-pool-afb6cece-grp",
        ]
        location                    = "europe-west1"
        logging_service             = "logging.googleapis.com/kubernetes"
        master_version              = "1.15.11-gke.15"
        min_master_version          = "1.15.11-gke.15"
        monitoring_service          = "monitoring.googleapis.com/kubernetes"
        name                        = "forseti-cluster"
        network                     = "projects/My-project/global/networks/forseti-gke-network"
        node_locations              = [
            "europe-west1-b",
            "europe-west1-c",
            "europe-west1-d",
        ]
        node_version                = "1.15.11-gke.15"
        project                     = "My-project"
        remove_default_node_pool    = true
        resource_labels             = {}
        services_ipv4_cidr          = "10.3.0.0/20"
        subnetwork                  = "projects/My-project/regions/europe-west1/subnetworks/gke-sub-network"

        addons_config {

            horizontal_pod_autoscaling {
                disabled = false
            }

            http_load_balancing {
                disabled = false
            }

            istio_config {
                disabled = true
            }

            network_policy_config {
                disabled = false
            }
        }

        cluster_autoscaling {
            enabled = false
        }

        database_encryption {
            state = "DECRYPTED"
        }

        ip_allocation_policy {
            cluster_ipv4_cidr_block       = "10.2.0.0/20"
            cluster_secondary_range_name  = "gke-pod-ip-range"
            create_subnetwork             = false
            services_ipv4_cidr_block      = "10.3.0.0/20"
            services_secondary_range_name = "gke-service-ip-range"
            use_ip_aliases                = false
        }

        maintenance_policy {
            daily_maintenance_window {
                duration   = "PT4H0M0S"
                start_time = "05:00"
            }
        }

        master_auth {
            cluster_ca_certificate = "*******"

            client_certificate_config {
                issue_client_certificate = false
            }
        }

        network_policy {
            enabled  = true
            provider = "CALICO"
        }

        node_config {
            disk_size_gb      = 100
            disk_type         = "pd-ssd"
            guest_accelerator = []
            image_type        = "COS"
            labels            = {
                "cluster_name" = "forseti-cluster"
                "node_pool"    = "default-node-pool"
            }
            local_ssd_count   = 0
            machine_type      = "e2-standard-4"
            metadata          = {
                "cluster_name"             = "forseti-cluster"
                "disable-legacy-endpoints" = "true"
                "node_pool"                = "default-node-pool"
            }
            oauth_scopes      = [
                "https://www.googleapis.com/auth/cloud-platform",
            ]
            preemptible       = false
            service_account   = "tf-gke-forseti-cluster-ja9h@My-project.iam.gserviceaccount.com"
            tags              = [
                "gke-forseti-cluster",
                "gke-forseti-cluster-default-node-pool",
            ]
            taint             = []

            shielded_instance_config {
                enable_integrity_monitoring = true
                enable_secure_boot          = false
            }

            workload_metadata_config {
                node_metadata = "GKE_METADATA_SERVER"
            }
        }

        node_pool {
            initial_node_count  = 1
            instance_group_urls = [
                "https://www.googleapis.com/compute/v1/projects/My-project/zones/europe-west1-c/instanceGroupManagers/gke-forseti-cluster-default-node-pool-f00ae016-grp",
                "https://www.googleapis.com/compute/v1/projects/My-project/zones/europe-west1-b/instanceGroupManagers/gke-forseti-cluster-default-node-pool-017cbf54-grp",
                "https://www.googleapis.com/compute/v1/projects/My-project/zones/europe-west1-d/instanceGroupManagers/gke-forseti-cluster-default-node-pool-afb6cece-grp",
            ]
            max_pods_per_node   = 110
            name                = "default-node-pool"
            node_count          = 1
            node_locations      = [
                "europe-west1-b",
                "europe-west1-c",
                "europe-west1-d",
            ]
            version             = "1.15.11-gke.15"

            autoscaling {
                max_node_count = 1
                min_node_count = 1
            }

            management {
                auto_repair  = true
                auto_upgrade = false
            }

            node_config {
                disk_size_gb      = 100
                disk_type         = "pd-ssd"
                guest_accelerator = []
                image_type        = "COS"
                labels            = {
                    "cluster_name" = "forseti-cluster"
                    "node_pool"    = "default-node-pool"
                }
                local_ssd_count   = 0
                machine_type      = "e2-standard-4"
                metadata          = {
                    "cluster_name"             = "forseti-cluster"
                    "disable-legacy-endpoints" = "true"
                    "node_pool"                = "default-node-pool"
                }
                oauth_scopes      = [
                    "https://www.googleapis.com/auth/cloud-platform",
                ]
                preemptible       = false
                service_account   = "tf-gke-forseti-cluster-ja9h@My-project.iam.gserviceaccount.com"
                tags              = [
                    "gke-forseti-cluster",
                    "gke-forseti-cluster-default-node-pool",
                ]
                taint             = []

                shielded_instance_config {
                    enable_integrity_monitoring = true
                    enable_secure_boot          = false
                }

                workload_metadata_config {
                    node_metadata = "GKE_METADATA_SERVER"
                }
            }

            upgrade_settings {
                max_surge       = 1
                max_unavailable = 0
            }
        }

      + pod_security_policy_config {
          + enabled = false
        }

        release_channel {
            channel = "UNSPECIFIED"
        }

        timeouts {
            create = "30m"
            delete = "30m"
            update = "30m"
        }

        vertical_pod_autoscaling {
            enabled = false
        }

        workload_identity_config {
            identity_namespace = "My-project.svc.id.goog"
        }
    }

  # module.forseti.module.client_config.data.template_file.forseti_client_config[0] will be read during apply
  # (config refers to values not yet known)
 <= data "template_file" "forseti_client_config"  {
      + id       = (known after apply)
      + rendered = (known after apply)
      + template = <<~EOT
            server_ip: ${forseti_server_ip}
        EOT
      + vars     = {
          + "forseti_server_ip" = (known after apply)
        }
    }

  # module.forseti.module.client_config.google_storage_bucket_object.forseti_client_config[0] must be replaced
-/+ resource "google_storage_bucket_object" "forseti_client_config" {
        bucket         = "forseti-client-a9d259e3"
      ~ content        = (sensitive value)
      ~ content_type   = "text/plain; charset=utf-8" -> (known after apply)
      ~ crc32c         = "j0sMtQ==" -> (known after apply)
      ~ detect_md5hash = "8iBFXHZU8vM8FeB2LAl6XA==" -> "different hash" # forces replacement
      ~ id             = "forseti-client-a9d259e3-configs/forseti_conf_client.yaml" -> (known after apply)
      ~ md5hash        = "8iBFXHZU8vM8FeB2LAl6XA==" -> (known after apply)
        name           = "configs/forseti_conf_client.yaml"
      ~ output_name    = "configs/forseti_conf_client.yaml" -> (known after apply)
      ~ self_link      = "https://www.googleapis.com/storage/v1/b/forseti-client-a9d259e3/o/configs%2Fforseti_conf_client.yaml" -> (known after apply)
      ~ storage_class  = "STANDARD" -> (known after apply)
    }

Plan: 1 to add, 2 to change, 1 to destroy.

[gkowalski-google]

Hi @Jean-Mercier, thanks for raising this up, I was not aware of that. It looks like some of the values are determined at apply, and they are used in the configs, which have the md5 hash as you mentioned. Is this causing any issues for you?

[Jean-Mercier]

md5 it's for this ressource module.forseti.module.client_config.google_storage_bucket_object.forseti_client_config[0] must be replaced

for this ressource module.gke.google_container_cluster.primary will be updated in-place it's the pod security policy

  + pod_security_policy_config {
      + enabled = false
    }

Install is going to the end i have a forseti server who start and tried to make inventory (it's failed for gsuite group)

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 15 days if no further activity occurs. Thank you for your contributions.