Cloud Run env variables reordering causing resource to update in-place

[skreinberg97]

Our cloud run resource needs to be updated every run as the environment variables are reordering. This seems to be completely on the Terraform side as our actual Cloud Run instance does not seem to be affected.

Terraform Version

v0.13.2

Affected Resource(s)

• google_cloud_run_service

Terraform Configuration Files

 containers {
        image = "XXXXX"

        env {
          name = "ENV_1"
          value = data.google_secret_manager_secret_version.secrets["XXXX"].secret_data
        }

        env {
          name = "ENV_2"
          value = data.google_secret_manager_secret_version.secrets["XXXX"].secret_data
        }
}

Expected Behavior

For the environment variables to be in the same order, leading to no change in the resource.

Actual Behavior

~ env {
      ~ name  = "ENV_1" -> "ENV_2"
      ~ value = "XXX" -> "XXXX"
}
~ env {
      ~ name  = "ENV_2" -> "ENV_1"
      ~ value = "XXXX" -> "XXX"
}

Steps to Reproduce

1. terraform apply and/or terraform plan

Important Factoids

• environment variables are coming from google_secret_manager_secret_version data object

References

b/272365080

[venkykuberan]

I don't see that happening for the sample config below

  name     = "cloudrun-srv"
  location = "us-central1"

  template {
    spec {
      containers {
        image = "gcr.io/cloudrun/hello"
        env {
          name = "SOURCE"
          value = "remote"
        }
        env {
          name = "TARGET"
          value = "home"
        }
      }
    }
  }

  traffic {
    percent         = 100
    latest_revision = true
  }
  autogenerate_revision_name = true
} 

Can you please attach the plan output and debug log for your apply

[skreinberg97]

Hey! Thanks for your help, hopefully this is useful. I believe the issue is caused by the google_secret_manager_secret_version data object -- if the environment variables are hard-coded in plain text then we do not see the same behavior

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # google_cloud_run_service.default will be updated in-place
  ~ resource "google_cloud_run_service" "default" {
        id       = "REDACTED"
        location = "us-central1"
        name     = "terraform"
        project  = "REDACTED"
        status   = [
            {
                conditions                   = [
                    {
                        message = ""
                        reason  = ""
                        status  = "True"
                        type    = "Ready"
                    },
                    {
                        message = ""
                        reason  = ""
                        status  = "True"
                        type    = "ConfigurationsReady"
                    },
                    {
                        message = ""
                        reason  = ""
                        status  = "True"
                        type    = "RoutesReady"
                    },
                ]
                latest_created_revision_name = "terraform-bdff6"
                latest_ready_revision_name   = "terraform-bdff6"
                observed_generation          = 4
                url                          = "REDACTED"
            },
        ]

        metadata {
            annotations      = {
                "serving.knative.dev/creator"      = "REDACTED"
                "serving.knative.dev/lastModifier" = "REDACTED"
            }
            generation       = 4
            labels           = {
                "cloud.googleapis.com/location" = "us-central1"
            }
            namespace        = "REDACTED"
            resource_version = "AAWw4zi+fg0"
            self_link        = "REDACTED"
            uid              = "77fe171e-b695-427c-9eb0-833643c1523e"
        }

      ~ template {
            metadata {
                annotations = {
                    "autoscaling.knative.dev/maxScale"        = "1000"
                    "run.googleapis.com/vpc-access-connector" = "REDACTED"
                }
                generation  = 0
                labels      = {}
            }

          ~ spec {
                container_concurrency = 20
                service_account_name  = "REDACTED"

              ~ containers {
                    args    = []
                    command = []
                    image   = "REDACTED"

                  ~ env {
                      ~ name  = "ENV" -> "EXISTING_ENV"
                      ~ value = "EXISTING_ENV_VALUE" -> "ENV_VALUE"
                    }

                  ~ env {
                      ~ name  = "ENV" -> "EXISTING_ENV"
                      ~ value = "EXISTING_ENV_VALUE" -> "ENV_VALUE"
                    }

                   ~ env {
                      ~ name  = "ENV" -> "EXISTING_ENV"
                      ~ value = "EXISTING_ENV_VALUE" -> "ENV_VALUE"
                    }

                    ~ env {
                      ~ name  = "ENV" -> "EXISTING_ENV"
                      ~ value = "EXISTING_ENV_VALUE" -> "ENV_VALUE"
                    }

                   ~ env {
                      ~ name  = "ENV" -> "EXISTING_ENV"
                      ~ value = "EXISTING_ENV_VALUE" -> "ENV_VALUE"
                    }

                 ~ env {
                      ~ name  = "ENV" -> "EXISTING_ENV"
                      ~ value = "EXISTING_ENV_VALUE" -> "ENV_VALUE"
                    }
                    resources {
                        limits   = {
                            "cpu"    = "1000m"
                            "memory" = "256Mi"
                        }
                        requests = {}
                    }
                }
            }
        }

        traffic {
            latest_revision = true
            percent         = 100
        }
    }

Debug Output

2020/10/08 17:12:42 [DEBUG] command: asking for input: "\nDo you want to perform these actions in workspace \"REDACTED"?"

[venkykuberan]

Please attach the debug output, I want to see API request/response.

[blueyed]

I assume that it is using the data source, isn't it? (https://registry.terraform.io/providers/hashicorp/google/latest/docs/data-sources/secret_manager_secret_version)

If so, I could not verify this to be an issue, using:

Terraform v1.0.8
on linux_amd64
+ provider registry.terraform.io/hashicorp/google v3.58.0
+ provider registry.terraform.io/hashicorp/google-beta v3.86.0
+ provider registry.terraform.io/hashicorp/random v3.1.0

@skreinberg97 is it still an issue for you? If so, can you provide more information, please?

[markesha]

Hi @blueyed, I am experiencing the same with:

Terraform v1.0.9
on darwin_amd64
+ provider registry.terraform.io/hashicorp/google v3.89.0
+ provider registry.terraform.io/hashicorp/google-beta v3.89.0
+ provider registry.terraform.io/hashicorp/vault v2.24.1

In my case, it is google_secret_manager_secret_version resource, not data source.

Terraform will perform the following actions:
# google_cloud_run_service.service will be updated in-place
  ~ resource "google_cloud_run_service" "service" {
        id                         = "locations/region/namespaces/project/services/service"
        name                       = "service"
        # (4 unchanged attributes hidden)
      ~ template {

          ~ spec {
                # (3 unchanged attributes hidden)

              ~ containers {
                    # (2 unchanged attributes hidden)

                  - env {
                      - name = "ENV_VAR" -> null

                      - value_from {
                          - secret_key_ref {
                              - key  = "1" -> null
                              - name = "secret_manager_secret_name" -> null
                            }
                        }
                    }
                  + env {
                      + name = "ENV_VAR"

                      + value_from {
                          + secret_key_ref {
                              + key  = "1"
                              + name = "secret_manager_secret_name"
                            }
                        }
                    }

2021-10-19T12:59:44.812+0200 [INFO]  provider.terraform-provider-google-beta_v3.89.0_x5: 2021/10/19 12:59:44 [DEBUG] Retry Transport: Stopping retries, last request was successful: timestamp=2021-10-19T12:59:44.812+0200
2021-10-19T12:59:44.812+0200 [INFO]  provider.terraform-provider-google-beta_v3.89.0_x5: 2021/10/19 12:59:44 [DEBUG] Retry Transport: Returning after 1 attempts: timestamp=2021-10-19T12:59:44.812+0200
2021-10-19T12:59:44.816+0200 [WARN]  Provider "registry.terraform.io/hashicorp/google-beta" produced an invalid plan for google_secret_manager_secret_version.vault_secrets["ENV_VAR"], but we are tolerating it because it is using the legacy plugin SDK.
    The following problems may be the cause of any confusing errors from downstream operations:
      - .enabled: planned value cty.True for a non-computed attribute
2021-10-19T12:59:44.818+0200 [INFO]  provider.terraform-provider-google-beta_v3.89.0_x5: 2021/10/19 12:59:44 [DEBUG] Google API Response Details:
---[ RESPONSE ]--------------------------------------
HTTP/2.0 200 OK

2021-10-19T12:59:46.016+0200 [INFO]  provider.terraform-provider-google-beta_v3.89.0_x5: 2021/10/19 12:59:46 [DEBUG] Retry Transport: Returning after 1 attempts: timestamp=2021-10-19T12:59:46.016+0200
2021-10-19T12:59:46.046+0200 [WARN]  Provider "registry.terraform.io/hashicorp/google-beta" produced an unexpected new value for google_cloud_run_service.service during refresh.
      - .template[0].spec[0].containers[0].env: planned set element cty.ObjectVal(map[string]cty.Value{"name":cty.StringVal("ENV_VAR"), "value":cty.NullVal(cty.String), "value_from":cty.ListVal([]cty.Value{cty.ObjectVal(map[string]cty.Value{"secret_key_ref":cty.ListVal([]cty.Value{cty.ObjectVal(map[string]cty.Value{"key":cty.StringVal("2"), "name":cty.StringVal("secret_manager_secret_name")})})})})}) does not correlate with any element in actual

2021-10-19T12:59:46.128+0200 [WARN]  Provider "registry.terraform.io/hashicorp/google-beta" produced an invalid plan for google_cloud_run_service.service, but we are tolerating it because it is using the legacy plugin SDK.
    The following problems may be the cause of any confusing errors from downstream operations:
      - .traffic: block count in plan (1) disagrees with count in config (0)
      - .template[0].spec[0].containers[0].args: planned value cty.ListValEmpty(cty.String) for a non-computed attribute
      - .template[0].spec[0].containers[0].working_dir: planned value cty.StringVal("") for a non-computed attribute
      - .template[0].spec[0].containers[0].ports[0].protocol: planned value cty.StringVal("") for a non-computed attribute
2021-10-19T12:59:46.195+0200 [DEBUG] provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = transport is closing"
2021-10-19T12:59:46.198+0200 [DEBUG] provider: plugin process exited: path=.terraform/providers/registry.terraform.io/hashicorp/google-beta/3.89.0/darwin_amd64/terraform-provider-google-beta_v3.89.0_x5 pid=43591
2021-10-19T12:59:46.198+0200 [DEBUG] provider: plugin exited
2021-10-19T12:59:46.199+0200 [INFO]  backend/local: plan operation completed

[quulah]

Encountering the same thing. We also have some Secret versions being managed directly, not via a data source.

For those I have generated an UUID as the name prefixed with secret-, since that's the way the UI / cloud side of Cloud Run seems to handle things. Not sure if it has an effect on this.

In any case, even the non-secret environment variables are shuffled around.

I wonder if we could just sort the environment variables a certain way to keep Terraform state happy and have the order be the same during all runs.

[jeremad]

It looks it is fixed by using https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/cloud_run_v2_service

[FabianFrank]

This also reproduces with google_cloud_run_v2_service if you have a value_source of secret_key_ref.

[TheXienator]

I am also having this issue still

[alina-bylkova]

Still having same issue using google_cloud_run_v2_service resource

[FredoPhxlabs]

Same issue here with google_cloud_run_v2_service

[dominik1001]

A temporary workaround is to delete all env variables in the UI and then terraform apply again. But would really be nice to see this fixed.

[heidi-manish]

facing same problem with google_cloud_run_v2_service

[XGManuelJager]

The same is happening to me, but with secrets volumes

[xiujuan-li]

This my terraform version. I have the same problem. Is there any solution for help?

[xiujuan-li]

I found this env change happend after import cloud_run_v2_service resourve,looks like import changed the env sort in terraform state ***.tfstate .

~ env {
      ~ name  = "ENV_1" -> "ENV_2"
      ~ value = "XXX" -> "XXXX"
}
~ env {
      ~ name  = "ENV_2" -> "ENV_1"
      ~ value = "XXXX" -> "XXX"
}

[meer-online]

facing same problem with google_cloud_run_v2_service, was anyone able to figure out a solution?

[scuba-st3v3]

Same issue here

[duksis]

Same here and also for regular variables - no secrets

[ajoy39]

Throwing my voice into the void here, same issue, spot checking ENV VAR changes gets really hard when you have more than a handful of them.