Unable to apply labels to dataflow flex template jobs

[napierk]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
• Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.
• If an issue is assigned to the modular-magician user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned to hashibot, a community member has claimed the issue already.

Terraform Version

1.4.0 Provider version: hashicorp/google-beta v5.1.0

Affected Resource(s)

• google_dataflow_flex_template_job

Terraform Configuration Files

locals {
  local_labels = {
    keya = "valuea"
    keyb = "valueb"
    keyc = "valuec"
  }
}

resource "google_dataflow_flex_template_job" "debug_job" {
  provider  = google-beta
  project   = "dummy-project"
  region    = "australia-southeast1"
  on_delete = "drain"
  name      = "debug-dataflow-job"

  labels    = local.local_labels

  # Dataflow Configuration
  container_spec_gcs_path = "gs://dummy-bucket/metadata.json"
  subnetwork              = "https://www.googleapis.com/compute/v1/projects/dummy-network-project/regions/australia-southeast1/subnetworks/dummy-dataflow-subnet"
  ip_configuration        = "WORKER_IP_PRIVATE"
  temp_location           = "gs://dummy-bucket/temp"
  staging_location        = "gs://dummy-bucket/staging"
  autoscaling_algorithm  = "THROUGHPUT_BASED"
  service_account_email  = "dummy-sa@dummy-project.iam.gserviceaccount.com"
  additional_experiments = ["use_runner_v2"]
}

Expected Behavior

Terraform apply should apply the labels to the dataflow job when it is up and running

Actual Behavior

Effective labels are shown in the terraform plan:

terraform plan
...
resource "google_dataflow_flex_template_job" "debug_job" {
  ...
  + effective_labels             = {
      + "keya" = "valuea"
      + "keyb" = "valueb"
      + "keyc" = "valuec"
    }
  + labels                       = {
      + "keya" = "valuea"
      + "keyb" = "valueb"
      + "keyc" = "valuec"
    }
    name                         = "debug-dataflow-job"
  + terraform_labels             = {
      + "keya" = "valuea"
      + "keyb" = "valueb"
      + "keyc" = "valuec"
    }
}

No labels are applied to the dataflow job after terraform apply. A further terraform plan shows that that labels still need to be applied:

terraform plan
...
- resource "google_dataflow_flex_template_job" "debug_job" {
  ...
  ~ effective_labels             = {
      + "keya" = "valuea"
      + "keyb" = "valueb"
      + "keyc" = "valuec"
    }
  ~ labels                       = {
      + "keya" = "valuea"
      + "keyb" = "valueb"
      + "keyc" = "valuec"
    }
    name                         = "debug-dataflow-job"
  ~ terraform_labels             = {
      + "keya" = "valuea"
      + "keyb" = "valueb"
      + "keyc" = "valuec"
    }
}

Steps to Reproduce

1. Configure a flex template dataflow job
2. terraform plan and check for labels in the resource output
3. terraform apply and check labels applied to the dataflow job resource

Important Factoids

It is possible to apply labels to the dataflow job by commenting the labels field and using the custom parameters in the resource as below:

  # labels = local.local_labels

  # Custom Parameters
  parameters = {
    # terraform unable to apply labels to dataflow job, instead need to
    # apply them via the dataflow runtime environment variables, see:
    # https://github.com/hashicorp/terraform-provider-google/issues/9291#issuecomment-1194419808
    labels = jsonencode(local.local_labels)
  }

However, this results in the following error when making further terraform apply operations to the same resource:

Error: googleapi: Error 400: The template parameters are invalid. Details: 
labels: Runtime parameter labels should not be specified in both parameters field and environment field. Specifying runtime parameters in environment field is recommended.
Details:
[
  {
    "@type": "type.googleapis.com/google.dataflow.v1beta3.InvalidTemplateParameters",
    "parameterViolations": [
      {
        "description": "Runtime parameter labels should not be specified in both parameters field and environment field. Specifying runtime parameters in environment field is recommended.",
        "parameter": "labels"
      }
    ]
  }
]
, badRequest

References

• Similar issue is open: https://github.com/hashicorp/terraform-provider-google/issues/9291

b/304967518

[iuliananz]

Any idea when this'll get fixed please?

[bvolpato]

I haven't setup this myself, but can you pass in the labels inside the parameters?

It seems that the high level is propagating into 2 different places

[napierk]

hey @bvolpato , we tried passing them in this fashion, however it made the dataflow non-idempotent and failed on the next apply. Details are in the important factoids above.