Error Applying Default Tags in Datadog Monitor with Established Monitor Tag Policy

Datadog Terraform Provider Version

3.48.0

Terraform Version

1.9.3

What resources or data sources are affected?

datadog_monitor

Terraform Configuration Files

terraform {
  required_version = "~> 1.9.0"

  required_providers {
    datadog = {
      source  = "DataDog/datadog"
      version = "~> 3"
    }
  }
}

provider "datadog" {
  api_key = ####
  app_key = ####
  api_url = "https://api.datadoghq.eu/"
  default_tags {
    tags = {
      service : "lambda-pdf"
      env : "production"
      aws_account : "doctrine-production"
      repository : "lambda-pdf"
    }
  }  
}

resource "datadog_monitor" "error_rate_too_high" {
  escalation_message = ""
  evaluation_delay   = "900"
  include_tags       = "true"
  message            = "@slack-errors_legal_graph"

  monitor_thresholds {
    critical = "0"
  }

  name                 = "[${var.aws_account_name}] [Lambda] PDF Generator : error rate too high on {{functionname.name}}"
  new_group_delay      = "60"
  no_data_timeframe    = "0"
  notify_audit         = "false"
  notify_no_data       = "false"
  priority             = "0"
  query                = "sum(last_5m):max:aws.lambda.errors{functionname:${var.lambda_function_name_or_regex}, aws_account:${var.aws_account_name}} by {functionname}.as_count() > 0"
  renotify_interval    = "0"
  renotify_occurrences = "0"
  require_full_window  = "false"
  timeout_h            = "0"
  type                 = "query alert"

}

resource "datadog_monitor_config_policy" "ensure_monitor_tag_terraform" {
  policy_type = "tag"

  tag_policy {
    tag_key          = "repository"
    tag_key_required = true
    valid_tag_values = ["*"]
  }
}

resource "datadog_monitor_config_policy" "ensure_monitor_tag_aws_account" {
  policy_type = "tag"

  tag_policy {
    tag_key          = "aws_account"
    tag_key_required = true
    valid_tag_values = [
      "doctrine-production",
      "aws-legacy-production",
      "doctrine-shared",
      "doctrine-staging",
      "doctrine-dev"
    ]
  }
}

resource "datadog_monitor_config_policy" "ensure_monitor_tag_env" {
  policy_type = "tag"

  tag_policy {
    tag_key          = "env"
    tag_key_required = true
    valid_tag_values = ["*"]
  }
}

resource "datadog_monitor_config_policy" "ensure_monitor_tag_service" {
  policy_type = "tag"

  tag_policy {
    tag_key          = "service"
    tag_key_required = true
    valid_tag_values = ["*"]
  }
}

Relevant debug or panic output

data.vault_generic_secret.datadog_vault_path: Reading...
module.vault_engine_kvv2.vault_mount.this: Refreshing state... [id=/secret/lambda_pdf/lambda_pdf/kv]
data.aws_caller_identity.current: Reading...
module.bucket_lambda_pdf.aws_s3_bucket.this: Refreshing state... [id=lambda-pdf-doctrine-production]
data.aws_caller_identity.current: Read complete after 0s [id=756057317366]
aws_iam_role.this: Refreshing state... [id=pdf-generator-production]
data.vault_generic_secret.datadog_vault_path: Read complete after 0s [id=secret/lambda_pdf/lambda_pdf/kv/datadog]
module.vault_engine_kvv2.vault_kv_secret_backend_v2.this: Refreshing state... [id=/secret/lambda_pdf/lambda_pdf/kv/config]
module.datadog_monitor.datadog_monitor.error_rate_too_high: Refreshing state... [id=15776469]
module.bucket_lambda_pdf.aws_s3_bucket_public_access_block.this: Refreshing state... [id=lambda-pdf-doctrine-production]
module.bucket_lambda_pdf.aws_s3_bucket_policy.this[0]: Refreshing state... [id=lambda-pdf-doctrine-production]
module.lambda-pdf-generator-url.aws_lambda_function.this: Refreshing state... [id=pdf-generator-production-url]
module.lambda-pdf-generator-html.aws_lambda_function.this: Refreshing state... [id=pdf-generator-production-html]

Planning failed. Terraform encountered an error while generating this plan.

╷
│ Error: 1 error occurred:
│       * error validating monitor from /api/v1/monitor/15776469/validate: 400 Bad Request: {"errors":["Missing required tag key(s): repository, service, env, aws_account"]}
│ 
│ 
│ 
│   with module.datadog_monitor.datadog_monitor.error_rate_too_high,
│   on ../_modules/datadog/monitor.tf line 1, in resource "datadog_monitor" "error_rate_too_high":
│    1: resource "datadog_monitor" "error_rate_too_high" {

Expected Behavior

The default tags should be taken into account during the plan, even if the current state does not comply with the policy

Actual Behavior

The datadog_monitor_config_policy resources are already created.

Since I have specified the default tags in the Datadog provider configuration, I expect the Terraform plan to reflect the changes that align with the monitor policy. However, the provider does not account for the default tags

Steps to Reproduce

Create a Terraform stack that includes a datadog_monitor
establish the Datadog monitor tag policy with the new required tags.
Use the default_tags field in the Datadog provider configuration to fill the datadog_monitor with these new required tags.

Important Factoids

No response

References

https://github.com/DataDog/terraform-provider-datadog/issues/1478

DataDog / terraform-provider-datadog

Error Applying Default Tags in Datadog Monitor with Established Monitor Tag Policy #2668