dynatrace_autotag_v2 always detects changes

[ajoga]

Describe the bug I've set my dynatrace_autotag_v2 resource as I want and applied it successfully.

Subsequent runs of terraform (tofu in my env) will detect changes in the plan for all rules in the rules block and re-apply them, every time.

To Reproduce Steps to reproduce the behavior:

1. Create this resource:

resource "dynatrace_autotag_v2" "mrc" {
  name = "mrc"
  rules {
    rule {
      type                = "ME"
      enabled             = true
      value_normalization = "Leavetextas_is"
      attribute_rule {
        entity_type               = "PROCESS_GROUP"
        conditions {
          condition {
            key        = "PROCESS_GROUP_TECHNOLOGY"
            operator   = "EQUALS"
            enum_value = "ELASTIC_SEARCH"
            tag        = "Application:Elasticsearch"
          }
        }
      }
    }
  }
}

2. Run tofu apply -target dynatrace_autotag_v2.mrc, accept the change
3. Run tofu apply -target dynatrace_autotag_v2.mrc again, observe there is again changes to apply for some reason:

dynatrace_autotag_v2.mrc: Refreshing state... [id=REDACTED]

OpenTofu used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  ~ update in-place

OpenTofu will perform the following actions:

  # dynatrace_autotag_v2.mrc will be updated in-place
  ~ resource "dynatrace_autotag_v2" "mrc" {
        id                          = "REDACTED"
        name                        = "mrc"
        # (1 unchanged attribute hidden)

      ~ rules {
          - rule {
              - enabled             = true -> null
              - type                = "ME" -> null
              - value_normalization = "Leave text as-is" -> null

              - attribute_rule {
                  - azure_to_pgpropagation       = false -> null
                  - azure_to_service_propagation = false -> null
                  - entity_type                  = "PROCESS_GROUP" -> null
                  - host_to_pgpropagation        = false -> null
                  - pg_to_host_propagation       = false -> null
                  - pg_to_service_propagation    = false -> null
                  - service_to_host_propagation  = false -> null
                  - service_to_pgpropagation     = false -> null

                  - conditions {
                      - condition {
                          - case_sensitive = false -> null
                          - enum_value     = "ELASTIC_SEARCH" -> null
                          - integer_value  = 0 -> null
                          - key            = "PROCESS_GROUP_TECHNOLOGY" -> null
                          - operator       = "EQUALS" -> null
                        }
                    }
                }
            }
          + rule {
              + enabled             = true
              + type                = "ME"
              + value_normalization = "Leavetextas_is"

              + attribute_rule {
                  + entity_type = "PROCESS_GROUP"

                  + conditions {
                      + condition {
                          + enum_value = "ELASTIC_SEARCH"
                          + key        = "PROCESS_GROUP_TECHNOLOGY"
                          + operator   = "EQUALS"
                          + tag        = "Application:Elasticsearch"
                        }
                    }
                }
            }
          + rule {
            }
        }
    }

Plan: 0 to add, 1 to change, 0 to destroy.

Expected behavior No change in plan for the second run.

Notice that in the plan there is a desire to add an empty rule, which seems suspicious:

          + rule {
            }

Additional context

Dynatrace managed 1.290.53.20240501-192107

$ tofu version
OpenTofu v1.6.2
on linux_amd64
+ provider registry.opentofu.org/dynatrace-oss/dynatrace v1.61.0

NB: also tried with OpenTofu v1.8.0 with the same effects

[Dynatrace-Reinhard-Pilz]

Hello @ajoga,

There are a couple of things playing together in your case that lead to an empty plan.

First, it looks like the Schema for Auto Tags has changed a bit since we've implemented that resource. The possible values for the attribute value_normalization accepted by the Dynatrace REST API are no longer Leavetextas_is, Tolowercase and Touppercase but now Leave text as-is, To lower case and To upper case. I've done a bit of research in our project history. Apparently we've picked up on that change automatically. What is very often not completely automated are the descriptions of the attributes (essentially because relying too much on automating things leads to the confusion you've stumbled across in #512). The next release will contain the correct values mentioned in the description for that attribute.

In other words, you'll have to specify

value_normalization = "Leave text as-is"

instead of

value_normalization = "Leavetextas_is"

But there exists another issue in your HCL code. Your condition

condition {
  key        = "PROCESS_GROUP_TECHNOLOGY"
  operator   = "EQUALS"
  enum_value = "ELASTIC_SEARCH"
  tag        = "Application:Elasticsearch"
}

specifies the attribute tag - which has no effect in combination with the key PROCESS_GROUP_TECHNOLOGY. The Dynatrace REST API simply ignores it in that case. Once Terraform asks for the latest state of the auto tag, the REST API also doesn't provide a value here. Hence, a non-empty plan - Terraform still sees the attribute in your config specified.

An example for when the tag attribute DOES make sense is something like this.

condition {
  key      = "PROCESS_GROUP_TAGS"
  operator = "EQUALS"
  tag      = "Application:Elasticsearch"
}

After correcting the value for attribute value_normalization and omitting the attribute tag your plan should stay empty.

Finally, about the issue with the empty blocks within the terraform plans like this:

          + rule {
            }

These artifacts originate from a well known bug within the Terraform Plugin SDK. Because HashiCorp has already moved on and released the Terraform Plugin Framework, it's unlikely that this bug ever will get fixed. The Terraform Provider unfortunately cannot prevent these empty blocks from being generated (Terraform produces plans without invoking providers specifically for that functionality). They tend to emerge for resources that are allowed to contain an unordered list of sublocks of the same type. And they only come up when Terraform detects a change. But the Provider is able to detect them and filters them out. In other words, you may notice these empty blocks every now and then, when you're making changes to your configuration. But the Terraform Provider will nevertheless produce correct payload when addressing the REST API.

Having explained all that, my final recommendation for producing valid content for the resource dynatrace_autotag_v2 is to configure them via WebUI and then use the Export Functionality of the provider in order to get the matching HCL code automatically produced. We've implemented that functionality exactly because of Settings like Auto Tags, where it is everything else than apparent what combination of attributes allows for which values to get specified. The Schema for AutoTags offered by the REST API also doesn't go into detail here - and the resource documentations of the Terraform Provider originate from these Schemas.

[ajoga]

Hello @Dynatrace-Reinhard-Pilz,

Thank you very much for the fix in the doc and the very detailed explanation. You are correct, fixing value_normalization and the tag makes the unwanted behavior disappear.

I could've discovered the value_normalization discrepancy as it's shown in the plan on my ticket, but I didn't likely because of the plan showing so many lines when there's only one (key,value) that changed.

Is it bug-report worthy to report that changes to one (key,value) in a 5 levels deep block shouldn't lead to a complete remove/add of the 2nd level block ? (I can't remember if this is a common way of working of Terraform or not)

[Dynatrace-Reinhard-Pilz]

Is it bug-report worthy to report that changes to one (key,value) in a 5 levels deep block shouldn't lead to a complete remove/add of the 2nd level block ? (I can't remember if this is a common way of working of Terraform or not)

That behavior is to be expected. The schema our provider hands over to Terraform specifies, that a dynatrace_autotag_v2 contains an unordered list of rule blocks. The "unordered" nature tells Terraform, that it cannot expect to find the same rule block at the same position the next time it prepares a plan. The same rule may be at a different index. Therefore, because it cannot simply use the index for finding identical rules, it creates a hash code for every block (based on the complete contents) - and looks for the hash codes instead of array indices. ... which explains why Terraform assumes that a block has been removed and a completely different block is getting introduced if you change just a tiny bit in there (irrelevant whether that change is 5 levels deep or not). For Terraform the block with hash code A is gone and another block with hash code B has emerged.

But eventually it all just LOOKS more drastic within the plan Terraform produces. Effectively, when you're running terraform apply, the Dynatrace REST API receives a list of rules within the payload that contains one small difference.

Makes sense?

[ajoga]

Makes sense, I trust you for the workings of Terraform as a developper, thank you again for that answer @Dynatrace-Reinhard-Pilz