Volatile Terraform state with Splunk provider and HEREDOC

[Koleon]

Hello everyone,

our team use the Terraform Splunk provider for managing saved searches. Usually terraform plan detects changes within HEREDOC section of all alerts and 112 unchanged attributes hidden after several days from previous successful terraform apply without any human intervention.

All alerts use Terraform module and theirs syntax is very similar, something like:

module "CloudTrail-AWSRootConsoleLogin" {
  source                     = "git::ssh://XYZ"
  alert_name                 = "CloudTrail-AWSRootConsoleLogin"
  search                     = <<-EOT
    ```!!! This alert is managed via Terraform, DON'T modify it directly in Splunk !!! https://XYZ```

    index=secops_cloudtrail eventName="ConsoleLogin" userIdentity.type="Root"
    | eval "Account ID" = aws_account_id
    | lookup XYZ
    | stats values(eventName) as "Event name" XYZ
    EOT
  action_email_to            = "fakeaddress@fake.opsgenie.net"
  action_email_message_alert = <<-EOT
    The alert condition for '$name$' was triggered.

    Please review results below and follow the playbook
    https://fakedomain

    Event name: $result.Event name$
    Action: $result.Action$
    Request group: $result.Request group$
    Source: $result.Source$
    Agent: $result.Agent$
    User: $result.User$
    Role: $result.Role$
    Environment: $result.Environment$
    Region: $result.Region$
    Account name: $result.Account name$
    Account ID: $result.Account ID$

    ----------------------
    $result._raw$
    EOT
  description                = "This alert is managed via Terraform"
}

Running $ terraform plan -target=module.CloudTrail-AWSRootConsoleLogin after couple days since successful apply, the output looks like:

Acquiring state lock. This may take a few moments...
module.CloudTrail-AWSRootConsoleLogin.splunk_saved_searches.saved_search: Refreshing state... [id=CloudTrail-AWSRootConsoleLogin]

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # module.CloudTrail-AWSRootConsoleLogin.splunk_saved_searches.saved_search will be updated in-place
  ~ resource "splunk_saved_searches" "saved_search" {
      ~ action_email_message_alert                      = <<-EOT
            The alert condition for '$name$' was triggered.

            Please review results below and follow the playbook
            https://fakedomain

            Event name: $result.Event name$
            Action: $result.Action$
            Request group: $result.Request group$
            Source: $result.Source$
            Agent: $result.Agent$
            User: $result.User$
            Role: $result.Role$
            Environment: $result.Environment$
            Region: $result.Region$
            Account name: $result.Account name$
            Account ID: $result.Account ID$

            ----------------------
            $result._raw$
        EOT
        id                                              = "CloudTrail-AWSRootConsoleLogin"
        name                                            = "CloudTrail-AWSRootConsoleLogin"
      ~ search                                          = <<-EOT
            ```!!! This alert is managed via Terraform, DON'T modify it directly in Splunk !!! https://XYZ```

            index=secops_cloudtrail eventName="ConsoleLogin" userIdentity.type="Root"
            | eval "Account ID" = aws_account_id
            | lookup XYZ
            | stats values(eventName) as "Event name" XYZ
        EOT
        # (112 unchanged attributes hidden)

        # (1 unchanged block hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.
╷
│ Warning: Resource targeting is in effect
│
│ You are creating a plan with the -target option, which means that the result of this plan may not represent all of the changes requested by the current configuration.
│
│ The -target option is not for routine use, and is provided only for exceptional situations such as recovering from errors or mistakes, or when Terraform specifically suggests to use it as part of an
│ error message.

I wonder is it a bug within Splunk provider or terraform itself? Does anyone have an idea how to debug it please? Or do you think the discrepancy is caused by some Splunk backend service?

Thank you for any help.

[Koleon]

Bump.

[cbaxtersf]

I'm facing a similar issue. My terraform is trying to update the following on an alert:

• alert_comparator : "" change to "greater than"
• alert_threshold : "" change to "0"
• alert_type : "always" change to "number of events"

The apply says it worked but it actually never does. I confirmed via the Splunk API that it does work based on what the provider is doing.