Error on Terraform provider while creating Knowledge objects

[Manickam-Annamalai]

I try to create KOs (Dashboard/Saved search) on Splunk cloud and it successfully created the object on the splunk stack, but when it try to confirm, it gets error outs, as we have 1000s of KOs, I suspect its getting the time out after 1m when queries the endpoint. Note: I do set timeout as 600 on the provider connection..

Code:

terraform {
  required_providers {
    splunk = {
      source  = "splunk/splunk"
    }
  }
}

provider "splunk" {
  url           = "xyz.splunkcloud.com:8089"
  auth_token    = "<jwt_token>"
  insecure_skip_verify = true
  timeout = 600
}

resource "splunk_data_ui_views" "terraform_operations" {
  name     = "Terraform"
  eai_data = "<dashboard><label>Terraform</label><description>Terraform operations</description><row><panel><chart><search><query>index=_internal sourcetype=splunkd_access useragent=\"splunk-simple-go-client\" | timechart fixedrange=f values(status) by uri_path</query><earliest>-24h@h</earliest><latest>now</latest><sampleRatio>1</sampleRatio></search><option name=\"charting.axisLabelsX.majorLabelStyle.overflowMode\">ellipsisNone</option><option name=\"charting.axisLabelsX.majorLabelStyle.rotation\">0</option><option name=\"charting.axisTitleX.visibility\">collapsed</option><option name=\"charting.axisTitleY.text\">HTTP status codes</option><option name=\"charting.axisTitleY.visibility\">visible</option><option name=\"charting.axisTitleY2.visibility\">visible</option><option name=\"charting.axisX.abbreviation\">none</option><option name=\"charting.axisX.scale\">linear</option><option name=\"charting.axisY.abbreviation\">none</option><option name=\"charting.axisY.scale\">linear</option><option name=\"charting.axisY2.abbreviation\">none</option><option name=\"charting.axisY2.enabled\">0</option><option name=\"charting.axisY2.scale\">inherit</option><option name=\"charting.chart\">column</option><option name=\"charting.chart.bubbleMaximumSize\">50</option><option name=\"charting.chart.bubbleMinimumSize\">10</option><option name=\"charting.chart.bubbleSizeBy\">area</option><option name=\"charting.chart.nullValueMode\">connect</option><option name=\"charting.chart.showDataLabels\">none</option><option name=\"charting.chart.sliceCollapsingThreshold\">0.01</option><option name=\"charting.chart.stackMode\">default</option><option name=\"charting.chart.style\">shiny</option><option name=\"charting.drilldown\">none</option><option name=\"charting.layout.splitSeries\">0</option><option name=\"charting.layout.splitSeries.allowIndependentYRanges\">0</option><option name=\"charting.legend.labelStyle.overflowMode\">ellipsisMiddle</option><option name=\"charting.legend.mode\">standard</option><option name=\"charting.legend.placement\">right</option><option name=\"charting.lineWidth\">2</option><option name=\"trellis.enabled\">0</option><option name=\"trellis.scales.shared\">1</option><option name=\"trellis.size\">small</option><option name=\"trellis.splitBy\">_aggregation</option></chart></panel></row></dashboard>"
  acl {
    app     = "ops"
    owner   = "abc"
    sharing = "global"
  }
}

resource "splunk_saved_searches" "terraform-search-01" {
  actions                   = "email"
  action_email_format       = "table"
  action_email_max_time     = "5m"
  action_email_send_results = true
  action_email_subject      = "Splunk Alert: $name$"
  action_email_to           = "abc@xyz.com"
  action_email_track_alert  = true
  alert_comparator          = "greater than"
  alert_digest_mode         = true
  alert_expires             = "30d"
  alert_threshold           = "0"
  alert_type                = "number of events"
  description               = "Terraform activities alerts"
  dispatch_earliest_time    = "-15m@m"
  dispatch_latest_time      = "-1m@m"
  cron_schedule             = "*/15 * * * *"
  name                      = "terraform-search-01"
  schedule_priority         = "default"
  search                    = "index=_internal sourcetype=splunkd_access useragent=splunk-simple-go-client"

  acl {
    app     = "ops"
    owner   = "abc"
    sharing = "app"
  }
}

To perform exactly these actions, run the following command to apply: terraform apply "splunk1.tf.plan"

sh-3.2$ terraform apply "splunk1.tf.plan"
splunk_data_ui_views.terraform_operations: Creating...
splunk_saved_searches.terraform-search-01: Creating...
splunk_saved_searches.terraform-search-01: Still creating... [10s elapsed]
splunk_data_ui_views.terraform_operations: Still creating... [10s elapsed]
splunk_saved_searches.terraform-search-01: Still creating... [20s elapsed]
splunk_data_ui_views.terraform_operations: Still creating... [20s elapsed]
splunk_data_ui_views.terraform_operations: Still creating... [30s elapsed]
splunk_saved_searches.terraform-search-01: Still creating... [30s elapsed]
splunk_saved_searches.terraform-search-01: Still creating... [40s elapsed]
splunk_data_ui_views.terraform_operations: Still creating... [40s elapsed]
splunk_saved_searches.terraform-search-01: Still creating... [50s elapsed]
splunk_data_ui_views.terraform_operations: Still creating... [50s elapsed]
splunk_saved_searches.terraform-search-01: Still creating... [1m0s elapsed]
splunk_data_ui_views.terraform_operations: Still creating... [1m0s elapsed]
splunk_saved_searches.terraform-search-01: Still creating... [1m10s elapsed]

Error: Get "https://xyz.splunkcloud.com:8089/servicesNS/-/-/data/ui/views?count=-1&output_mode=json": EOF
│ 
│   with splunk_data_ui_views.terraform_operations,
│   on main.tf line 25, in resource "splunk_data_ui_views" "terraform_operations":
│   25: resource "splunk_data_ui_views" "terraform_operations" {
│ 
╵
╷
│ Error: Get "https://xyz.splunkcloud.com:8089/servicesNS/-/-/saved/searches?count=-1&output_mode=json": EOF
│ 
│   with splunk_saved_searches.terraform-search-01,
│   on main.tf line 35, in resource "splunk_saved_searches" "terraform-search-01":
│   35: resource "splunk_saved_searches" "terraform-search-01" {
│ 

[micahkemp-splunk]

The issue at hand is the provider was reading all views (servicesNS/-/-/saved/searches) and looking for the view with the name of the created resource:

    resp, err := (*provider.Client).ReadAllDashboardObject()
    if err != nil {
        return err
    }
    defer resp.Body.Close()

    entry, err := getDashboardByName(name, resp)
    if err != nil {
        return err
    }

    if entry == nil {
        return fmt.Errorf("unable to find resource: %v", name)
    }

Within getDashboardByName, the logic to find the dashboard was also flawed, in that it returned the first dashboard with a matching name, and multiple dashboards in different namespaces can have the same name:

        for _, entry := range response.Entry {
            if name == re.FindStringSubmatch(entry.Name)[1] {
                return &entry, nil
            }
        }

[micahkemp-splunk]

Re-opened because I missed that search resources were also failing.