Can not set "threshold = 0" as alert condition

[sandnabba]

I would like to setup an alert that notifies if the sum of a metric goes above 0. However, Terraform fails when applying this change:

Error: Error applying plan:

1 error(s) occurred:

* module.staging-goalert-processingEvent.appoptics_alert.sqs_incorrect_format: 1 error(s) occurred:

* appoptics_alert.sqs_incorrect_format: Error creating AppOptics alert staging-goalert.processingEvent.incorrect_format: 

Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.

TF code:

resource "appoptics_alert" "sqs_incorrect_format" {
  name        = "${var.environment_name}-goalert.processingEvent.incorrect_format"
  description = "Managed by Terraform"
  services    = ["${lookup(var.alert_services, var.environment_tag_name)}"]
  rearm_seconds = 10800

  condition {
    type        = "above"
    threshold   = 0
    metric_name = "goalert.processingEvent"
    duration    = 60
    summary_function = "sum"

    tag {
      name    = "environment"
      grouped = true
      values  = ["${var.environment_tag_name}"]
    }

    tag {
      name    = "event_status"
      grouped = true
      values  = ["sqs message has incorrect format"]
    }

  }
}

Plan:

  + module.staging-goalert-processingEvent.appoptics_alert.sqs_incorrect_format
      id:                                    <computed>
      active:                                "true"
      attributes.%:                          "1"
      condition.#:                           "1"
      condition.1983545495.detect_reset:     ""
      condition.1983545495.duration:         "60"
      condition.1983545495.metric_name:      "goalert.processingEvent"
      condition.1983545495.summary_function: "sum"
      condition.1983545495.tag.#:            "2"
      condition.1983545495.tag.0.grouped:    "true"
      condition.1983545495.tag.0.name:       "environment"
      condition.1983545495.tag.0.values.#:   "1"
      condition.1983545495.tag.0.values.0:   "staging"
      condition.1983545495.tag.1.grouped:    "true"
      condition.1983545495.tag.1.name:       "event_status"
      condition.1983545495.tag.1.values.#:   "1"
      condition.1983545495.tag.1.values.0:   "sqs message has incorrect format"
      condition.1983545495.threshold:        "0"
      condition.1983545495.type:             "above"
      description:                           "Managed by Terraform"
      name:                                  "staging-goalert.processingEvent.incorrect_format"
      rearm_seconds:                         "10800"
      services.#:                            "1"
      services.1790749379:                   "160641"

Plan: 1 to add, 0 to change, 0 to destroy.

Just changing the threshold to 1 in the condition block will make this code work. Setting up this condition (threshold = 0) in the web GUI works fine.

I'm not sure if this is an issue in the AO API or in the TF provider, since I'm not getting the error message from the API.

[solarchad]

Thanks for the report @sandnabba. Was able to reproduce this. We definitely need to get errors from the API back to Terraform, currently this isn't happening. I'm not sure if the SDK/client is always delivering the errors in a way that we'd expect.

[solarchad]

I think issue might be with the Go SDK/client. I created an issue after additional testing here: https://github.com/appoptics/appoptics-api-go/issues/69. I'll close this for now but feel free to re-open if the issue with the SDK points back to something we can do to fix it here.

[sandnabba]

I think I would like to reopen this issue.

It seems like the root cause was fixed in the appoptics-api-go library in the following PR: https://github.com/appoptics/appoptics-api-go/pull/70

This was released in version 0.5.1 of the go-client, which is included in this provider now.

But it does still not work.

Plan

Terraform will perform the following actions:

  ~ module.goalert.appoptics_alert.sqs_incorrect_format
      condition.1983545495.detect_reset:     "" => ""
      condition.1983545495.duration:         "" => "60"
      condition.1983545495.metric_name:      "" => "goalert.processingEvent"
      condition.1983545495.summary_function: "" => "sum"
      condition.1983545495.tag.#:            "0" => "2"
      condition.1983545495.tag.0.grouped:    "" => "true"
      condition.1983545495.tag.0.name:       "" => "environment"
      condition.1983545495.tag.0.values.#:   "0" => "1"
      condition.1983545495.tag.0.values.0:   "" => "staging"
      condition.1983545495.tag.1.grouped:    "" => "true"
      condition.1983545495.tag.1.name:       "" => "event_status"
      condition.1983545495.tag.1.values.#:   "0" => "1"
      condition.1983545495.tag.1.values.0:   "" => "sqs message has incorrect format"
      condition.1983545495.threshold:        "" => "0"
      condition.1983545495.type:             "" => "above"
      condition.2880198930.detect_reset:     "false" => "false"
      condition.2880198930.duration:         "60" => "0"
      condition.2880198930.metric_name:      "goalert.processingEvent" => ""
      condition.2880198930.summary_function: "sum" => ""
      condition.2880198930.tag.#:            "2" => "0"
      condition.2880198930.tag.0.grouped:    "true" => "false"
      condition.2880198930.tag.0.name:       "environment" => ""
      condition.2880198930.tag.0.values.#:   "1" => "0"
      condition.2880198930.tag.0.values.0:   "staging" => ""
      condition.2880198930.tag.1.grouped:    "true" => "false"
      condition.2880198930.tag.1.name:       "event_status" => ""
      condition.2880198930.tag.1.values.#:   "1" => "0"
      condition.2880198930.tag.1.values.0:   "sqs message has incorrect format" => ""
      condition.2880198930.threshold:        "1" => "0"
      condition.2880198930.type:             "above" => ""

Plan: 0 to add, 1 to change, 0 to destroy.

Apply

module.goalert.appoptics_alert.sqs_incorrect_format: Modifying... (ID: 61792281)
  condition.1983545495.detect_reset:     "" => ""
  condition.1983545495.duration:         "" => "60"
  condition.1983545495.metric_name:      "" => "goalert.processingEvent"
  condition.1983545495.summary_function: "" => "sum"
  condition.1983545495.tag.#:            "0" => "2"
  condition.1983545495.tag.0.grouped:    "" => "true"
  condition.1983545495.tag.0.name:       "" => "environment"
  condition.1983545495.tag.0.values.#:   "0" => "1"
  condition.1983545495.tag.0.values.0:   "" => "staging"
  condition.1983545495.tag.1.grouped:    "" => "true"
  condition.1983545495.tag.1.name:       "" => "event_status"
  condition.1983545495.tag.1.values.#:   "0" => "1"
  condition.1983545495.tag.1.values.0:   "" => "sqs message has incorrect format"
  condition.1983545495.threshold:        "" => "0"
  condition.1983545495.type:             "" => "above"
  condition.2880198930.detect_reset:     "false" => "false"
  condition.2880198930.duration:         "60" => "0"
  condition.2880198930.metric_name:      "goalert.processingEvent" => ""
  condition.2880198930.summary_function: "sum" => ""
  condition.2880198930.tag.#:            "2" => "0"
  condition.2880198930.tag.0.grouped:    "true" => "false"
  condition.2880198930.tag.0.name:       "environment" => ""
  condition.2880198930.tag.0.values.#:   "1" => "0"
  condition.2880198930.tag.0.values.0:   "staging" => ""
  condition.2880198930.tag.1.grouped:    "true" => "false"
  condition.2880198930.tag.1.name:       "event_status" => ""
  condition.2880198930.tag.1.values.#:   "1" => "0"
  condition.2880198930.tag.1.values.0:   "sqs message has incorrect format" => ""
  condition.2880198930.threshold:        "1" => "0"
  condition.2880198930.type:             "above" => ""

Error: Error applying plan:

1 error(s) occurred:

* module.goalert.appoptics_alert.sqs_incorrect_format: 1 error(s) occurred:

* appoptics_alert.sqs_incorrect_format: Error updating AppOptics alert: 

[solarchad]

@trevrosen any idea why we're not getting the string from the error? Is the SDK now returning a string for all errors?

[trevrosen]

@sandnabba @solarchad hey guys sorry for late response -- taking a look at this stuff today and tomorrow. Will hopefully have something for you here soon.

[trevrosen]

@sandnabba hey the error reporting is now fixed on the master branch. Later today I'm going to have releasing in place so you'll no longer need to build the plugin from scratch each time you want to use it. If you want to try this out when you're back at the office, I'd appreciate any update you can add here. I'm working to knock out as much of the backlog as I can today (Tuesday) and tomorrow.

[sandnabba]

Hi @trevrosen Thanks a lot!

I just tried this and I can confirm that we are now getting both the error message from the API as well as it works to set a threshold value of 0. :+1: