event alert' type is not supported, please use 'event-v2 alert' type

[prashanthbgoud]

Terraform Version

1.18

Datadog provider Version

3.11.0

Affected Resource(s)

Please list the resources as a list, for example:

• datadog_monitor

ERROR

 │ Error: error validating monitor from https://api.datadoghq.com/api/v1/monitor/11666103/validate: 400 Bad Request: {"errors": ["'event alert' type is not supported, please use 'event-v2 alert' type."]}
│ 
│   with datadog_monitor.nginx_watchdog,
│   on main.tf line 6, in resource "datadog_monitor" "nginx_watchdog":
│    6: resource "datadog_monitor" "nginx_watchdog" {

│ Error: error validating monitor from https://api.datadoghq.com/api/v1/monitor/validate: 400 Bad Request: {"errors": ["The value provided for parameter 'query' is invalid: invalid operator specified: "]}
│ 
│   with datadog_monitor.nginx_watchdog,
│   on main.tf line 6, in resource "datadog_monitor" "nginx_watchdog":
│    6: resource "datadog_monitor" "nginx_watchdog" {

Steps to Reproduce

tf code:

resource "datadog_monitor" "nginx_watchdog" {
  name               = "[Watchdog] nginx {{event.title}}"
  type               = "event alert"
  message            = "@slack-datadog-alerts"
  escalation_message = "Escalation message @pagerduty"

  query = "events('priority:all sources:watchdog tags:story_type:service,env:production,service:nginx).by('cluster,service,resource_name').rollup('count').last('30m') > 0"

  notify_no_data    = false
  renotify_interval = 0

  notify_audit = false
  timeout_h    = 0
  include_tags = true

  tags = ["service:nginx"]
}

1. terraform apply

[pguinard-public-com]

Try setting the type to event-v2 alert and fixing your query to have valid quoting.

[StefanSchmidtOz]

I believe this is a matter of migrating the monitors properly, this is not a bug. If you are stuck maybe reach out to DD support directly. Here are the migration instructions: https://docs.datadoghq.com/events/guides/migrating_to_new_events_features/

What is the migration timeline?

The following timeline applies to US-based customers:

March 5, 2022 - New Events Explorer and analytics are accessible. Datadog begins to migrate customer dashboards and monitors not managed using the API.

April 30, 2022 - The event stream is retired in favor of the Event Explorer.

May 15, 2022 - Starting on this date, while Datadog continues to evaluate event monitors that have not been migrated, they are no longer editable. New event monitors must use the new syntax.

June 30, 2022 - Datadog stops evaluating event monitors that have not been migrated. Legacy event monitors will stop working.

What action do I need to take?

If you do not manage your dashboard or monitors using external API-based tools (such as Terraform or scripts), then no action is required on your end. Datadog will migrate your dashboards and monitors before April 30, 2022. We will leave your old monitors in place but they will be muted and Datadog will stop evaluating them on June 30, 2022.

If you use Terraform or other API-based scripts to manage all or some of your dashboards, Datadog will migrate queries in your event widgets and overlays to the new syntax, but you will need to update your scripts to keep them in sync before June 30, 2022.

If you use Terraform or other API-based scripts to manage all or some of your monitors, you have until June 30, 2022, to update them. Please see our migration guide for more details. You may also reach out to the support team to get assistance. After this date, Datadog will create new versions of any non-migrated monitors and mute the existing monitors to ensure that you continue to have alerting.

Please see our migration guide for more details and steps for how to migrate your dashboards and monitors. Datadog can also assist you with migrating your monitors by suggesting updates or by applying the updates to your monitors.

[barywhyte]

I am encountering same issue described by @prashanthbgoud. For users that use terraform to manage datadog monitors, I can only say that this migration link is at best nothing as it does not describe how to actually carry out this migration. Changing event alert to event-v2 alert only produced a new error and so this doesn't help at all.

[jmctune]

Also ran into this issue when attempting to migrate datadog_monitor to event-v2 alert. Had to update my query to use double quotes and additionally escape them. The previous method of using single quotes didn't want to work anymore.

As an example, from this:

"events('priority:all oom status:error sources:docker tags:map_env:prod').rollup('count').last('5m') > 0"

to this:

"events(\"status:error map_env:prod source:docker oom\").rollup(\"count\").last(\"5m\") > 0"

Note that the query also looks slightly different. When only porting the double quote change, I found facets that were no longer valid that I had to remove to get this to plan. This may not be applicable to your environment, but it was for mine. Was running into this error:

Error: error validating monitor from https://api.datadoghq.com/api/v1/monitor/validate: 400 Bad Request: {"errors": ["The value provided for parameter 'query' is invalid: Invalid query: Check for invalid tags or facets in your query."]}

Removing priority:all from my query fixed this error for me.

I'm on 3.10.0 of the datadog provider.

[h4xnoodle]

I also ran into the above problems around quoting. I was also engaged with DD support around this, and it took a very long time to reach the point of "you must double quote everything". I found that a bit strange that the single quotes no longer worked at all.

[maziadi]

Same issue happened for me. Double quoted everything. One other thing : be careful of number of allowed facets with event-v2 type, it can not go beyond 4 with this version.

[DevBaca]

I recommend using the export monitor feature to grab the query in the correct syntax as well as reviewing when migrating monitors to terraform. It provides the properly quoted and escaped query for you