error creating synthetics API test from https://api.datadoghq.com/api/v1/synthetics/tests/api: 400 Bad Request: {"errors":["Global variable invalid names: GENERIC_OPS_USER_EMAIL, GENERIC_OPS_USER_PASSWORD, GENERIC_PASSWORD"]}, terraform apply is failing with this error all of a sudden with no changes in the code

[shwetaarshad]

Hi there,

Thank you for opening an issue. Please note that we try to keep the Terraform issue tracker reserved for bug reports and feature requests. For general usage questions, please see: https://www.terraform.io/community.html.

Terraform Version

⇒  terraform -v
Terraform v1.5.0
on darwin_amd64
+ provider registry.terraform.io/datadog/datadog v3.25.0

Affected Resource(s)

Datadog Synthetics tests

If this issue appears to affect multiple resources, it may be an issue with Terraform's core, so please mention this.

Terraform Configuration Files

terraform {

  required_providers {
    datadog = {
      source = "DataDog/datadog"
      version = "~> 3.25.0"
    }
  }

  backend "s3" {
    key = "<none set>"
    encrypt = true
  }

}

# Configure the Datadog provider
provider "datadog" {
  api_key = var.datadog_api_key
  app_key = var.datadog_app_key
  api_url = "https://api.datadoghq.com/"
}

The global variables which are failing to get created /updated

resource "datadog_synthetics_global_variable" "user_test_password" {
  name        = "USER_TEST_PASSWORD_${upper(replace(var.namespace, "-", "_"))}"
  description = "A common password for all the users"
  value       = var.generic_password
  secure      = true
}
=====================================*

config_variable {
    type = "global"
    name = "GENERIC_PASSWORD"
    id   = datadog_synthetics_global_variable.user_test_password.id
  }

resource "datadog_synthetics_global_variable" "generic_ops_user_email" { name = "OPS_USER_TESTEMAIL${upper(replace(var.namespace, "-", "_"))}" description = "An OPS User email" value = var.generic_ops_user_email secure = true }

resource "datadog_synthetics_global_variable" "generic_ops_user_password" { name = "OPS_USER_TESTPASSWORD${upper(replace(var.namespace, "-", "_"))}" description = "An OPS user password" value = var.generic_ops_user_password secure = true }

=============================* config_variable { type = "global" name = "GENERIC_OPS_USER_EMAIL" id = datadog_synthetics_global_variable.generic_ops_user_email.id }

config_variable { type = "global" name = "GENERIC_OPS_USER_PASSWORD" id = datadog_synthetics_global_variable.generic_ops_user_password.id }

### Debug Output

Error: error creating synthetics API test from https://api.datadoghq.com/api/v1/synthetics/tests/api: 400 Bad Request: {"errors":["Global variable invalid names: GENERIC_PASSWORD"]} │ │ with datadog_synthetics_test.admin_user_crud_environments, │ on AdminCRUDEnvironments.tf line 8, in resource "datadog_synthetics_test" "admin_user_crud_environments": │ 8: resource "datadog_synthetics_test" "admin_user_crud_environments" { │ ╵ ╷ │ Error: error creating synthetics API test from https://api.datadoghq.com/api/v1/synthetics/tests/api: 400 Bad Request: {"errors":["Global variable invalid names: GENERIC_PASSWORD"]} │ │ with datadog_synthetics_test.admin_user_can_view_audit_logs, │ on AdminCanVerifyAuditLogs.tf line 8, in resource "datadog_synthetics_test" "admin_user_can_view_audit_logs": │ 8: resource "datadog_synthetics_test" "admin_user_can_view_audit_logs" { │ ╵ ╷ │ Error: error creating synthetics API test from https://api.datadoghq.com/api/v1/synthetics/tests/api: 400 Bad Request: {"errors":["Global variable invalid names: GENERIC_PASSWORD"]} │ │ with datadog_synthetics_test.Multi_org_user_login, │ on MultiOrgLogin.tf line 8, in resource "datadog_synthetics_test" "Multi_org_user_login": │ 8: resource "datadog_synthetics_test" "Multi_org_user_login" { │ ╵ ╷ │ Error: error creating synthetics API test from https://api.datadoghq.com/api/v1/synthetics/tests/api: 400 Bad Request: {"errors":["Global variable invalid names: GENERIC_PASSWORD"]} │ │ with datadog_synthetics_test.admin_user_invite_other_user, │ on inviteUserToTheOrg.tf line 8, in resource "datadog_synthetics_test" "admin_user_invite_other_user": │ 8: resource "datadog_synthetics_test" "admin_user_invite_other_user" { │ ╵ ╷ │ Error: error creating synthetics API test from https://api.datadoghq.com/api/v1/synthetics/tests/api: 400 Bad Request: {"errors":["Global variable invalid names: GENERIC_OPS_USER_PASSWORD, GENERIC_OPS_USER_EMAIL"]} │ │ with datadog_synthetics_test.ops_admin_set_limits, │ on opsUserLoginAndSetLimits.tf line 22, in resource "datadog_synthetics_test" "ops_admin_set_limits": │ 22: resource "datadog_synthetics_test" "ops_admin_set_limits" { │ ╵ ╷ │ Error: error creating synthetics API test from https://api.datadoghq.com/api/v1/synthetics/tests/api: 400 Bad Request: {"errors":["Global variable invalid names: GENERIC_OPS_USER_EMAIL, GENERIC_OPS_USER_PASSWORD"]} │ │ with datadog_synthetics_test.ops_admin_login_retry, │ on opsUserLoginRetriesWithWrongCreds.tf line 8, in resource "datadog_synthetics_test" "ops_admin_login_retry": │ 8: resource "datadog_synthetics_test" "ops_admin_login_retry" { │ ╵ ╷ │ Error: error creating synthetics API test from https://api.datadoghq.com/api/v1/synthetics/tests/api: 400 Bad Request: {"errors":["Global variable invalid names: GENERIC_PASSWORD"]} │ │ with datadog_synthetics_test.user_creates_api_token, │ on userCreatesAPIToken.tf line 8, in resource "datadog_synthetics_test" "user_creates_api_token": │ 8: resource "datadog_synthetics_test" "user_creates_api_token" { │ ╵ ╷ │ Error: error creating synthetics API test from https://api.datadoghq.com/api/v1/synthetics/tests/api: 400 Bad Request: {"errors":["Global variable invalid names: GENERIC_PASSWORD"]} │ │ with datadog_synthetics_test.Admin_user_login, │ on userRoles.tf line 15, in resource "datadog_synthetics_test" "Admin_user_login": │ 15: resource "datadog_synthetics_test" "Admin_user_login" { │ ╵ ╷ │ Error: error creating synthetics API test from https://api.datadoghq.com/api/v1/synthetics/tests/api: 400 Bad Request: {"errors":["Global variable invalid names: GENERIC_OPS_USER_EMAIL, GENERIC_OPS_USER_PASSWORD, GENERIC_PASSWORD"]} │ │ with datadog_synthetics_test.user_signup, │ on userSignupWorkflow.tf line 7, in resource "datadog_synthetics_test" "user_signup": │ 7: resource "datadog_synthetics_test" "user_signup" { │ ╵ ╷ │ Error: error creating synthetics API test from https://api.datadoghq.com/api/v1/synthetics/tests/api: 400 Bad Request: {"errors":["Global variable invalid names: GENERIC_PASSWORD"]} │ │ with datadog_synthetics_test.user_is_able_to_update_their_notifications, │ on userUpdateNotificationSettingsTest.tf line 7, in resource "datadog_synthetics_test" "user_is_able_to_update_their_notifications": │ 7: resource "datadog_synthetics_test" "user_is_able_to_update_their_notifications" { │ ╵ make: *** [apply] Error 1

### Panic Output

NA

### Expected Behavior

terraform apply should work and variables should get created.

### Actual Behavior

The Apply is failing with Global variable invalid names error

### Steps to Reproduce

Please list the steps required to reproduce the issue, for example:
1. `terraform init`
2. `terraform plan`
1. `terraform apply`

NOTE: This error started popping up recently, I have upgraded Terraform to latest and also Datadog to 3.25, not sure what is causing this as I have cleaned up everything (existing global variables and including terraform state file) and tried to deploy it from scratch but its still failing with this error.

### Important Factoids

Are there anything atypical about your accounts that we should know? For example: Running in EC2 Classic? Custom version of OpenStack? Tight ACLs?

### References

Are there any other GitHub issues (open or closed) or Pull Requests that should be linked here? For example:

-   GH-1234

[therve]

Can you try 3.26? Thanks.

[shwetaarshad]

Can you try 3.26? Thanks.

Thanks for reverting, thats failing with plugin crash error, I just raised a bug around that and I was told its a duplicate:

terraform the plugin.(*grpcprovider).upgraderesourcestate request was cancelled

[therve]

OK, we'll need a new release then. Thanks for your patience.

[shwetaarshad]

OK, we'll need a new release then. Thanks for your patience.

Thanks for reverting, When can we expect the next release? We are kind of stuck here :(

[therve]

I don't think we'll revert, we have a fix in place.

[shwetaarshad]

I don't think we'll revert, we have a fix in place.

Thanks, I was able to use 3.26 as well now and didn't see the plugin error anymore (looks like trying multiple times in the environment resolves the issue, which doesn't seem rational but true), but the terraform apply is still failing. Interestingly, I have noticed a pattern, the same variables are not failing when used as browser_variable but only failing for the tests where it is defined as config_variable (that is API tests) not sure if that makes sense but this is an interesting pattern.

[jsteinberg-rbi]

hitting this too

[therve]

hitting this too

Are you able to try 3.26, or is it not working for you either?

[jsteinberg-rbi]

already on 3.26

[therve]

If you have a reproducer that'd be great, thanks.

[therve]

Ah I see where the issue might come from. There is some additional validation, now the name inside the the test must match the name of the global variable. You need to replace GENERIC_PASSWORD by USER_TEST_PASSWORD_XXX in the test configuration.

[shwetaarshad]

Ah I see where the issue might come from. There is some additional validation, now the name inside the the test must match the name of the global variable. You need to replace GENERIC_PASSWORD by USER_TEST_PASSWORD_XXX in the test configuration.

Thanks for your help, I will try this. So this limitation is only for the API tests but not for the Browser tests, cause I am using same style in Browser tests but those are not failing? Nonetheless, I will give it a try, if thats what it is, so be it :)

[shwetaarshad]

Ah I see where the issue might come from. There is some additional validation, now the name inside the the test must match the name of the global variable. You need to replace GENERIC_PASSWORD by USER_TEST_PASSWORD_XXX in the test configuration.

I tried this but it still throw the same error, I changed all occurrences of GENERIC_PASSWORD with USER_TEST_PASSWORD but its still complaining also changed resource name to USER_TEST_PASSWORD from USER_TEST_PASSWORD_XXX to maintain the similarity.

│ Error: error creating synthetics API test from https://api.datadoghq.com/api/v1/synthetics/tests/api: 400 Bad Request: {"errors":["Global variable invalid names: USER_TEST_PASSWORD"]}
│
│   with datadog_synthetics_test.admin_user_crud_environments,
│   on AdminCRUDEnvironments.tf line 8, in resource "datadog_synthetics_test" "admin_user_crud_environments":
│    8: resource "datadog_synthetics_test" "admin_user_crud_environments" {
│
╵
╷
│ Error: error creating synthetics API test from https://api.datadoghq.com/api/v1/synthetics/tests/api: 400 Bad Request: {"errors":["Global variable invalid names: USER_TEST_PASSWORD"]}
│
│   with datadog_synthetics_test.admin_user_can_view_audit_logs,
│   on AdminCanVerifyAuditLogs.tf line 8, in resource "datadog_synthetics_test" "admin_user_can_view_audit_logs":
│    8: resource "datadog_synthetics_test" "admin_user_can_view_audit_logs" {
│
╵
╷
│ Error: error creating synthetics API test from https://api.datadoghq.com/api/v1/synthetics/tests/api: 400 Bad Request: {"errors":["Global variable invalid names: USER_TEST_PASSWORD"]}
│
│   with datadog_synthetics_test.Multi_org_user_login,
│   on MultiOrgLogin.tf line 8, in resource "datadog_synthetics_test" "Multi_org_user_login":
│    8: resource "datadog_synthetics_test" "Multi_org_user_login" {

[therve]

In the original config you had a suffix with the names, are you passing it as well? They need to match exactly.

[shwetaarshad]

In the original config you had a suffix with the names, are you passing it as well? They need to match exactly.

I removed the suffix completely to match it as is.

#
#   The Terraform datadg synthetic tests controller for user roles update tests
#   Admin user logins to console and update an existing user's role
#
################################################################################

resource "datadog_synthetics_global_variable" "user_test_password" {
  name        = "USER_TEST_PASSWORD"
  description = "A common password for all the users"
  value       = var.user_test_password
  secure      = true
}

resource "datadog_synthetics_test" "Admin_user_login" {
  name = "[${var.namespace}] Admin user login and update roles for existing users"
  type = "api"
  message = "The admin user login test is failing @shweta.arshad@solace.com @slack-solacedotcom-saas-datadog-smoke-test-suite-${var.datadog_account}"
  subtype = "multi"
  status  = var.run_status
  locations = [var.region]
  tags = [var.domain, var.namespace, "maas-gateway", var.datadog_account]

  options_list {
    tick_every = local.run-priority["standard"]["tick_every"]
    monitor_priority = local.run-priority["standard"]["monitor_priority"]
    min_failure_duration = local.run-priority["standard"]["min_failure_duration"]
    min_location_failed = local.run-priority["standard"]["min_location_failed"]
    retry {
      count    = local.run-priority["standard"]["retry_count"]            
      interval = local.run-priority["standard"]["retry_interval"]
    }
    monitor_options {
      renotify_interval = local.run-priority["standard"]["renotify_interval"]
    }
  }

  api_step {
    name    = "A user login API test on console"
    subtype = "http"
    allow_failure = "true"
    is_critical = "true"

    assertion {
      type     = "statusCode"
      operator = "is"
      target   = "200"
    }

    request_definition {
      method = "POST"
      url    = "https://${var.api_url}/api/v1/iam/tokens"
      body = "{\"username\": \"${var.maas_admin_user_username}\", \"password\": \"{{USER_TEST_PASSWORD}}\"}"
    }

    request_headers = {
      Content-Type   = "application/json"
    }

    extracted_value {
      name = "Token"
      parser {
        type = "json_path"
        value = "$.token"
      }
      type = "http_body"

    }
    retry {
      count    = "1"
      interval = "500"
    }

  }

api_step {
  name    = "An admin user gets all the users in the organization"
  subtype = "http"
  allow_failure = "true"
  is_critical = "true"

  assertion {
    type     = "statusCode"
    operator = "is"
    target   = "200"
  }

  request_definition {
    method = "GET"
    url    = "https://${var.api_url}/api/v0/users"
  }

  request_headers = {
    Content-Type   = "application/json"
    Authorization =  "Bearer {{Token}}"
  }

  extracted_value {
    name = "UserId"
    parser {
      type = "json_path"
      value = "$.data[?(@.lastName=='viewer')].userId"
    }
    type = "http_body" 
  }
  retry {
      count    = "1"
      interval = "500"
  }
}

api_step {
  name    = "An admin user changes the role of the extracted user"
  subtype = "http"
  allow_failure = "true"
  is_critical = "true"

  assertion {
    type     = "statusCode"
    operator = "is"
    target   = "200"
  }

  request_definition {
    method = "PUT"
    url    = "https://${var.api_url}/api/v0/users/{{UserId}}/roles"
    body = "[{\"id\": \"messaging-service-editor\"}, {\"id\": \"event-portal-user\"}]"
  }

  request_headers = {
    Content-Type   = "application/json"
    Authorization =  "Bearer {{Token}}"
  }
  retry {
      count    = "1"
      interval = "500"
  }

}

api_step {
  name    = "An admin user reverts the change to role of the extracted user"
  subtype = "http"
  allow_failure = "true"
  is_critical = "true"

  assertion {
    type     = "statusCode"
    operator = "is"
    target   = "200"
  }

  request_definition {
    method = "PUT"
    url    = "https://${var.api_url}/api/v0/users/{{UserId}}/roles"
    body = "[{\"id\": \"messaging-service-viewer\"}]"
  }

  request_headers = {
    Content-Type   = "application/json"
    Authorization =  "Bearer {{Token}}"
  }

  retry {
      count    = "1"
      interval = "500"
  }

}

  api_step {
    name    = "Admin user logs out from the console"
    subtype = "http"
    allow_failure = "true"
    is_critical = "true"

    assertion {
      type     = "statusCode"
      operator = "is"
      target   = "200"
    }

    request_definition {
      method = "DELETE"
      url = "https://${var.api_url}/api/v0/iam/token"
    }

    request_headers = {
      Content-Type   = "application/json"
      Authorization =  "Bearer {{Token}}"
    }
    retry {
      count    = "1"
      interval = "500"
    }
  }

  config_variable {
    type = "global"
    name = "USER_TEST_PASSWORD"
    id   = datadog_synthetics_global_variable.user_test_password.id
  }

}

//tick_every: The running frequency of the test in seconds
//retry_count: Number of retries needed to consider a location as failed before sending a notification alert.
//retry_interval: Interval between a failed test and the next retry in milliseconds.
//renotify_interval: The renotification frequency in seconds.

locals {
  run-priority = {
    critical = { 
      tick_every=300, retry_count=2, retry_interval=500, renotify_interval=10, min_failure_duration=90, min_location_failed=1, monitor_priority=1
    }
    standard = {
      tick_every=900, retry_count=2, retry_interval=1000, renotify_interval=20, min_failure_duration=120, min_location_failed=1, monitor_priority=3
    } 
    low = {
      tick_every=3600, retry_count=2, retry_interval=1500, renotify_interval=30, min_failure_duration=150, min_location_failed=1, monitor_priority=4
    } 
  } 
}

[shwetaarshad]

In the original config you had a suffix with the names, are you passing it as well? They need to match exactly.

Actually I take the above back, I cleaned all the existing Global variables and state files and looks like the number of error reduced:

 Error: error creating synthetics API test from https://api.datadoghq.com/api/v1/synthetics/tests/api: 400 Bad Request: {"errors":["Global variable invalid names: OPS_USER_EMAIL, OPS_USER_PASSWORD"]}
│
│   with datadog_synthetics_test.ops_admin_set_limits,
│   on opsUserLoginAndSetLimits.tf line 22, in resource "datadog_synthetics_test" "ops_admin_set_limits":
│   22: resource "datadog_synthetics_test" "ops_admin_set_limits" {
│
╵
╷
│ Error: error creating synthetics API test from https://api.datadoghq.com/api/v1/synthetics/tests/api: 400 Bad Request: {"errors":["Global variable invalid names: OPS_USER_PASSWORD, OPS_USER_EMAIL"]}
│
│   with datadog_synthetics_test.ops_admin_login_retry,
│   on opsUserLoginRetriesWithWrongCreds.tf line 8, in resource "datadog_synthetics_test" "ops_admin_login_retry":
│    8: resource "datadog_synthetics_test" "ops_admin_login_retry" {
│
╵
╷
│ Error: error creating synthetics global variable from https://api.datadoghq.com/api/v1/synthetics/variables: 400 Bad Request: {"errors":["The value provided for parameter 'Synthetics variable with same name and type already exists.' is invalid"]}
│
│   with datadog_synthetics_global_variable.user_test_password,
│   on userRoles.tf line 8, in resource "datadog_synthetics_global_variable" "user_test_password":
│    8: resource "datadog_synthetics_global_variable" "user_test_password" {
│
╵
make: *** [apply] Error 1

Looks like keeping same Global variable names does change something actually. I will cleanup further to eliminate these error as well. I think we are good to close this one.

[shwetaarshad]

Its working now, Closing the issue.