Environment Variable Values...

[Bryan-Meier]

I am wondering if it would be possible to implement the ability to set environment variable values? Since we use a lot of environment variables, it would be good to set the values on some or most of environment variables because we know what they are going to be ahead of time. Is this something that is possible?

[GtheSheep]

Hey, would you be able to give these a shot for me? https://registry.terraform.io/providers/GtheSheep/dbt-cloud/latest/docs/resources/dbt_cloud_environment_variable

Admittedly I haven't used them myself, and currently don't have access to dbt cloud to do so but did an implementation of them a while ago, the API may have changed since though as it was a little odd back then

[GtheSheep]

@Bryan-Meier - could you lmk if this suits your needs? Thanks 😺

[Bryan-Meier]

Hi @GtheSheep, as luck would have it the project my team is on just heated up a couple weeks ago and I have not been able to put any time towards this quite yet.

[GtheSheep]

@Bryan-Meier - no sweat, always the way! Shout if I any issues when you get a chance 👌

[edosheen]

Hi, I'm trying out the resource now as below (with Terragrunt):

resource "dbt_cloud_environment_variable" "env-vars" {
  provider = dbt-cloud
  for_each = var.env_var_map

  project_id = var.project_id

  environment_values = var.env_var_map
  name = var.name
}

My inputs:

inputs = {
  env_var_map = {
    "production" = "testvalueprod"
    "development" = "testvaluedev"
  }

  name = "DBT_TERRAFORM_TEST"

  project_id = "<redacted>"
}

Planning this goes as expected:

Terraform will perform the following actions:

  # dbt_cloud_environment_variable.env-vars["development"] will be created
  + resource "dbt_cloud_environment_variable" "env-vars" {
      + environment_values = {
          + "development" = "testvaluedev"
          + "production"  = "testvalueprod"
        }
      + id                 = (known after apply)
      + name               = "DBT_TERRAFORM_TEST"
      + project_id         = <redacted>
    }

  # dbt_cloud_environment_variable.env-vars["production"] will be created
  + resource "dbt_cloud_environment_variable" "env-vars" {
      + environment_values = {
          + "development" = "testvaluedev"
          + "production"  = "testvalueprod"
        }
      + id                 = (known after apply)
      + name               = "DBT_TERRAFORM_TEST"
      + project_id         = <redacted>
    }

Plan: 2 to add, 0 to change, 0 to destroy.

However, when it comes to applying it, it errors out with a 500:

Error: POST url: https://cloud.getdbt.com/api/v3/accounts/${account_id}/projects/${project_id}/environment-variables/bulk/, status: 500, body: {"status":{"code":500,"is_success":false,"user_message":"There was a server error. Please contact support!","developer_message":""},"data":null}
│ 
│   with dbt_cloud_environment_variable.env-vars["production"],
│   on env-var.tf line 1, in resource "dbt_cloud_environment_variable" "env-vars":
│    1: resource "dbt_cloud_environment_variable" "env-vars" {

I dug up the API reference for V3 and it appears that the URL isn't quite correct:

https://documenter.getpostman.com/view/14183654/UyxohieD#52c09942-4e7d-469d-bd28-455a353ecb37

It appears to be the only one under Update URIs that does not have a trailing slash, although I'm not certain if this is the root cause.

[edosheen]

Reached out to DBT support and got connected with the guy that implemented the environment variables call.

Turns out creating an environment variable requires a different body than updating one. Here are successful cURL calls to create then update an environment variable (in that order):

Create:

curl --location --request POST 'https://cloud.getdbt.com//api/v3/accounts/<account-id>/projects/<project-id>/environment-variables/bulk' \
--header 'Content-Type: application/json' \
--header 'Authorization: Token <service-token>' \
--data '{
    "env_var": {
        "name": "DBT_TEST_ENV_VAR",
        "ids": [],
        "new_name": "DBT_TEST_ENV_VAR",
        "project": "PROJECT_DEFAULT_VALUE",
        "Development": "dev_value",
        "Production": "deploy_value"
    }
}'

Update:

curl --location --request PUT 'https://cloud.getdbt.com//api/v3/accounts/<account-id>/projects/<project-id>/environment-variables/bulk' \
--header 'Content-Type: application/json' \
--header 'Authorization: Token <service-token>' \
--data '{
    "env_vars": {
        "<var-id>": "TEST_UPDATE_VALUE",
        "name": "DBT_TEST_ENV_VAR",
        "ids": [
        ]
    }
}'

The in the update call can be found using another cURL, as they are not shown in the web interface:

https://cloud.getdbt.com/api/v3/accounts/<account-id>/projects/<project-id>/environment-variables/environment/

[GtheSheep]

Hey, yeah the provider doesn't send the same body for each, depending on which values get set in the request. Need to test it with the exact setup above using the relevant project names etc to ensure it's nothing to do with casing, etc, as we do unit test this. I'm wondering if it doesn't like "production" vs "Production"? Or the trailing slash thing, as that has caught me out previously in this provider on a seemingly inconsistent way, though I'd expect unit tests to catch that here. Should get to test this out sometime this week, but thanks for helping to dig into this! 🙂

[GtheSheep]

Interesting that ids is in both!!

[GtheSheep]

@edosheen - Tried the above without Terragrunt and seems ok? Could you please try setting a project key in your env_var_map to define the project default for the env var? It's mentioned in the docs and maps to that project key in your curl example

[edosheen]

Just tried, Terragrunt returns the same 500 error.

I've reached out to DBT support to ask about the ids fields

Edit:

Scratch that. I had ran the code without case-matching my environment names (needed to capitalize them).

Terragrunt successfully created the environment variable:

[GtheSheep]

oh awesome! glad to hear it 😸 I might add an example to use dbt_project.name or something in the env var map as it needs to be pretty specific. I'll see what the API responds with in case I can catch something like "project not found" to make this clearer than the 500 👌

[edosheen]

Looks like the ids fields aren't necessary in Create call.

It can be used in the Update call to update multiple environment variables at once:

PUT https://cloud.getdbt.com/api/v3/accounts/12345/projects/123456/environment-variables/bulk/
{
  "env_vars": {
    "92981": "alpha",
    "92982": "bravo",
    "name": "DBT_ENV_FOO",
    "ids": [
      92981,
      92982
    ]
  }
}

[github-actions[bot]]

This issue has been marked as Stale because it has been open for 90 days with no activity. If you would like the issue to remain open, please comment on the issue or else it will be closed in 7 days.