dbtcloud_environment_variable resource appears to have a caching issue, preventing re-creation of named environment variable.

[BrianMcGrathCox]

Describe the bug I am attempting to create, delete, then re-create a dbt environment variable. The create apply works successfully. The delete (destroy) apply works successfully. The subsequent create apply fails.

Error message The request was invalid. Please double check the provided data and try again. An environment variable by this name already exists in this project and account.

Resource configuration resource "dbtcloud_environment_variable" "deploy-environment-variable-dbt-test" { name = "DBT_TEST" project_id = data.aws_ssm_parameter.project-id.value environment_values = { "project" : "test_project_default", "test" : "test_value" }

Expected behavior After creating the dbt environment variable, then deleting the variable, I expect another creation apply to succeed.

Config (please complete the following information): dbtcloud = { source = "dbt-labs/dbtcloud" version = ">=0.2.1" }

Additional context Add any other context about the problem here.

[b-per]

Thanks for the report. I will try to reproduce it in the next days and release a fix.

[b-per]

Hi @BrianMcGrathCox , I just tried to replicate it locally and it didn't fail. I did the following:

1. add an env var from Terraform and apply: the env var is created
2. after it is created, comment the Terraform code for the env var and apply: the env var is deleted
3. Uncomment the code of the env var and apply: the env var is created

---

I also did the following and the resource got deleted and recreated

terraform apply -replace=dbtcloud_environment_variable.deploy-environment-variable-dbt-test

---

Could you check if the env var in question has really been deleted from dbt Cloud? And/or if someone has recreated it manually via the dbt Cloud UI?

If it has been created manually, it will need to be either deleted from the UI or imported into the Terraform state before it can be managed via Terraform.

The import instructions are in the docs.

[BrianMcGrathCox]

Benoit,

Thanks for testing this.

Yes, we did verify that the environment variable was deleted before creating/adding back the environment variable.

I removed the environment variable from dbt cloud project. I deployed (plan/apply) terragrunt module without the environment variable. This was successful. I verified the environment variable is not in dbt cloud project. I added the environment variable to the terragrunt module. I deployed (plan/apply) terragrunt module. This was not successful.

Error: POST url: https://cloud.getdbt.com/api/v3/accounts/97616/projects/223007/environment-variables/bulk/, status: 400, body: {"status":{"code":400,"is_success":false,"user_message":"The request was invalid. Please double check the provided data and try again.","developer_message":""},"data":{"reason":"An environment variable by this name already exists in this project and account"}}

│

│ with dbtcloud_environment_variable.deploy-environment-variable-dbt-test,

│ on dbt-deploy-environment-variable-dbt-test.tf line 1, in resource "dbtcloud_environment_variable" "deploy-environment-variable-dbt-test":

│ 1: resource "dbtcloud_environment_variable" "deploy-environment-variable-dbt-test" {

Following is my resource file (dbt-deploy-environment-variable-dbt-test.tf) resource "dbtcloud_environment_variable" "deploy-environment-variable-dbt-test" { name = "DBT_TEST" project_id = data.aws_ssm_parameter.project-id.value environment_values = { "project" : "test_project_default", "test" : "test_value" } depends_on = [ dbtcloud_environment.deploy-environment-test, dbtcloud_job.deploy-job-test ] }

From: Benoit Perigaud @.> Date: Monday, November 13, 2023 at 7:25 AM To: dbt-labs/terraform-provider-dbtcloud @.> Cc: McGrath, Brian (CAI - Carmel) @.>, Mention @.> Subject: Re: [dbt-labs/terraform-provider-dbtcloud] dbtcloud_environment_variable resource appears to have a caching issue, preventing re-creation of named environment variable. (Issue #213)

Hi @BrianMcGrathCoxhttps://urldefense.com/v3/__https:/github.com/BrianMcGrathCox__;!!Gh9pqc0J0AYYLQ!ORcdwAIrPql84z1j-6VNR8nWr31tIBugZ7t1f02uKocyZ0pOU2oQxWuQ_tO0B7nvsRT2m3RNCANg3ZHqGcGQMQrbuVETK_4$ , I just tried to replicate it locally and it didn't fail. I did the following:

1. add an env var from Terraform and apply: the env var is created
2. after it is created, comment the Terraform code for the env var and apply: the env var is deleted
3. Uncomment the code of the env var and apply: the env var is created

---

I also did the following and the resource got deleted and recreated

terraform apply -replace=dbtcloud_environment_variable.deploy-environment-variable-dbt-test

---

Could you check if the env var in question has really been deleted from dbt Cloud? And/or if someone has recreated it manually via the dbt Cloud UI?

If it has been created manually, it will need to be either deleted from the UI or imported into the Terraform state before it can be managed via Terraform.

The import instructionshttps://urldefense.com/v3/__https:/registry.terraform.io/providers/dbt-labs/dbtcloud/latest/docs/resources/environment_variable*import__;Iw!!Gh9pqc0J0AYYLQ!ORcdwAIrPql84z1j-6VNR8nWr31tIBugZ7t1f02uKocyZ0pOU2oQxWuQ_tO0B7nvsRT2m3RNCANg3ZHqGcGQMQrbnlu5Qio$ are in the docs.

— Reply to this email directly, view it on GitHubhttps://urldefense.com/v3/__https:/github.com/dbt-labs/terraform-provider-dbtcloud/issues/213*issuecomment-1808069433__;Iw!!Gh9pqc0J0AYYLQ!ORcdwAIrPql84z1j-6VNR8nWr31tIBugZ7t1f02uKocyZ0pOU2oQxWuQ_tO0B7nvsRT2m3RNCANg3ZHqGcGQMQrbz72BJt0$, or unsubscribehttps://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/AVSD6B35RFEHEKTP2SJEAGDYEIGRRAVCNFSM6AAAAAA7BQN4KCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMBYGA3DSNBTGM__;!!Gh9pqc0J0AYYLQ!ORcdwAIrPql84z1j-6VNR8nWr31tIBugZ7t1f02uKocyZ0pOU2oQxWuQ_tO0B7nvsRT2m3RNCANg3ZHqGcGQMQrbsUnkXRE$. You are receiving this because you were mentioned.Message ID: @.***>

[b-per]

dbtcloud_job.deploy-job-test makes me think of something...

Would you by any chance overwrite the specific env var in some jobs? (and/or maybe at the developer profile level)

And if you overwrite the env var for the job, do you do it via the UI? (I don't think that the provider supports it yet)

[BrianMcGrathCox]

I tried another test case.

I removed deploy for dbt environment and dbt job. So, my deployment only consists of a single dbt environment variable.

I confirmed the dbt environment variable is not in dbt project.

The deployment failed with the below error. It appears like the environment variable is cached in dbt or there is something reference it that triggers the error.

Error: POST url: https://cloud.getdbt.com/api/v3/accounts/97616/projects/223007/environment-variables/bulk/, status: 400, body: {"status":{"code":400,"is_success":false,"user_message":"The request was invalid. Please double check the provided data and try again.","developer_message":""},"data":{"reason":"An environment variable by this name already exists in this project and account"}}

│

│ with dbtcloud_environment_variable.deploy-environment-variable-dbt-test,

│ on dbt-deploy-environment-variable-dbt-test.tf line 1, in resource "dbtcloud_environment_variable" "deploy-environment-variable-dbt-test":

│ 1: resource "dbtcloud_environment_variable" "deploy-environment-variable-dbt-test" {

│

From: Benoit Perigaud @.> Date: Tuesday, November 14, 2023 at 3:04 AM To: dbt-labs/terraform-provider-dbtcloud @.> Cc: McGrath, Brian (CAI - Carmel) @.>, Mention @.> Subject: Re: [dbt-labs/terraform-provider-dbtcloud] dbtcloud_environment_variable resource appears to have a caching issue, preventing re-creation of named environment variable. (Issue #213)

dbtcloud_job.deploy-job-test makes me think of something...

Would you by any chance overwrite the specific env var in some jobs? (and/or maybe at the developer profile level)

And if you overwrite the env var for the job, do you do it via the UI? (I don't think that the provider supports it yethttps://urldefense.com/v3/__https:/github.com/dbt-labs/terraform-provider-dbtcloud/issues/99__;!!Gh9pqc0J0AYYLQ!Kk_WFBDVbx0n8PkGB0lvRmwmjHkkJ_IF-iQccMdJHbszaCnQbS3kLhWVwMFP_sHuBXt8oGHTPtHwppRkftCiEMBhGIso-dU$)

— Reply to this email directly, view it on GitHubhttps://urldefense.com/v3/__https:/github.com/dbt-labs/terraform-provider-dbtcloud/issues/213*issuecomment-1809711780__;Iw!!Gh9pqc0J0AYYLQ!Kk_WFBDVbx0n8PkGB0lvRmwmjHkkJ_IF-iQccMdJHbszaCnQbS3kLhWVwMFP_sHuBXt8oGHTPtHwppRkftCiEMBh0gjwHeo$, or unsubscribehttps://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/AVSD6BZVN6DV6RKYD2LWYETYEMQ2NAVCNFSM6AAAAAA7BQN4KCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMBZG4YTCNZYGA__;!!Gh9pqc0J0AYYLQ!Kk_WFBDVbx0n8PkGB0lvRmwmjHkkJ_IF-iQccMdJHbszaCnQbS3kLhWVwMFP_sHuBXt8oGHTPtHwppRkftCiEMBhDQHuITk$. You are receiving this because you were mentioned.Message ID: @.***>

[b-per]

Hi @BrianMcGrathCox

I have tried again to replicate the issue but can't do it.

The next course of action would be to raise a support ticket from dbt Cloud and mention that issue in the ticket. From my perspective, it doesn't seem to be an issue with the provider itself (which I could help with) but might be a problem with the API and/or some dangling config in your account and project (which the support team should be able to help with).

If you can, please mention the ticket number here once raised so I can follow up with the Support team.

[BrianMcGrathCox]

Thank you.

Can you send me the resource file you used in your test case(s)?

From: Benoit Perigaud @.> Date: Wednesday, November 15, 2023 at 3:54 AM To: dbt-labs/terraform-provider-dbtcloud @.> Cc: McGrath, Brian (CAI - Carmel) @.>, Mention @.> Subject: Re: [dbt-labs/terraform-provider-dbtcloud] dbtcloud_environment_variable resource appears to have a caching issue, preventing re-creation of named environment variable. (Issue #213)

Hi @BrianMcGrathCoxhttps://urldefense.com/v3/__https:/github.com/BrianMcGrathCox__;!!Gh9pqc0J0AYYLQ!JdX7Qgxw7LjVX6Pznkc90WIaNEL1R-IozYGPSywX3clQAGAuAoe9o670-iwz6ach6vQ5USGqsdt_pYSITzzOqe8nvEj1uLQ$

I have tried again to replicate the issue but can't do it.

The next course of action would be to raise a support ticket from dbt Cloud and mention that issue in the ticket. From my perspective, it doesn't seem to be an issue with the provider itself (which I could help with) but might be a problem with the API and/or some dangling config in your account and project (which the support team should be able to help with).

If you can, please mention the ticket number here once raised so I can follow up with the Support team.

— Reply to this email directly, view it on GitHubhttps://urldefense.com/v3/__https:/github.com/dbt-labs/terraform-provider-dbtcloud/issues/213*issuecomment-1812040869__;Iw!!Gh9pqc0J0AYYLQ!JdX7Qgxw7LjVX6Pznkc90WIaNEL1R-IozYGPSywX3clQAGAuAoe9o670-iwz6ach6vQ5USGqsdt_pYSITzzOqe8nr7VlkWw$, or unsubscribehttps://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/AVSD6B73IOBYJ7T2FHAP35LYER7M5AVCNFSM6AAAAAA7BQN4KCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMJSGA2DAOBWHE__;!!Gh9pqc0J0AYYLQ!JdX7Qgxw7LjVX6Pznkc90WIaNEL1R-IozYGPSywX3clQAGAuAoe9o670-iwz6ach6vQ5USGqsdt_pYSITzzOqe8nADMP3Ro$. You are receiving this because you were mentioned.Message ID: @.***>

[b-per]

Sure, here is an example:

resource "dbtcloud_environment_variable" "dbt_my_env_var" {
  name       = "DBT_TF_ENV_VAR"
  project_id = 2570
  environment_values = {
    "project" : "1",
    "DEV" : "2",
    "PROD": "4"
  }
}

I am able to create it, destroy it, recreate it and it works fine

[BrianMcGrathCox]

Thanks,

Can you try one more case for me?

Please remove “DEV” : “2” from environment values, then redeploy.

Please validate in dbt project after deploying.

I am testing this case, terraform says 1 resource updated successfully. However, in dbt, the resource did not get updated.

Thanks, Brian

From: Benoit Perigaud @.> Date: Wednesday, November 15, 2023 at 8:42 AM To: dbt-labs/terraform-provider-dbtcloud @.> Cc: McGrath, Brian (CAI - Carmel) @.>, Mention @.> Subject: Re: [dbt-labs/terraform-provider-dbtcloud] dbtcloud_environment_variable resource appears to have a caching issue, preventing re-creation of named environment variable. (Issue #213)

Sure, here is an example:

resource "dbtcloud_environment_variable" "dbt_my_env_var" {

name = "DBT_TF_ENV_VAR"

project_id = 2570

environment_values = {

"project" : "1",

"DEV" : "2",

"PROD": "4"

}

}

I am able to create it, destroy it, recreate it and it works fine

— Reply to this email directly, view it on GitHubhttps://urldefense.com/v3/__https:/github.com/dbt-labs/terraform-provider-dbtcloud/issues/213*issuecomment-1812558135__;Iw!!Gh9pqc0J0AYYLQ!JGSlIWmrQx8mCNV7-BadZgBr4K9E0uxiE0qDKzL8FNA99kXCNn3QJDBg9cuPjWv9vuhDmg1vKelhFsjaQePvl5hqKdbHODs$, or unsubscribehttps://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/AVSD6B5FXJOOFJFY3USHWJLYETBDHAVCNFSM6AAAAAA7BQN4KCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMJSGU2TQMJTGU__;!!Gh9pqc0J0AYYLQ!JGSlIWmrQx8mCNV7-BadZgBr4K9E0uxiE0qDKzL8FNA99kXCNn3QJDBg9cuPjWv9vuhDmg1vKelhFsjaQePvl5hqbpuDJBs$. You are receiving this because you were mentioned.Message ID: @.***>

[b-per]

I was able to reproduce this second case 👍

I will raise an issue about it to track them separately as I couldn't reproduce the first one.

[BrianMcGrathCox]

Benoit,

Have you added the second case for tracking?

From: Benoit Perigaud @.> Date: Thursday, November 16, 2023 at 10:33 AM To: dbt-labs/terraform-provider-dbtcloud @.> Cc: McGrath, Brian (CAI - Carmel) @.>, Mention @.> Subject: Re: [dbt-labs/terraform-provider-dbtcloud] dbtcloud_environment_variable resource appears to have a caching issue, preventing re-creation of named environment variable. (Issue #213)

I was able to reproduce this second case 👍

I will raise an issue about it to track them separately as I couldn't reproduce the first one.

— Reply to this email directly, view it on GitHubhttps://urldefense.com/v3/__https:/github.com/dbt-labs/terraform-provider-dbtcloud/issues/213*issuecomment-1814692117__;Iw!!Gh9pqc0J0AYYLQ!ODwkyJP-EI2-HfTUsfPOBGZVVMKCfts_dgfiaZHy-hY_Kam1ePytsS9_hY1JXqlDQi43bSQAi95_j7C5V_PpLSTMnNh7Bp0$, or unsubscribehttps://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/AVSD6BZIW2JXL5U6BYMLBL3YEYW3HAVCNFSM6AAAAAA7BQN4KCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMJUGY4TEMJRG4__;!!Gh9pqc0J0AYYLQ!ODwkyJP-EI2-HfTUsfPOBGZVVMKCfts_dgfiaZHy-hY_Kam1ePytsS9_hY1JXqlDQi43bSQAi95_j7C5V_PpLSTMkNOGKWI$. You are receiving this because you were mentioned.Message ID: @.***>

[b-per]

Yes, for the second case, I have raised #221 and added info on how to potentially fix it.

As an immediate workaround, you could just do a terraform apply -replace=dbtcloud_environment_variable.dbt_my_env_var . With the current resource we just need to destroy and recreate the env var whenever we want to delete the value for 1 env.

For the other case, please share the support ticket number if you had a chance to raise it.

[b-per]

I actually just did a new release to fix the second use case tracked in #221

[BrianMcGrathCox]

I will validate the fix for #221.

The first case I raised is #213.

Thanks, Brian

From: Benoit Perigaud @.> Date: Wednesday, November 22, 2023 at 11:58 AM To: dbt-labs/terraform-provider-dbtcloud @.> Cc: McGrath, Brian (CAI - Carmel) @.>, Mention @.> Subject: Re: [dbt-labs/terraform-provider-dbtcloud] dbtcloud_environment_variable resource appears to have a caching issue, preventing re-creation of named environment variable. (Issue #213)

I actually just did a new release to fix the second use case tracked in #221https://urldefense.com/v3/__https:/github.com/dbt-labs/terraform-provider-dbtcloud/issues/221__;!!Gh9pqc0J0AYYLQ!JpboIJZl2x_3StEAaau-c9zKsbo1Xa0hWLBikgkRkzKeIh5VxOrphXH7_ZKp5n_6WiNRBe4X6XExWTJBL1TlRmnQ9jPX-6o$

— Reply to this email directly, view it on GitHubhttps://urldefense.com/v3/__https:/github.com/dbt-labs/terraform-provider-dbtcloud/issues/213*issuecomment-1823148661__;Iw!!Gh9pqc0J0AYYLQ!JpboIJZl2x_3StEAaau-c9zKsbo1Xa0hWLBikgkRkzKeIh5VxOrphXH7_ZKp5n_6WiNRBe4X6XExWTJBL1TlRmnQrikmq2k$, or unsubscribehttps://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/AVSD6B46NDJ2D7VGXHCBMJ3YFYVLTAVCNFSM6AAAAAA7BQN4KCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMRTGE2DQNRWGE__;!!Gh9pqc0J0AYYLQ!JpboIJZl2x_3StEAaau-c9zKsbo1Xa0hWLBikgkRkzKeIh5VxOrphXH7_ZKp5n_6WiNRBe4X6XExWTJBL1TlRmnQULvDyFk$. You are receiving this because you were mentioned.Message ID: @.***>

[b-per]

Hi @BrianMcGrathCox

Great, let me know about #221.

I am not sure if I was clear when I mentioned the support ticket, but I meant raising the ticket from dbt Cloud directly

I am not on the support team myself and won't be able to troubleshoot further.

Once you have created the support ticket, you should get a ticket number back, and if you share it here I can then have a look at the discussion between you and our support team.

[BrianMcGrathCox]

Benoit,

I created the dbt cloud support ticket for the first case #213. Please let me know if I can be of further assistance.

Your request (58204) has been updated. To add additional comments, reply to this email.

58204 is the ticket number.

[b-per]

Thanks! I will keep an eye on the support ticket.

[BrianMcGrathCox]

Benoit,

I validated the fix for #221https://urldefense.com/v3/__https:/github.com/dbt-labs/terraform-provider-dbtcloud/issues/221__;!!Gh9pqc0J0AYYLQ!JlbLKJBlrG9b6kIALiiHiSDQ58isdCHpMmnqcA7OnFYO5vOhtJffODdBWjMo8P_taJAjf67oI8mlASZQAEc-pYVhyoeGCuU$.

All the test passed. This is good to go.

Thanks, Brian

From: Benoit Perigaud @.> Date: Thursday, November 30, 2023 at 4:30 AM To: dbt-labs/terraform-provider-dbtcloud @.> Cc: McGrath, Brian (CAI - Carmel) @.>, Mention @.> Subject: Re: [dbt-labs/terraform-provider-dbtcloud] dbtcloud_environment_variable resource appears to have a caching issue, preventing re-creation of named environment variable. (Issue #213)

Thanks! I will keep an eye on the support ticket.

— Reply to this email directly, view it on GitHubhttps://urldefense.com/v3/__https:/github.com/dbt-labs/terraform-provider-dbtcloud/issues/213*issuecomment-1833395351__;Iw!!Gh9pqc0J0AYYLQ!MusCmloqt2rwHmRreFgKtWe67xJbnR4R5zQFU7GdoUb5kXsKdbhQsmcND0tJiS3Z2tir_we3L5Ft2zZ47LjhJmHMxYFJsZw$, or unsubscribehttps://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/AVSD6BY5JH2Z6ZH2GKJXTTLYHBGZRAVCNFSM6AAAAAA7BQN4KCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMZTGM4TKMZVGE__;!!Gh9pqc0J0AYYLQ!MusCmloqt2rwHmRreFgKtWe67xJbnR4R5zQFU7GdoUb5kXsKdbhQsmcND0tJiS3Z2tir_we3L5Ft2zZ47LjhJmHMtQLaWeA$. You are receiving this because you were mentioned.Message ID: @.***>

[github-actions[bot]]

This issue has been marked as Stale because it has been open for 90 days with no activity. If you would like the issue to remain open, please comment on the issue or else it will be closed in 7 days.