`terraform apply` fails when trying to update `deferring_job_id` on a `dbt_cloud_job` resource.

[waligob]

We had previously imported a dbt Cloud job into Terraform with the following specifications:

resource "dbt_cloud_job" "cicd" {
  environment_id = dbt_cloud_environment.cicd.environment_id
  execute_steps = [
    "dbt build --select state:modified+ result:error+"
  ]
  generate_docs        = false
  is_active            = true
  name                 = "CICD"
  num_threads          = 4
  project_id           = var.dbt_cloud_project_id
  run_generate_sources = false
  target_name          = "dev"
  deferring_job_id     = dbt_cloud_job.production_run.id
  timeout_seconds      = 0
  schedule_type        = "custom_cron"
  schedule_cron        = "0 12 0 0 0"
  triggers = {
    "custom_branch_only" : false,
    "github_webhook" : false,
    "git_provider_webhook" : false,
    "schedule" : false
  }
}

We have made no recent changes to the CICD dbt Cloud job, but we first noticed today that terraform plan is detecting the following needed change for the deferring_job_id:

Terraform will perform the following actions:

  # dbt_cloud_job.cicd will be updated in-place
  ~ resource "dbt_cloud_job" "cicd" {
      ~ deferring_job_id     = 0 -> 225359
        id                   = "225361"
        name                 = "CICD"
        # (16 unchanged attributes hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

Note that job ID 225359 has been the stable job ID for our production run for several months now, so it's surprising to see Terraform reporting 0 as the job Id. However, upon running terraform apply, we get the following error:

dbt_cloud_job.cicd: Modifying... [id=225361]
╷
│ Error: POST url: https://cloud.getdbt.com/api/v2/accounts/38962/jobs/225361/, status: 405, body: {"status":{"code":405,"is_success":false,"user_message":"The requested method isn't allowed on this resource. Please double-check your request method and try again.","developer_message":""},"data":{"can_only_defer_to_an_environment_or_self.":"A job can only defer to an environment or itself."}}
│ 
│   with dbt_cloud_job.cicd,
│   on dbt_cloud.tf line 108, in resource "dbt_cloud_job" "cicd":
│  108: resource "dbt_cloud_job" "cicd" {
│ 
╵

Given that the configuration provided for dbt_cloud_job.cicd agrees with the documented schema, we would expect this plan to be successfully applied (even though it's not clear how deferring_job_id was changed to 0)

Affected versions:

• Terraform v1.4.6 on darwin_arm64
• provider registry.terraform.io/dbt-labs/dbtcloud v0.2.7

[b-per]

Hi @waligob ! Thanks for raising this issue and adding all those details.

I think that this is related to the improvements the team is releasing around CI where deferral can then be done at the environment level rather than at the job level.

I will most likely need to release a new version of the provider allowing deferral to be made at the env_id rather than the job_id. I am reaching out to the internal engineering team and will provide updates here.

[b-per]

In the meantime, could you please raise a support ticket sending an email to support@getdbt.com , mentioning this ticket?

[waligob]

In the meantime, could you please raise a support ticket sending an email to support@getdbt.com , mentioning this ticket?

@b-per I just emailed support. Thank you!

[waligob]

Support ticket number is 54243.

[b-per]

Thanks, I asked the team to revert to the older behaviour temporarily on your account until I release an update to the Terraform provider.

[b-per]

The team moved your account back to the previous behaviour for now. I will release a new version of the provider early next week most likely allowing people to set the deferral at the environment level.

[waligob]

The team moved your account back to the previous behaviour for now. I will release a new version of the provider early next week most likely allowing people to set the deferral at the environment level.

Thanks, @b-per! I've confirmed that we're successfully able to run terraform apply against the resource now:

Terraform will perform the following actions:

  # dbt_cloud_job.cicd will be updated in-place
  ~ resource "dbt_cloud_job" "cicd" {
      ~ deferring_job_id     = 0 -> 225359
        id                   = "225361"
        name                 = "CICD"
        # (16 unchanged attributes hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

dbt_cloud_job.cicd: Modifying... [id=225361]
dbt_cloud_job.cicd: Modifications complete after 0s [id=225361]

Apply complete! Resources: 0 added, 1 changed, 0 destroyed.

[b-per]

Thanks for reporting back!

Just be aware that we will soon roll out deferral to environments instead of jobs to all dbt Cloud accounts, and when it is available for your account you will need to replace deferring_job_id = 225359 by deferring_environment_id = <your_prod_env_id> (I am finalising the updates to the resource)

[b-per]

This is now available in 0.2.8 and docs are updated! https://registry.terraform.io/providers/dbt-labs/dbtcloud/latest/docs/resources/job

[waligob]

That's great, @b-per. When we upgrade to >=0.2.8, will we need to ask the dbt team to restore the API behavior that was reverted here?

[b-per]

No, 0.2.8 allows you to set the deferral at both the environment level and job level (depending on what your account is set to right now). So you can use >=0.2.8 even if your account still defers at job level.

You will only need to use the deferral to env level when your account will be updated.