[Crash] Plugin error when creating dbt_cloud_snowflake_credential

[alex-ld]

Context: I am trying to define dbt cloud resources:

1. project
2. database (snowflake) connection
3. environment
4. project repo (connected to github)
5. credentials (snowflake)
6. job

terraform apply threw an error when I provide the password parameter to dbt_cloud_snowflake_credential, and ran fine without it.

Code:

resource "dbt_cloud_project" "dbt_cloud_project" {
    name = "test-project"
    dbt_project_subdirectory = "dbt"
}

resource "dbt_cloud_connection" "dbt_cloud_connection" {
    name = "snowflake-sandbox-connection"
    account = "${var.snowflake_account_id}.${var.snowflake_region}"
    database = snowflake_database.database.name
    project_id = dbt_cloud_project.dbt_cloud_project.id
    role = snowflake_role.data_scientist.name
    type = "snowflake"
    warehouse = local.workspace.warehouse_size
}

resource "dbt_cloud_project_connection" "dbt_cloud_project_connection" {
    connection_id = dbt_cloud_connection.dbt_cloud_connection.connection_id
    project_id = dbt_cloud_project.dbt_cloud_project.id
}

resource "dbt_cloud_environment" "dev_environment" {
    dbt_version = "1.0.0"
    name = "Development"
    project_id = dbt_cloud_project.dbt_cloud_project.id
    type = "development"
}

resource "dbt_cloud_repository" "github_repo" {
    project_id = dbt_cloud_project.dbt_cloud_project.id
    remote_url = "git@github.com:xxx/xxx.git"
}

resource "dbt_cloud_project_repository" "github_repo" {
    project_id = dbt_cloud_project.dbt_cloud_project.id
    repository_id = dbt_cloud_repository.github_repo.repository_id
}

resource "dbt_cloud_snowflake_credential" "snowflake_credential" {
    auth_type = "password"
    num_threads = 4
    project_id = dbt_cloud_project.dbt_cloud_project.id
    schema = snowflake_schema.schema.name
    user = var.snowflake_username
    password = var.snowflake_password
}

resource "dbt_cloud_job" "test_job" {
    environment_id = dbt_cloud_environment.dev_environment.environment_id
    execute_steps = ["dbt compile"]
    name = "test_job"
    project_id = dbt_cloud_project.dbt_cloud_project.id
    triggers = {schedule: true}
    schedule_days = [1]
    schedule_hours = [0]
}

Terraform plan:

  # dbt_cloud_snowflake_credential.snowflake_credential will be updated in-place
  ~ resource "dbt_cloud_snowflake_credential" "snowflake_credential" {
        id            = "146395:154576"
      + password      = (sensitive value)
        # (7 unchanged attributes hidden)
    }

Error log:

dbt_cloud_snowflake_credential.snowflake_credential: Modifying... [id=146395:154576]
dbt_cloud_job.test_job: Modifying... [id=117792]
╷
│ Error: Plugin did not respond
│ 
│   with dbt_cloud_snowflake_credential.snowflake_credential,
│   on dbt-cloud.tf line 38, in resource "dbt_cloud_snowflake_credential" "snowflake_credential":
│   38: resource "dbt_cloud_snowflake_credential" "snowflake_credential" {
│ 
│ The plugin encountered an error, and failed to respond to the plugin.(*GRPCProvider).ApplyResourceChange call. The plugin logs may contain more
│ details.
╵
╷
│ Error: Plugin did not respond
│ 
│   with dbt_cloud_job.test_job,
│   on dbt-cloud.tf line 47, in resource "dbt_cloud_job" "test_job":
│   47: resource "dbt_cloud_job" "test_job" {
│ 
│ The plugin encountered an error, and failed to respond to the plugin.(*GRPCProvider).ApplyResourceChange call. The plugin logs may contain more
│ details.
╵

Stack trace from the terraform-provider-dbt-cloud_v0.1.0 plugin:

panic: interface conversion: interface {} is nil, not bool

goroutine 39 [running]:
github.com/gthesheep/terraform-provider-dbt-cloud/pkg/resources.resourceJobUpdate({0x18b3b08, 0xc0005431a0}, 0xc00040ce00, {0x179c200, 0xc00042a900})
        github.com/gthesheep/terraform-provider-dbt-cloud/pkg/resources/job.go:326 +0x1045
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).update(0xc000186fc0, {0x18b3b40, 0xc000606a80}, 0xd, {0x179c200, 0xc00042a900})
        github.com/hashicorp/terraform-plugin-sdk/v2@v2.12.0/helper/schema/resource.go:376 +0x12e
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).Apply(0xc000186fc0, {0x18b3b40, 0xc000606a80}, 0xc000544f70, 0xc00040cc80, {0x179c200, 0xc00042a900})
        github.com/hashicorp/terraform-plugin-sdk/v2@v2.12.0/helper/schema/resource.go:482 +0x785
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*GRPCProviderServer).ApplyResourceChange(0xc0003923f0, {0x18b3a98, 0xc000308000}, 0xc00043a370)
        github.com/hashicorp/terraform-plugin-sdk/v2@v2.12.0/helper/schema/grpc_provider.go:1021 +0xdaa
github.com/hashicorp/terraform-plugin-go/tfprotov5/tf5server.(*server).ApplyResourceChange(0xc000240640, {0x18b3b40, 0xc000606390}, 0xc0003da150)
        github.com/hashicorp/terraform-plugin-go@v0.8.0/tfprotov5/tf5server/server.go:812 +0x56b
github.com/hashicorp/terraform-plugin-go/tfprotov5/internal/tfplugin5._Provider_ApplyResourceChange_Handler({0x1787be0, 0xc000240640}, {0x18b3b40, 0xc000606390}, 0xc000542180, 0x0)
        github.com/hashicorp/terraform-plugin-go@v0.8.0/tfprotov5/internal/tfplugin5/tfplugin5_grpc.pb.go:385 +0x170
google.golang.org/grpc.(*Server).processUnaryRPC(0xc0002aa8c0, {0x18c0818, 0xc0004024e0}, 0xc00060c480, 0xc0003a2150, 0x1d7ba60, 0x0)
        google.golang.org/grpc@v1.45.0/server.go:1282 +0xccf
google.golang.org/grpc.(*Server).handleStream(0xc0002aa8c0, {0x18c0818, 0xc0004024e0}, 0xc00060c480, 0x0)
        google.golang.org/grpc@v1.45.0/server.go:1619 +0xa2a
google.golang.org/grpc.(*Server).serveStreams.func1.2()
        google.golang.org/grpc@v1.45.0/server.go:921 +0x98
created by google.golang.org/grpc.(*Server).serveStreams.func1
        google.golang.org/grpc@v1.45.0/server.go:919 +0x294

Error: The terraform-provider-dbt-cloud_v0.1.0 plugin crashed!

This is always indicative of a bug within the plugin. It would be immensely
helpful if you could report the crash with the plugin's maintainers so that it
can be fixed. The output above should help diagnose the issue.

[GtheSheep]

Hey! Thanks for finding this and providing such detail, super helpful, will take a look later today 👌

[GtheSheep]

Hey, think I've tracked this down, for triggers on the dbt_cloud_job to be valid, I seem to be requiring all keys are set (rather than defaulting to false), I'll see if I can add a validation func to that resource input and log a better error, that trace was grim 😬 i.e.

triggers = {
    github_webhook: false,
    git_provider_webhook: false,
    schedule: true,
    custom_branch_only: false,
}

tbh I thought this was handled, but clearly not, so will add another test also 😞

[alex-ld]

Hi @GtheSheep thanks for the response. That seems to solve the problem.

Another question related to this resource if you don't mind me reusing this thread: Where is dbt_cloud_snowflake_credential actually being used? With this resource created, dbt Cloud still complains about IDE credential not being set.

Credentials Required
You'll need to configure your development credentials to write code in dbt Cloud. For more information, consult the [IDE setup guide](https://docs.getdbt.com/docs/running-a-dbt-project/using-the-dbt-ide/#setting-up-developer-credentials).

Currently, it seems like the only way to provide IDE credentials is via the console.

It is also not used as environment credentials for running job.

This environment does not have any credentials associated with it. Edit the environment to add credentials, then try this job again.

[GtheSheep]

hey @alex-ld - Just gave it a go using the code you provided (with my credentials in place ofc), If I run TF apply then the snowflake credential is found at Account Settings -> Project (the new test one) -> Connection, if I open it up I can see the credentials I provided, is this not what you expected/ is happening for you? I did notice the line warehouse = local.workspace.warehouse_size, just wanted to check that's just a weird name? as that should be the warehouse name not the size. Hope that's some help 😸

[alex-ld]

Thanks for the heads up on the warehouse, I renamed it to warehouse_name which is more proper.

As for the connection, please see the screenshot. There's no user/pw here. They are in the project environment credential page.

[GtheSheep]

Hmm interesting, presumably you're doing this with an admin level access token? Won't get round to it today but will try out your example again tomorrow and see if I can replicate this

[GtheSheep]

Hey, so after a bit more playing around with the API, credentials just seem to disappear into a bit of an abyss until they're associated with something, like an Environment, where you can then browse the UI for them, otherwise they seem to just come back in the API GET response, otherwise most of the pages show Connection info rather than Credential info

[GtheSheep]

tbh looking into this has made me realise I could learn a bit more about it and ideally refactor how connections/ credentials look in the provider to make it more obvious how the flow should work (the other open issue is about Databricks so v relevant to this)

[alex-ld]

@GtheSheep I believe in this package you are using the V3 API? To be honest I think dbt Cloud and the API is half-baked, and poorly documented. Even doing click-ops is not clear sometimes, let alone doing IAC.

If you could provide more details or documentations you used to develop this package, I might be able to contribute to this package too.

[GtheSheep]

Hey, yeah it's mostly V3, little bit of V2 in places but yeah it's not been the easiest, in terms of figuring out how things work I either introspect the UI using chrome or something, or use the postman collection https://www.postman.com/dbtlabs/workspace/dbt-cloud-api-documentation/overview

Quite a lot involves trial and error of the read/ update/ create/ delete requests in a notebook to get all the args, etc, as it expects

Any help would be hugely appreciated!

[GtheSheep]

But thanks again for bringing this up, has been a while since the initial credential/ connection stuff went in, was v specific to my use case and for sure could do with a revisit/ better docs on how it mimics the UI flow

[github-actions[bot]]

This issue is stale because it has been open for 30 days with no activity.

[github-actions[bot]]

This issue is stale because it has been open for 45 days with no activity.

[github-actions[bot]]

This issue is stale because it has been open for 45 days with no activity.

[github-actions[bot]]

This issue was closed because it has been inactive for 7 days since being marked as stale.