dbtcloud provider fails for DbtCloudRepository resource when using CDKTF

jaklan commented 4 months ago

Originally created here by @kpawlo7:

Expected Behavior

I am trying to run below code to add this 2 resources. But it is falling:

import os
from constructs import Construct
from cdktf import App, TerraformStack, TerraformOutput, Token
from imports.dbtcloud.provider import DbtcloudProvider
from imports.dbtcloud.project import Project
from imports.dbtcloud.connection import Connection
from imports.dbtcloud.dbt_cloud_repository import DbtCloudRepository

class MyStack(TerraformStack):
    def __init__(self, scope: Construct, id: str):
        super().__init__(scope, id)
        DbtcloudProvider(
            self,
            id="dbtcloud",
            account_id=1,
            token=os.environ.get("DBT_CLOUD_TOKEN"),
            host_url=os.environ.get("HOST_URL"),
        )
        POC_project = Project(self, id_="dbtcloud_project", name="Personal: terraform")

        redshift_database = Connection(
            self,
            id_="dbtcloud_connection",
            project_id=Token.as_number(POC_project.id),
            type="redshift",
            name="connection for POC",
            database="gtm_glo_src_temp",
            port=5439,
            host_name=os.environ.get("HOST_NAME"),
        )
  #THIS RESOURCE FAILS!:
        gitlab_repo = DbtCloudRepository(
            self,
            id_="dbtcloud_repository",
            project_id=Token.as_number(POC_project.id),
            remote_url="",
            gitlab_project_id=8765,
            git_clone_strategy="deploy_token",
        )

app = App()
MyStack(app, "cdktf-gitlab")

app.synth()

Actual Behavior

cdktf diff output:

~/terraform/cdktf-gitlab$ cdktf diff
cdktf-gitlab  Initializing the backend...
cdktf-gitlab  Initializing provider plugins...
              - Reusing previous version of dbt-labs/dbtcloud from the dependency lock file
cdktf-gitlab  - Finding latest version of hashicorp/dbt...
cdktf-gitlab  - Using previously-installed dbt-labs/dbtcloud v0.2.20
cdktf-gitlab  ╷
              │ Error: Failed to query available provider packages
              │ 
              │ Could not retrieve the list of available versions for provider hashicorp/dbt:
              │ provider registry registry.terraform.io does not have a provider named
              │ registry.terraform.io/hashicorp/dbt
              │ 
              │ All modules should specify their required_providers so that external
              │ consumers will get the correct providers when using a module. To see which
              │ modules are currently depending on hashicorp/dbt, run the following command:
              │     terraform providers
              ╵

⠴  Processing
Error: terraform init failed with exit code 1

Steps to Reproduce

Set 3 env variables: DBT_CLOUD_TOKEN, HOST_URL, HOST_NAME to connect to dbt cloud: export DBT_CLOUD_TOKEN="" export HOST_URL="" export HOST_NAME =""
Run cdktf diff on this code:

import os
from constructs import Construct
from cdktf import App, TerraformStack, TerraformOutput, Token
from imports.dbtcloud.provider import DbtcloudProvider
from imports.dbtcloud.project import Project
from imports.dbtcloud.connection import Connection
from imports.dbtcloud.dbt_cloud_repository import DbtCloudRepository

class MyStack(TerraformStack):
    def __init__(self, scope: Construct, id: str):
        super().__init__(scope, id)
        DbtcloudProvider(
            self,
            id="dbtcloud",
            account_id=1,
            token=os.environ.get("DBT_CLOUD_TOKEN"),
            host_url=os.environ.get("HOST_URL"),
        )
        POC_project = Project(self, id_="dbtcloud_project", name="Personal: terraform")

        redshift_database = Connection(
            self,
            id_="dbtcloud_connection",
            project_id=Token.as_number(POC_project.id),
            type="redshift",
            name="connection for POC",
            database="gtm_glo_src_temp",
            port=5439,
            host_name=os.environ.get("HOST_NAME"),
        )
        # gitlab_repo = DbtCloudRepository(
        #     self,
        #     id_="dbtcloud_repository",
        #     project_id=Token.as_number(POC_project.id),
        #     remote_url="",
        #     gitlab_project_id=8765,
        #     git_clone_strategy="deploy_token",
        # )

app = App()
MyStack(app, "cdktf-gitlab")

app.synth()

It gives this output:

cdktf diff
cdktf-gitlab  Initializing the backend...
cdktf-gitlab  Initializing provider plugins...
              - Reusing previous version of dbt-labs/dbtcloud from the dependency lock file
cdktf-gitlab  - Using previously-installed dbt-labs/dbtcloud v0.2.20
cdktf-gitlab  Terraform has been successfully initialized!

              You may now begin working with Terraform. Try running "terraform plan" to see
              any changes that are required for your infrastructure. All Terraform commands
              should now work.

              If you ever set or change modules or backend configuration for Terraform,
              rerun this command to reinitialize your working directory. If you forget, other
              commands will detect it and remind you to do so if necessary.
cdktf-gitlab  Terraform used the selected providers to generate the following execution
              plan. Resource actions are indicated with the following symbols:
cdktf-gitlab  + create

              Terraform will perform the following actions:

                # dbtcloud_connection.dbtcloud_connection (dbtcloud_connection) will be created
                + resource "dbtcloud_connection" "dbtcloud_connection" {
                    + adapter_id       = (known after apply)
                    + allow_keep_alive = false
                    + allow_sso        = false
                    + connection_id    = (known after apply)
                    + database         = "gtm_glo_src_temp"
                    + host_name        = ""
                    + id               = (known after apply)
                    + is_active        = true
                    + name             = "connection for POC"
                    + port             = 5439
                    + project_id       = (known after apply)
                    + tunnel_enabled   = false
                    + type             = "redshift"
                  }

                # dbtcloud_project.dbtcloud_project (dbtcloud_project) will be created
                + resource "dbtcloud_project" "dbtcloud_project" {
                    + id   = (known after apply)
                    + name = "Personal: terraform"
                  }

              Plan: 2 to add, 0 to change, 0 to destroy.

              ─────────────────────────────────────────────────────────────────────────────

              Saved the plan to: plan

              To perform exactly these actions, run the following command to apply:
                  terraform apply "plan"

Uncoment DbtCloudRepository part.
Run it again and error will appear with this resource.

Versions

cdktf debug

language: python
cdktf-cli: 0.20.3
node: v18.17.0
cdktf: 0.20.3
constructs: 10.3.0
jsii: 1.94.0
terraform: 1.7.4
arch: x64
os: linux 5.4.226-129.415.amzn2.x86_64
python: Python 3.10.6
pip: pip 22.0.2 from /home/jovyan/platform/.venv3/lib/python3.10/site-packages/pip (python 3.10)
pipenv: pipenv, version 2023.12.1
providers
dbt-labs/dbtcloud@~> 0.2 (LOCAL)
        terraform provider version: 0.2.20

Providers

┌───────────────────┬──────────────────┬───────┬────────────┬──────────────┬─────────────────┐ │ Provider Name │ Provider Version │ CDKTF │ Constraint │ Package Name │ Package Version │ ├───────────────────┼──────────────────┼───────┼────────────┼──────────────┼─────────────────┤ │ dbt-labs/dbtcloud │ 0.2.20 │ │ ~> 0.2 │ │ │ └───────────────────┴──────────────────┴───────┴────────────┴──────────────┴─────────────────┘

Gist

No response

Possible Solutions

Probably the problem is with cdktf itself: Looking inside: imports/dbtcloud/dbt_cloud_repository I can see this class is referencing wrong terraform resource:

it reference:

https://registry.terraform.io/providers/dbt-labs/dbtcloud/0.2.20/docs/resources/dbt_cloud_repository

and it should:

https://registry.terraform.io/providers/dbt-labs/dbtcloud/0.2.20/docs/resources/repository

Workarounds

No response

Anything Else?

No response

References

No response

Help Wanted

[ ] I'm interested in contributing a fix myself

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

b-per commented 4 months ago

Hi!

I will need to spend a bit of time to debug it, but my first idea is that this line looks odd:

from imports.dbtcloud.dbt_cloud_repository import DbtCloudRepository

Should/could it be

from imports.dbtcloud.repository import Repository

or

from imports.dbtcloud.dbtcloud_repository import DbtcloudRepository

The Terraform provider contains:

old resources, starting with dbt_cloud.
new resources, starting with dbtcloud.

I kep the old ones for a bit because they were used in the original provider but I will remove them in the next minor verson

kpawlo7 commented 4 months ago

Yep. With this one:

from imports.dbtcloud.repository import Repository

Works:

           # dbtcloud_repository.dbtcloud_repository (dbtcloud_repository) will be created
                + resource "dbtcloud_repository" "dbtcloud_repository" {
                    + azure_bypass_webhook_registration_failure = false
                    + deploy_key                                = (known after apply)
                    + fetch_deploy_key                          = false
                    + git_clone_strategy                        = "deploy_token"
                    + gitlab_project_id                         = 8765
                    + id                                        = (known after apply)
                    + is_active                                 = true
                    + project_id                                = (known after apply)
                    + repository_credentials_id                 = (known after apply)
                    + repository_id                             = (known after apply)
                  }

              Plan: 5 to add, 0 to change, 0 to destroy.

Thx!

dbt-labs / terraform-provider-dbtcloud