dbt-cloud-jobs

Version control your dbt Cloud jobs with YML.

Installation

pip install dbt-cloud-jobs

Quickstart

1. Create an API token in dbt Cloud:

   • Generate a service account token, make sure to grant the Jobs Admin permission set.
   • If you do not have access to create a service account token you can create a User API token. Note that the service account method is preferred.

2. Set an environment variable with the value of the token:

   export DBT_API_TOKEN="<VALUE_FROM_PREVIOUS_STEP>"

3. Set an environment variable for the region where your dbt Cloud account is hosted. The value must be one of "AU", "Europe" or "US" (see docs here):

   export DBT_CLOUD_REGION="<REGION>"

4. Import your existing dbt Cloud jobs:

   dbt_cloud_jobs --import --account-id 123456 --file dbt_cloud_jobs.yml

5. Edit the definition of your jobs in dbt_cloud_jobs.yml.

6. Sync the updated definitions to dbt Cloud:

   dbt_cloud_jobs --sync --file dbt_cloud_jobs.yml

Recommended usage in CI/CD

CI

In CI dbt-cloud-jobs should be used to verify that the provided YML file is valid. For example:

    - name: Install dbt_cloud_jobs
      run: pip install dbt-cloud-jobs

    - name: Validate `dbt_cloud_jobs.yml`
      run: dbt_cloud_jobs --validate --file dbt_cloud_jobs.yml

CD

In CD dbt-cloud-jobs should be used to sync the provided YML file to dbt Cloud. For example:

    - name: Install dbt_cloud_jobs
      run: pip install dbt-cloud-jobs

    - name: Sync `dbt_cloud_jobs.yml`
      run: dbt_cloud_jobs --sync --file dbt_cloud_jobs.yml

Example

For an example of how this package can be used, take a look at dbt-cloud-jobs-example-repo, specifically:

• The dbt_cloud_jobs.yml file.
• The CI pipeline.
• The CD pipeline.
• This CI pipeline run that validated the jobs definitions added in PR1.
• This CD pipeline run that updated the Daily job dbt Cloud job.

Limitations/Warnings

• Service account tokens are created at the account level. This means that if you have multiple dbt Cloud accounts you will need to create different dbt_cloud_jobs.yml files for each account. If you try to use dbt-cloud-jobs with a file that contains multiple account_id values, an error will be raised.

• ⚠️ dbt_cloud_jobs expects to "own" all the jobs in the projects where it is used. This means that if you are running dbt_cloud_jobs --sync --file dbt_cloud_jobs.yml for the first time in a project, any pre-existing jobs not present in your dbt_cloud_jobs.yml file will be deleted.

Development

To setup your development environment, fork this repository and run:

poetry shell
poetry install

Set the following environment variables:

export DBT_ACCOUNT_ID=<DBT_ACCOUNT_ID>
export DBT_CLOUD_REGION="<DBT_CLOUD_REGION>"
export DBT_ENVIRONMENT_ID=<DBT_ENVIRONMENT_ID>
export DBT_PROJECT_ID=<DBT_PROJECT_ID>
export DBT_API_TOKEN="<DBT_API_TOKEN>"

It is highly recommended that a dedicated dbt Cloud environment be used for development.

All tests can be run via:

make test

Release

Trigger the Publish to PyPi workflow, inputting the version to publish to PyPi. This workflow will:

• Publish the version to PyPi.
• Tag the HEAD commit of the main branches (tags visible here).
• Create a release (releases visible here).

Publications

• This repo formed part of a talk at Coalesce 2024: Empowering dbt developers: Self-serve dbt Cloud jobs from your dbt repo, slides