dbt Cloud Webhooks - Datadog

Overview

This repo contains code to send data retreived from dbt Cloud's Metadata API and submit it as logs to datadog after receiving a webhook from dbt Cloud indicating run completion.

Requirements

Python 3.7+

datadog-api-client - Python API client for the [Datadog API](Datadog API)
dbtc - Unofficial python interface to dbt Cloud APIs
FastAPI - Modern, fast, web framework for building APIs with Python 3.7+
uvicorn - ASGI web server implementation for Python

Getting Started

Clone this repo

git clone https://github.com/dpguthrie/dbt-cloud-webhooks-datadog.git

Deploy on fly.io (Optional)

fly.io is a platform for running full stack apps and databases close to your users.

Install

Directions to install here

Once installed, sign up for fly.io

flyctl auth signup

Now sign in

flyctl auth login

Launch your app!

flyctl launch

Secrets

The following secrets need to be configured to your runtime environment for your application to work properly.

DBT_CLOUD_AUTH_TOKEN - This is the secret key that's shown after initailly creating your webhook subscription in dbt Cloud
DBT_CLOUD_SERVICE_TOKEN - Generate a service token in dbt Cloud. Ensure that it has at least the Metadata Only permission as we will be making requests against the Metadata API.
DD_API_KEY - Datadog API Key
DD_SITE - This is the datadog site (e.g. datadoghq.com)

To set a secret in your fly.io app, do the following:

flyctl secrets set DBT_CLOUD_AUTH_TOKEN=***

Or set them all at once:

flyctl secrets set DBT_CLOUD_AUTH_TOKEN=*** DBT_CLOUD_SERVICE_TOKEN=*** DD_API_KEY=*** DD_SITE=***

Other Helpful Commands

Check the secrets set in your app

flyctl secrets list

Monitor your app

flyctl monitor

Open browser to currently deployed app

flyctl open

dpguthrie / dbt-cloud-webhooks-datadog

readme