This repository contains the following:
Multiple examples of Databricks workspace and resources deployment on Azure, AWS and GCP using Databricks Terraform provider.
Examples of implementing CI/CD pipelines to automate your Terraform deployments using Azure DevOps or GitHub Actions.
There are two ways to use this repository:
examples folder for individual examples.modules folder.Code in the repository is organized into the following folders:
modules - implementation of specific Terraform modules.examples - specific instances that use Terraform modules.cicd-pipelines - Detailed examples of implementing CI/CD pipelines to automate your Terraform deployments using Azure DevOps or GitHub Actions.Note
For detailed information about the examples, modules, or CI/CD pipelines, refer toREADME.mdfile inside the corresponding folder for a detailed guide on setting up the CI/CD pipeline.
The folder examples contains the following Terraform implementation examples :
| Cloud | Example | Description |
|---|---|---|
| Azure | adb-lakehouse | Lakehouse terraform blueprints |
| Azure | adb-with-private-link-standard | Provisioning Databricks on Azure with Private Link - Standard deployment |
| Azure | adb-vnet-injection | A basic example of VNet injected Azure Databricks workspace |
| Azure | adb-exfiltration-protection | A sample implementation of Data Exfiltration Protection |
| Azure | adb-external-hive-metastore | Example template to implement external hive metastore |
| Azure | adb-kafka | ADB - single node kafka template |
| Azure | adb-private-links | Azure Databricks Private Links |
| Azure | adb-squid-proxy | ADB clusters with HTTP proxy |
| Azure | adb-teradata | ADB with single VM Teradata integration |
| Azure | adb-uc | ADB Unity Catalog Process |
| Azure | adb-unity-catalog-basic-demo | ADB Unity Catalog end-to-end demo including UC metastore setup, Users/groups sync from AAD to databricks account, UC Catalog, External locations, Schemas, & Access Grants |
| Azure | adb-overwatch | Overwatch multi-workspace deployment on Azure |
| AWS | aws-workspace-basic | Provisioning AWS Databricks E2 |
| AWS | aws-workspace-with-firewall | Provisioning AWS Databricks E2 with an AWS Firewall |
| AWS | aws-exfiltration-protection | An implementation of Data Exfiltration Protection on AWS |
| AWS | aws-workspace-with-private-link | Coming soon |
| AWS | aws-databricks-flat | AWS Databricks simple example |
| AWS | aws-databricks-modular-privatelink | Deploy multiple AWS Databricks workspaces |
| AWS | aws-workspace-uc-simple | Provisioning AWS Databricks E2 with Unity Catalog in a single apply |
| AWS | aws-databricks-uc | AWS UC |
| AWS | aws-databricks-uc-bootstrap | AWS UC |
| AWS | aws-remote-backend-infra | Simple example on remote backend |
| AWS | aws-workspace-config | Configure workspace objects |
| GCP | gcp-sa-provisionning | Provisionning of the identity with the permissions required to deploy on GCP. |
| GCP | gcp-basic | Workspace Deployment with managed vpc |
| GCP | gcp-byovpc | Workspace Deployment with customer-managed vpc |
The folder modules contains the following Terraform modules :
| Cloud | Module | Description |
|---|---|---|
| All | databricks-department-clusters | Terraform module that creates Databricks resources for a team |
| Azure | adb-lakehouse | Lakehouse terraform blueprints |
| Azure | adb-lakehouse-uc | Provisioning Unity Catalog resources and accounts principals |
| Azure | adb-with-private-link-standard | Provisioning Databricks on Azure with Private Link - Standard deployment |
| Azure | adb-exfiltration-protection | A sample implementation of Data Exfiltration Protection |
| Azure | adb-with-private-links-exfiltration-protection | Provisioning Databricks on Azure with Private Link and Data Exfiltration Protection |
| Azure | adb-overwatch-regional-config | Overwatch regional configuration on Azure |
| Azure | adb-overwatch-mws-config | Overwatch multi-workspace deployment on Azure |
| Azure | adb-overwatch-main-ws | Main Overwatch workspace deployment |
| Azure | adb-overwatch-ws-to-monitor | Overwatch deployment on the Azure workspace to monitor |
| Azure | adb-overwatch-analysis | Overwatch analysis notebooks deployment on Azure |
| AWS | aws-workspace-basic | Provisioning AWS Databricks E2 |
| AWS | aws-databricks-base-infra | Provisioning AWS Infrastructure to be used for the deployment of a Databricks E2 workspace |
| AWS | aws-databricks-unity-catalog | Provisioning the AWS Infrastructure and setting up the metastore for Databricks Unity Catalog |
| AWS | aws-databricks-workspace | Provisioning AWS Databricks E2 Workspace using pre-created AWS Infra |
| AWS | aws-workspace-with-firewall | Provisioning AWS Databricks E2 with an AWS Firewall |
| AWS | aws-exfiltration-protection | An implementation of Data Exfiltration Protection on AWS |
| AWS | aws-workspace-with-private-link | Coming soon |
| GCP | gcp-sa-provisionning | Provisions the identity (SA) with the correct permissions |
| GCP | gcp-workspace-basic | Provisions a workspace with managed VPC |
| GCP | gcp-workspace-byovpc | Workspace with customer-managed VPC. |
The cicd-pipelines folder contains the following implementation examples of pipeline:
| Tool | CI/CD Pipeline |
|---|---|
| GitHub Actions | manual-approve-with-github-actions |
| Azure DevOps | manual-approve-with-azure-devops |
When contributing the new code, please follow the structure described in the Repository content section:
modules directory to be easily included when it's published to the Terraform registry. Prefer to implement the modular design consisting of multiple smaller modules implementing a specific functionality vs. one big module that does everything. For example, a separate module for Unity Catalog objects could be used across all clouds, so we won't need to reimplement the same functionality in cloud-specific modules.examples directory - it should show how to use the given module(s).