Azure DevOps fixes - Githubissues

databricks / mlops-stacks

This repo provides a customizable stack for starting new ML projects on Databricks that follow production best-practices out of the box.

https://docs.databricks.com/en/dev-tools/bundles/mlops-stacks.html

Apache License 2.0

460 stars 156 forks source link

Azure DevOps fixes #25

Closed niall-turbitt closed 1 year ago

niall-turbitt commented 1 year ago

Summary of Azure DevOps fixes/updates:

Fix triggers for CI/CD pipelines. Previously, pipelines were triggered on pushes to any branch (IndividualCI). Now CI pipelines are only triggered on PR to default branch. Terraform deploy now triggers when PR has been successfully merged to default branch (and only on changes to /databricks/config/**) [ML-27221]
- Added required permissions for build service to enable this. We set up build validation on the default branch in terraform cicd boostrap. In order for the Terraform deploy step to be able to write to the default branch after successfully merging a PR to main, we must give the build service permissions to override this branch policy. [ML-27222]
Version requirements for terraform, databricks-cli, azure-cli [ML-27220]
List explicit PAT permissions required [ML-27219]
Change Terraform output variables to camelCase in order to make them compatible with Azure Key Vault [ML-27218]

niall-turbitt commented 1 year ago

Summary of Azure DevOps fixes/updates:

Fix triggers for CI/CD pipelines. Previously, pipelines were triggered on pushes to any branch (IndividualCI). Now CI pipelines are only triggered on PR to default branch. Terraform deploy now triggers when PR has been successfully merged to default branch (and only on changes to /databricks/config/**) [ML-27221]
- Added required permissions for build service to enable this. We set up build validation on the default branch in terraform cicd boostrap. In order for the Terraform deploy step to be able to write to the default branch after successfully merging a PR to main, we must give the build service permissions to override this branch policy. [ML-27222]
Version requirements for terraform, databricks-cli, azure-cli [ML-27220]
List explicit PAT permissions required [ML-27219]
Change Terraform output variables to camelCase in order to make them compatible with Azure Key Vault [ML-27218]

niall-turbitt commented 1 year ago

variable ARM_SUBSCRIPTION_ID is required in the terraform_cicd yml due to changing the variable name in the variable group from ARM_SUBSCRIPTION_ID to armSubscriptionId (required to be compatible with Key Vault)

See this issue for more details

niall-turbitt commented 1 year ago

@vladimirk-db tested setting up from scratch with cookiecutter https://github.com/niall-turbitt/mlops-stack.git -c niall-turbitt/ado-fixes and working for CUJs for changing ML code, and changing databricks-config/** code

jerrylian-db commented 1 year ago

@niall-turbitt Made some small fixes and merging this PR. Thanks for your work!