This extension brings a set of tasks for you to operationalize build, test and deployment of Databricks Jobs and Notebooks.
To run this set of tasks in your build/release pipeline, you first need to explicitly set a Python version. To do so, use this task as a first task for your pipeline.
With the new tasks added for supporting Scala Development, the agent support is now defined by task. See each task documentation to check its compatibility with the available Hosted Agents.
It's strongly recommend that you use Hosted Ubuntu 1604 for your pipelines.
This pipeline task installs and configures the Databricks CLI onto the agent. The following steps are performed:
pip
(that's why using Use
Python Version is required);~/.databrickscfg
so the CLI will know which
Databricks Workspace to connect to.
Your Databricks Personal Access Token (PAT) is used to grant access to your Databricks Workspace from the Azure DevOps agent which is running your pipeline, either being it Private or Hosted.
Given that the Microsoft Hosted Agents are discarded after one use, your PAT - which was used to create the
~/.databrickscfg
- will also be discarded. This means that your PAT will not be used for anything else other than running your own pipeline.
It is strongly recommended that you do not pass your Personal Access Token as a plain text to the task. Instead, store it as a Secret Variable and use the variable reference on the task.
This Pipeline task recursively deploys Notebooks from given folder to a Databricks Workspace.
$(System.DefaultWorkingDirectory)/<artifact name>/notebooks
/Shared
/Shared/Automation
/Users/user@domain.com
Executes a notebook given its workspace path. Parameters are:
Makes the Pipeline wait until the Notebook run - invoked by the previous task - finishes.
If the Notebook execution succeeds (status SUCCESS
), this task will also succeed.
If the Notebook execution fails (status FAILED
), the task (and the Pipeline) will fail.
You can have access to the run URL through the task logs. For example:
2019-06-18T21:22:56.9840342Z The notebook execution suceeded (status SUCCESS)
2019-06-18T21:22:56.9840477Z For details, go to the execution page: https://<region>.azuredatabricks.net/?o=<organization-id>#job/<run-id>/run/1
This task will start a given Databrick Cluster. It will do nothing if the cluster is already started.
Installs the following tools on the Agent:
Installs Spark libraries on the agent.
This task will:
build.sbt
lives.build.sbt
.build.sbt
.build.sbt
.Fortunately, no known issues so far. Please feel free to open a new issue on GitHub if you experience any problem.
Please check the Release Notes page on GitHub.
To know more about how to contribute to this project, please see CONTRIBUTING page.