Databricks does not natively support riggering the job run in the remote workspace. This feature implements the logic that either creates a native RunJobTask or leverages the new RunJobInRemoteWorkspace plugin to execute both local and remote workspace job runs. The functionality does not require any additional parameters, and leverage the host parameter of the RunJobTask with the assumtion that if host is provided, user wants remote run. If / when Databricks implements this functionality natively, the code can be removed, but existing workflows won't require any changes.
If host is not provided, Databricks SDK will retrieve host and token for the DEFAULT profile.
Related Issue
139
Motivation and Context
How Has This Been Tested?
Unit tests, custom workflow
Screenshots (if appropriate):
Types of changes
[ ] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)
Checklist:
[x] My code follows the code style of this project.
[x] My change requires a change to the documentation.
Description
Databricks does not natively support riggering the job run in the remote workspace. This feature implements the logic that either creates a native RunJobTask or leverages the new RunJobInRemoteWorkspace plugin to execute both local and remote workspace job runs. The functionality does not require any additional parameters, and leverage the
host
parameter of the RunJobTask with the assumtion that if host is provided, user wants remote run. If / when Databricks implements this functionality natively, the code can be removed, but existing workflows won't require any changes. If host is not provided, Databricks SDK will retrieve host and token for the DEFAULT profile.Related Issue
139
Motivation and Context
How Has This Been Tested?
Unit tests, custom workflow
![image](https://github.com/Nike-Inc/brickflow/assets/43565398/3cdf57ed-7697-4d3d-991a-834810b2f66b)
Screenshots (if appropriate):
Types of changes
Checklist: