Description

Databricks does not natively support riggering the job run in the remote workspace. This feature implements the logic that either creates a native RunJobTask or leverages the new RunJobInRemoteWorkspace plugin to execute both local and remote workspace job runs. The functionality does not require any additional parameters, and leverage the host parameter of the RunJobTask with the assumtion that if host is provided, user wants remote run. If / when Databricks implements this functionality natively, the code can be removed, but existing workflows won't require any changes. If host is not provided, Databricks SDK will retrieve host and token for the DEFAULT profile.

Related Issue

139

Motivation and Context

How Has This Been Tested?

Unit tests, custom workflow

Screenshots (if appropriate):

Types of changes

[ ] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

[x] My code follows the code style of this project.
[x] My change requires a change to the documentation.
[x] I have updated the documentation accordingly.
[x] I have read the CONTRIBUTING document.
[x] I have added tests to cover my changes.
[x] All new and existing tests passed.

Nike-Inc / brickflow

[FEATURE] Support remote workspace for RunJobTask #140