Nike-Inc / brickflow

Pythonic Programming Framework to orchestrate jobs in Databricks Workflow
https://engineering.nike.com/brickflow/
Apache License 2.0
183 stars 36 forks source link

[FEATURE] Support remote workspace for RunJobTask #139

Closed maxim-mityutko closed 1 month ago

maxim-mityutko commented 1 month ago

Is your feature request related to a problem? Please describe. Databricks does not natively support RunJob task for remote workspaces.

Cloud Information

Describe the solution you'd like Current implementation of the RunJobTask should be able to use native Run Job functionality for the jobs in the same workspace or use API to trigger job in the remote workspace.

Describe alternatives you've considered Custom script that calls API

Additional context Add any other context or screenshots about the feature request here.