Nike-Inc / brickflow

Pythonic Programming Framework to orchestrate jobs in Databricks Workflow
https://engineering.nike.com/brickflow/
Apache License 2.0
187 stars 44 forks source link

[FEATURE] Run Job Task #55

Closed ronnie-medscout closed 3 weeks ago

ronnie-medscout commented 1 year ago

Is your feature request related to a problem? Please describe. Databricks now supports creating a "Run Job" task, which can trigger another job by its job_id. It would be nice to have this feature in BrickFlow.

Cloud Information

Describe the solution you'd like Add a new run_job_task type. It would be nice to have something like this:

@wf.run_job_task
def trigger_downstream_job():
    return RunJobTask(
        job_id="12345",
    )

Describe alternatives you've considered I've looked at invoking dependent jobs with the Databricks API

Additional context Screenshot 2023-10-23 at 3 37 42 PM