Nike-Inc / brickflow

Pythonic Programming Framework to orchestrate jobs in Databricks Workflow
https://engineering.nike.com/brickflow/
Apache License 2.0
183 stars 36 forks source link

[FEATURE] SQL Job Task #119

Closed RajuGujjalapati closed 2 months ago

RajuGujjalapati commented 3 months ago

Is your feature request related to a problem? Please describe. Databricks now supports creating a "SQL" task, which can allowing you to create, schedule, operate, and monitor workflows that include Databricks SQL objects such as queries, legacy dashboards, sql files and alerts.. It would be nice to have this feature in BrickFlow.

Cloud Information

Describe the solution you'd like SQL task has 4 other sub tasks will be implementing all of them, also will add a new sql_type type. It would be nice to have something like this:

# For SQL query
@wf.sql_task
def sample_sql_task_query() -> any:
    return SqlTask(
        query_id="4e16dc24-e30d-4683-96d7-cf7da4e263ad", warehouse_id="044a6f42ad7d914a"
    )

# For File
@wf.sql_task
def sample_sql_task_file() -> any:
    return SqlTask(file_path="products/brickflow_test/src/sql/sql_task_file_test.sql", warehouse_id="044a6f42ad7d914a")

# For alerts
@wf.sql_task
def sample_sql_alert() -> any:
    return SqlTask(
        alert_id="41ca5e33-21c2-40a2-8f77-183351d2b566",
        pause_subscriptions=False,
        subscriptions={
            "usernames": ["raju.gujjalapati@nike.com", "Mohanasilpa.Palla@nike.com"]
        },
        warehouse_id="044a6f42ad7d914a",
    )

# For Dashboards
@wf.sql_task
def sample_sql_dashboard():
    return SqlTask(
        dashboard_id="f57447ca-e8d4-4dad-a66c-f524464e52a8",
        dashboard_custom_subject="Raju Legacy Dashboard Test",
        pause_subscriptions=True,
        subscriptions={
            "usernames": ["rabc@abc.com", "abc.cd@abc.com"],
            "destination_id": ["434354545"],
        },
        warehouse_id="044a6fad7d914a",
    )

Describe alternatives you've considered I've looked at invoking dependent jobs with the Databricks API

Additional context

image