astronomer / astro-provider-databricks

Orchestrate your Databricks notebooks in Airflow and execute them as Databricks Workflows
Apache License 2.0
21 stars 11 forks source link

The launch task currently has a downstream dependency on all tasks, not just direct ones #28

Open jlaneve opened 1 year ago

jlaneve commented 1 year ago

Because of this line below, the launch task has a dependency on everything downstream of it (instead of just direct dependencies). This doesn't actually affect the functionality, but the graph looks uglier. See graph screenshot.

            task.databricks_metadata = create_databricks_workflow_task.output
Screen Shot 2023-04-11 at 12 22 36 AM