While creating a dependency between two tasks created using DatabricksTaskOperator() does not use the task_key specified, but uses dagName__groupId__taskKey. This is inconsistent with the tasks created on Databricks because they correctly use the task_key specified.
To Reproduce
Steps to reproduce the behavior:
Run the following code with a valid cluster config and update the path to two notebooks on databricks which could simply print hello.
Describe the bug
While creating a dependency between two tasks created using
DatabricksTaskOperator()
does not use thetask_key
specified, but usesdagName__groupId__taskKey
. This is inconsistent with the tasks created on Databricks because they correctly use thetask_key
specified.To Reproduce
Steps to reproduce the behavior:
Run the following code with a valid cluster config and update the path to two notebooks on databricks which could simply print
hello
.Screenshots
Expected behavior
This should create a DAG with two tasks -
print_1
andprint_2
- andprint_2
should be dependent onprint_1
.Desktop (please complete the following information):
OS: macos Ventura 13.6.1 Browser Firefox Version 123.0.1