Describe the bug
While creating a dependency between two tasks created using DatabricksTaskOperator() does not use the task_key specified, but uses dagName__groupId__taskKey. This is inconsistent with the tasks created on Databricks because they correctly use the task_key specified.
To Reproduce
Steps to reproduce the behavior:
Run the following code with a valid cluster config and update the path to two notebooks on databricks which could simply print hello.
Describe the bug While creating a dependency between two tasks created using DatabricksTaskOperator() does not use the
task_key
specified, but usesdagName__groupId__taskKey
. This is inconsistent with the tasks created on Databricks because they correctly use thetask_key
specified.To Reproduce Steps to reproduce the behavior:
hello
.Expected behavior This should create a DAG with two tasks - print_1 and print_2 - and print_2 should be dependent on print_1.
Screenshots![image](https://github.com/astronomer/astronomer-providers/assets/9607477/e8d1eab0-8dd6-44c0-9efb-df4ddf324ed1)
Desktop (please complete the following information):