The databricks task names are generated as combination of dag_id and task_id in notebook operator but this will overflow the character limit in workflow jobs, which has only 100 characters limit. Need a change in _get_databricks_task_id to return the trimmed task id if the taskgroup is workflowtaskgroup type.
This will also give cleaner look on the databricks UI to easily identify the task names.
The databricks task names are generated as combination of dag_id and task_id in notebook operator but this will overflow the character limit in workflow jobs, which has only 100 characters limit. Need a change in
_get_databricks_task_idto return the trimmed task id if the taskgroup is workflowtaskgroup type.This will also give cleaner look on the databricks UI to easily identify the task names.