Add support for U2M Azure Active Directory authentication

[alexott]

Describe the feature

Many Azure customers prohibit the use of the PATs and force them to use AAD tokens. Right now, Databricks DBT extension supports only M2M AAD authentication, but there is no support for U2M AAD authentication, especially using azure-cli. Other Databricks components - Python SDK, Terraform, new CLI, Databricks Connect v2, ... support U2M AAD authentication in the form of unified authentication.

It would be really useful to implement U2M AAD authentication in Databricks DBT extension for cases when DBT is run from the local machine.

Describe alternatives you've considered

no real alternative

Additional context

Please include any other relevant context here.

Who will this benefit?

Unblock use of DBT + Databricks by Azure customers

Are you interested in contributing this feature?

maybe

[benc-db]

Thanks for filing the ticket. Per our discussion out of band, our documentation on this front is confusing at best. We will also be discussing what needs to be done to support unified auth.

[alexott]

One of the things that is handy in other tools is the use of azure-cli to obtain an AAD token - there is no need for creating a dedicated SPN, etc.

[ferdyh]

We're using a Service Principal in Azure Devops to authenticate for Databricks (Azure RM Service connection) and then we can define an AzureCLI Task with service connection to execute commands using the Databricks CLI; This doesn't work for dbt. Does this issue also cover this scenario? This would make deployments possible without storing credentials anywhere...

[github-actions[bot]]

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please remove the stale label or comment on the issue.