search

microsoft / dbt-fabric

MIT License
79 stars 28 forks source link

Add Synapse Spark Authentication Option #152

Closed marvinbuss closed 7 months ago

marvinbuss commented 7 months ago

Add option to authenticate from Synapse Spark using mssparkutils. This is required since the azure-identity library does not work in Azure Synapse. Microsoft Fabric Spark may have a similar limitation.

Reference: https://github.com/Azure/azure-sdk-for-python/issues/26997

marvinbuss commented 7 months ago

Just sharing here some manual tests from Synapse Spark: image image image

prdpsvs commented 7 months ago

@marvinbuss , Have you tried this in both Fabric and Synapse workspace notebooks?

prdpsvs commented 7 months ago

Note that mssparkutils library is not available in pypi to add it as a dependency. The dbt-fabric adapter is missing the packege dependency. Here is an example. image

marvinbuss commented 7 months ago

@prdpsvs I have only validated Synapse as for Fabric there is no DW supported yet for the token audience as documented here: https://learn.microsoft.com/en-us/fabric/data-engineering/microsoft-spark-utilities#get-token

Note that mssparkutils library is not available in pypi to add it as a dependency. The dbt-fabric adapter is missing the packege dependency. Here is an example. image

There is only a dummy library available as documented here: https://learn.microsoft.com/en-us/azure/synapse-analytics/spark/microsoft-spark-utilities?pivots=programming-language-python#package-dependencies Would you like me to add that to the requirements-dev.txt? But you are right, if this auth method is used within another context but Synapse or Fabric Spark, this import will fail. That is why I have included the import as part of the function. This import is only being triggered when the user chooses that authentication method.

marvinbuss commented 7 months ago

@prdpsvs Please let me know how to proceed here. We can also have a quick sync to discuss how we can integrate the proposed solution. Thanks!

prdpsvs commented 7 months ago

Closing this PR as I raised another one.