[Issue] AzureCredentialNotFoundException: Could not find ADLS Gen2 Token

[kecheung]

Did you read the pinned issues and search the error message?

Yes, but I didn't find the answer.

Summary of issue

I ran this code and it didn't work.

val outputContainer = "synapse"
var readDf = spark.read.format("com.microsoft.cdm")
.option("storage", "mystore12345.dfs.core.windows.net")
.option("manifestPath", outputContainer + "/dbtest2/default.manifest.cdm.json")
.option("entity", "TestEntity")
.load()

Error stack trace

23/02/09 20:01:52 ERROR Uncaught throwable from user code: com.databricks.backend.daemon.data.client.adl.AzureCredentialNotFoundException: Could not find ADLS Gen2 Token
    at com.databricks.backend.daemon.data.client.adl.AdlGen2CredentialContextTokenProvider.$anonfun$getToken$1(AdlGen2CredentialContextTokenProvider.scala:30)
    at scala.Option.getOrElse(Option.scala:189)
    at com.databricks.backend.daemon.data.client.adl.AdlGen2CredentialContextTokenProvider.getToken(AdlGen2CredentialContextTokenProvider.scala:24)
    at com.microsoft.cdm.utils.CDMTokenProvider.<init>(CDMTokenProvider.scala:15)
    at com.microsoft.cdm.HadoopTables.load(HadoopTables.scala:11)
    at com.microsoft.cdm.CDMCatalog.loadTable(CDMCatalog.scala:33)
    at com.microsoft.cdm.CDMCatalog.loadTable(CDMCatalog.scala:15)
    at org.apache.spark.sql.connector.catalog.CatalogV2Util$.$anonfun$loadTable$1(CatalogV2Util.scala:348)
    at com.databricks.spark.util.FrameProfiler$.record(FrameProfiler.scala:80)
    at org.apache.spark.sql.connector.catalog.CatalogV2Util$.loadTable(CatalogV2Util.scala:340)
    at org.apache.spark.sql.execution.datasources.v2.DataSourceV2Utils$.loadV2Source(DataSourceV2Utils.scala:136)
    at org.apache.spark.sql.DataFrameReader.$anonfun$load$1(DataFrameReader.scala:323)
    at scala.Option.flatMap(Option.scala:271)
    at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:321)
    at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:223)

Platform name

Databricks

Spark version

Spark 3.3

CDM jar version

spark3.3-1.19.4

What is the format of the data you are trying to read/write?

.csv

[kecheung]

Credential passthrough is a Synapse specific feature. Solution: Use app registration or SAS token auth.

Issue is the same as https://github.com/Azure/spark-cdm-connector/issues/108