search

databricks / spark-xml

XML data source for Spark SQL and DataFrames
Apache License 2.0
499 stars 226 forks source link

fs.azure.account.key error when reading files from Azure and OAuth #642

Closed DragonEnergy closed 1 year ago

DragonEnergy commented 1 year ago

Currently I am using 2.12scala on Databricks Runtime 13.1 ML GPU.

Trying to consume .xml from ADLSGen2. Getting error when trying to read with SAS token and SPN auth. Currently works with SPN to read csv parquet etc. I think the library does not use SPN OAuth from Spark config? I am not permitted to use Account Key.

srowen commented 1 year ago

Not sure, this library doesn't do anything particularly different with respect to the data source. It just builds on the text inputformat from Hadoop. It does not use DSv2, which could be an issue. See previous issues and comments on this topic. I'm not sure it can be resolved, or I don't know how; if it's related to DSv2 then I'm afraid that's out of scope for this project right now. If you can read it elsewhere, you can put it somewhere else as a copy that you can process without credentials. You can also try to read it as text then process with this library

DragonEnergy commented 1 year ago

will refer to DSv2 docs to see if that is the issue. I will try the text option still blocked.