Closed DragonEnergy closed 1 year ago
Not sure, this library doesn't do anything particularly different with respect to the data source. It just builds on the text inputformat from Hadoop. It does not use DSv2, which could be an issue. See previous issues and comments on this topic. I'm not sure it can be resolved, or I don't know how; if it's related to DSv2 then I'm afraid that's out of scope for this project right now. If you can read it elsewhere, you can put it somewhere else as a copy that you can process without credentials. You can also try to read it as text then process with this library
will refer to DSv2 docs to see if that is the issue. I will try the text option still blocked.
Currently I am using 2.12scala on Databricks Runtime 13.1 ML GPU.
Trying to consume .xml from ADLSGen2. Getting error when trying to read with SAS token and SPN auth. Currently works with SPN to read csv parquet etc. I think the library does not use SPN OAuth from Spark config? I am not permitted to use Account Key.