search

Azure / spark-cdm-connector

MIT License
76 stars 33 forks source link

[Bug] Spark 3.3: Reading a table gives: java.util.NoSuchElementException: None.get #138

Closed kecheung closed 1 year ago

kecheung commented 1 year ago

Did you read the pinned issues and search the error message?

Yes, but I didn't find the answer.

Summary of issue

I am reading a table and get the following exception.

Error stack trace

None.get
java.util.NoSuchElementException: None.get
    at scala.None$.get(Option.scala:529)
    at scala.None$.get(Option.scala:527)
    at org.apache.spark.sql.execution.datasources.v2.DataSourceV2Utils$.loadV2Source(DataSourceV2Utils.scala:129)
    at org.apache.spark.sql.DataFrameReader.$anonfun$load$1(DataFrameReader.scala:209)
    at scala.Option.flatMap(Option.scala:271)
    at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:207)
    at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:171)

Platform name

Databricks

Spark version

Spark 3.3

CDM jar version

spark3.3-1.19.5

What is the format of the data you are trying to read/write?

.csv

kecheung commented 1 year ago

It's a Spark bug, and we already fixed it in Synapse. There is nothing wrong with the connector. As mentioned in the repo, the connector is designed for Synapse.

For other Spark providers, ask them to backport SPARK-41344 to Spark 3.3. The open source code was fixed for Spark 3.4+.