Azure / spark-cdm-connector

MIT License
76 stars 33 forks source link

How to handle binary datatype while using Spark CDM connector in Azure Synapse Analytics #141

Open sultanmkhan opened 1 year ago

sultanmkhan commented 1 year ago

Did you read the pinned issues and search the error message?

Yes, but I didn't find the answer.

Summary of issue

I am trying to read F&O data with Spark CDM Connector but currently facing an error of Binary datatype. I do not want to define the schema myself as there are 30 tables is there any other solution?

Error stack trace

An error occurred while calling o3947.load. : java.util.NoSuchElementException: No value found for 'Binary' at scala.Enumeration.$anonfun$withName$1(Enumeration.scala:157) at scala.collection.immutable.HashMap$HashTrieMap.getOrElse0(HashMap.scala:596) at scala.collection.immutable.HashMap.getOrElse(HashMap.scala:73) at scala.Enumeration.withName(Enumeration.scala:157)

Platform name

Azure Synapse Spark Pool

Spark version

3.2

CDM jar version

1.19.4

What is the format of the data you are trying to read/write?

.csv