Azure / spark-cdm-connector

MIT License
76 stars 33 forks source link

[Issue] Problem datetime format read in spark connector #151

Open kaomiza opened 1 year ago

kaomiza commented 1 year ago

Did you read the pinned issues and search the error message?

No, but I will read and search it now before creating an issue.

Summary of issue

I have received data BC365 F&O a data source type Common Data Model stored in Azure Data Storage Gen2 and using pyspark read data format com.microsoft.cdm, It can't pares datetime datatype see below.

Please help me find a solution to this problem.

csv format image

pyspark read with connector image

Error stack trace

No response

Platform name

Azure Synapse

Spark version

3.3

CDM jar version

com.microsoft.cdm.BuildInfo.version

What is the format of the data you are trying to read/write?

.csv