Revert "Spark 3: Cache CDM Date and DateTime type methods (#100)"

Azure / spark-cdm-connector

MIT License

76 stars 33 forks source link

Revert "Spark 3: Cache CDM Date and DateTime type methods (#100)" #160

Closed kecheung closed 7 months ago

kecheung commented 7 months ago

This reverts commit 14048081dc4b7aa5306ad15578e1d41f928bbd59. It can lead to undesired behavior and is difficult to enforce. When the timestamp data is in CSV, there is many to one mapping in the common data model.

Recommended solution is to use parquet format.