Open gingergenius opened 4 years ago
The problem persists even after I drop ValidFrom and ValidTo from the salesBuyingGroupsDf dataframe. I try to display() or collect() it and the same error appears even though the date is not even there anymore!
Facing the same issue with Timestamp data type. Raised the error in the below thread https://github.com/Azure/spark-cdm/issues
I've reproduced the same issue, the CDM lib read fields which are defined as datetime format in the model.json as a date. spark reader try to read that fields using a wrong format and that cause the issue. any fix for that please ?
did anyone ever figure out a fix for this? @alibouhaddou
This issue is for a: (mark with an
x
)Minimal steps to reproduce
Any log messages given by the failure
Every step starting with display(salesBuyingGroupsDf) fails with date parsing error.
Expected/desired behavior
OS and Version?
Versions
Mention any other details that might be useful
The type of the columns causing the problem is DateTime in the CDM schema. When reading data within Databricks, the columns are assigned Data type (without the time part) even though Timestamp would be more appropriate.
I tried setting the schema on read myself (not allowed because of CDM). I tried setting DateFormat and TimestampFormat settings on read (no effect). I also tried converting the columns to Timestamp or String. Basically anything I try to do with the dataframe results in the same error.