search

spark-redshift-community / spark-redshift

Performant Redshift data source for Apache Spark
Apache License 2.0
136 stars 62 forks source link

Microsecond precision breaks datetime conversion #48

Closed eeshugerman closed 4 years ago

eeshugerman commented 4 years ago 
Caused by: java.time.format.DateTimeParseException: Text '2017-10-04 23:59:59.999999' could not be parsed, unparsed text found at index 23
        at java.time.format.DateTimeFormatter.parseResolved0(DateTimeFormatter.java:1952)
        at java.time.format.DateTimeFormatter.parse(DateTimeFormatter.java:1777)
        at io.github.spark_redshift_community.spark.redshift.Conversions$.parseRedshiftTimestamp(Conversions.scala:86)
        at io.github.spark_redshift_community.spark.redshift.Conversions$$anonfun$1$$anonfun$apply$11.apply(Conversions.scala:130)
        at io.github.spark_redshift_community.spark.redshift.Conversions$$anonfun$1$$anonfun$apply$11.apply(Conversions.scala:130)
        at io.github.spark_redshift_community.spark.redshift.Conversions$$anonfun$createRowConverter$1.apply(Conversions.scala:150)
        at io.github.spark_redshift_community.spark.redshift.Conversions$$anonfun$createRowConverter$1.apply(Conversions.scala:138)

Worth noting, from the Redshift docs:

TIMESTAMP columns store values with up to a maximum of 6 digits of precision for fractional seconds.

eeshugerman commented 4 years ago

Closing because I just noticed there is already https://github.com/spark-redshift-community/spark-redshift/issues/35

lucagiovagnoli commented 4 years ago

Yes, we should improve the library to handle 6 digits. Feel free to take a crack at it if you need this, I'm available to review

lucagiovagnoli commented 4 years ago

Fixed by #51