search

spark-redshift-community / spark-redshift

Performant Redshift data source for Apache Spark
Apache License 2.0
137 stars 63 forks source link

handle java.time.LocalDate/Timestamp from Spark 3.x #95

Closed honeybadgerdoesntcare closed 1 year ago

honeybadgerdoesntcare commented 3 years ago

ith Spark 3.x it's recommend to use java.time.instant over java.sql.timestamp . (reference: https://databricks.com/blog/2020/07/22/a-comprehensive-look-at-dates-and-timestamps-in-apache-spark-3-0.html) spark.conf.set("spark.sql.datetime.java8API.enabled", "true")

however with that turned on, the upload step fails, most likely because the spark-redshift lib still uses java.sql.timestamp related formats https://github.com/spark-redshift-community/spark-redshift/blob/master/src/main/scala/io/github/spark_redshift_community/spark/redshift/RedshiftWriter.scala#L240

smoy commented 1 year ago

close since it has been inactive for some time.