spark-redshift-community / spark-redshift

Performant Redshift data source for Apache Spark
Apache License 2.0
135 stars 62 forks source link

Writing Timestamps back to redshift truncates microseconds #66

Open rhelmstetter opened 4 years ago

rhelmstetter commented 4 years ago

The code to write timestamps truncates the microseconds to three digits. Since redshift supports up to six digits, the write should keep as much precision as Redshift supports.

/**
   * Formatter for formatting timestamps for insertion into Redshift TIMESTAMP columns.
   *
   * This formatter should not be used to parse timestamps returned from Redshift UNLOAD commands;
   * instead, use [[Timestamp.valueOf()]].
   *
   * Note that Java Formatters are NOT thread-safe, so you should not re-use instances of this
   * SimpleDateFormat across threads.
   */
  def createRedshiftTimestampFormat(): SimpleDateFormat = {
    new SimpleDateFormat("yyyy-MM-dd HH:mm:ss.SSS")
  }