search

snowflakedb / spark-snowflake

Snowflake Data Source for Apache Spark.
http://www.snowflake.net
Apache License 2.0
215 stars 99 forks source link

Support micro seconds precisison during copy unload #492

Open arthurli1126 opened 1 year ago

arthurli1126 commented 1 year ago

Snowflake connector uses SimpleDateFormat (legacy) for data time parsing during copy unload, internally, it only supports milliseconds. For instance, for string 2023-03-01 07:54:56.191173 it would consider it carries 191173 milliseconds so it will add 191000 / 1000 / 60 = 3 mins 11s and put 173 microseconds to milliseconds filed: 2023-03-01 07:58:07.173000.

arthurli1126 commented 1 year ago

@sfc-gh-mrui The test cases has been fixed please review. Thanks!

sfc-gh-mrui commented 1 year ago

@arthurli1126 Let's me know if you are ready for review. So far, below test case failed.

23/03/27 19:12:32 INFO DAGScheduler: Job 109 finished: collect at DataTypesIntegrationSuite.scala:244, took 0.006617 s
[info] - filter on timestamp column *** FAILED ***
arthurli1126 commented 1 year ago

@sfc-gh-mrui Yea the tests pass in my local and with instacart's snowflake account(based on aws), I realized the test only failed with azure and gcp copy unload so wondering if there's anything snowflake does differently when unload to gcp/azure. However, I'm not able to verify it due to my company only has aws deployment. Is this something we can help me with or can you provide a testing temp GCP deployment account?

arthurli1126 commented 1 year ago

Looks I can sign up for a trial account with snowflake on GCP. Let me also try that.

arthurli1126 commented 1 year ago

Sorry about the long break finally got chance to look into this again. For workflows in different envs, does it need approval from your side to run? @sfc-gh-mrui

arthurli1126 commented 1 year ago

Hi @sfc-gh-mrui, I was wondering if there's a way to trigger the test again? Thanks 🙏