Closed hima-gopisetty closed 1 week ago
Hello @hima-gopisetty ,
Thanks for raising the issue, we are looking into it, will update.
Regards, Sujan
Hello @hima-gopisetty ,
I tried the code with local testing and its working as expected. There are no Null or Nat values getting inserted, and you need to change the code to add 10 days to column 'RECORD_CREATED_TS'
For the current_timestamp addition:
Q1. I expected that adding the current timestamp would successfully append a new column current_timestamp to each row, showing the exact current timestamp, with no null or NaT values
Ans: Its not adding null or NaT values, proper current_timestamp is getting added to the dataframe.
Q2 For the date_plus_10_days calculation:
The expectation was that the dateadd function would add 10 days to the dates in the RECORD_CREATED_TS column without any errors, accurately reflecting the new dates.
Ans: The code needs to be modified, please use the following code
sf_df_with_added_days = sf_df.with_column("date_plus_10_days", dateadd("day", lit(10), col("RECORD_CREATED_TS"))) sf_df_with_added_days.show()
Please find the updated code and output
`from snowflake.snowpark.functions import dateadd, lit, current_timestamp from snowflake.snowpark.types import StringType, StructField, StructType, TimestampType from snowflake.snowpark import Session
session = Session.builder.config('local_testing', True).create()
sf_df = session.create_dataframe( data = [ ["RL_TEST_ROLE_GRANTS", "ABC123", "2000-01-01 00:00:00"], ["RL_TEST_ROLE_SUPPORT", "ABC123", "2000-01-01 00:00:00"] ], schema=StructType([ StructField("ROLE_NAME", StringType()), StructField("GIT_COMMIT_VALUE", StringType()), StructField("RECORD_CREATED_TS", TimestampType()), ]) ) sf_df.show()
sf_ts = sf_df.with_column("current_timestamp", current_timestamp()) sf_ts.show()
sf_df_with_added_days = sf_df.with_column("date_plus_10_days", dateadd("day", lit(10), current_timestamp())) sf_df_with_added_days.show()
sf_df_with_added_days = sf_df.with_column("date_plus_10_days", dateadd("day", lit(10), col("RECORD_CREATED_TS"))) sf_df_with_added_days.show()
`
Regards, Sujan
Hi @sfc-gh-sghosh ,
Could you let me know if you are using the same snowflake-snowpark-python==1.14.0 version as mine? When I run it again locally, I encounter the same issue where it's adding NaT to rows. In fact, it's adding NaT to all rows except the first one
-------------------------------------------------------------------------------------------------
|"ROLE_NAME" |"GIT_COMMIT_VALUE" |"RECORD_CREATED_TS" |"CURRENT_TIMESTAMP" |
-------------------------------------------------------------------------------------------------
|RL_TEST_ROLE_GRANTS |ABC123 |2000-01-01 00:00:00 |2024-07-04 21:53:13.171780 |
|RL_TEST_ROLE_SUPPORT |ABC123 |2000-01-01 00:00:00 |NaT |
|RL_TEST_ROLE_SUPPORT2 |ABC123 |2000-01-01 00:00:00 |NaT |
|RL_TEST_ROLE_SUPPORT3 |ABC123 |2000-01-01 00:00:00 |NaT |
-------------------------------------------------------------------------------------------------
Hello @hima-gopisetty ,
Yes, you are right. The issue is with snowpark python 1.14.0. I am also getting NaT. But with 1.16.0 or with the latest 1.19.0, there is no such error, which means the issue has been fixed already, so please use the latest Snowpark python connector, 1.19.0.
Regards, Sujan
Please answer these questions before submitting your issue. Thanks!
1. What version of Python are you using?
2. What are the Snowpark Python and pandas versions in the environment?
3. What did you do?
I was using the Snowflake Snowpark Python library to work with DataFrames, specifically adding current timestamps and manipulating date values. Below is the complete, executable code that led to the issues
4. What did you expect to see?
For the
current_timestamp
addition:current_timestamp
to each row, showing the exact current timestamp, with no null orNaT
values.For the
date_plus_10_days
calculation:dateadd
function would add 10 days to the dates in theRECORD_CREATED_TS
column without any errors, accurately reflecting the new dates.5. What happened instead?
current_timestamp
addition:current_timestamp
unexpectedly includedNaT
values for rows whereROLE_NAME
was 'RL_TEST_ROLE_SUPPORT'.The following table illustrates the issue with the
CURRENT_TIMESTAMP
column, where one row unexpectedly contains aNaT
value:date_plus_10_days
calculation:numpy.core._exceptions._UFuncBinaryResolutionError: ufunc 'add' cannot use operands with types dtype('<M8[ns]') and dtype('O')
, indicating a problem with how data types are handled during date arithmetic in Snowpark.Additional Note: