search

snowflakedb / spark-snowflake

Snowflake Data Source for Apache Spark.
http://www.snowflake.net
Apache License 2.0
211 stars 98 forks source link

spark connector for snowflake unable to handle NULLs in numeric columns #546

Open prabodh1194 opened 6 months ago

prabodh1194 commented 6 months ago

I have a glue job using the snowflake spark connector. It keeps failing when it sees null values in the dataframe. I am unable to debug this issue.

Glue 4 is using this snowflake driver -- spark-snowflake_2.12-2.12.0-spark_3.3.jar

error:
 Numeric value 'null' is not recognized
sfc-gh-sghosh commented 5 months ago

Hello @prabodh1194 ,

Thanks for raising the issue.

I tried to reproduce the issue, but its working fine. A table with numeric column having null. created a dafaframe out of it. ` df = spark.read.format(SNOWFLAKE_SOURCE_NAME).options(**sfOptions).option("dbtable", "my_table_int").load()

Show DataFrame content

df.show()`

Ouptut:

+----+-----+ | ID| NAME| +----+-----+ | 1| John| | 2|Alice| |null| null| | 4| Bob| |null| null| +----+-----+ Could you provide the code snippet ( small sample application) so we can reproduce the issue.

Regards, Sujan