search

snowflakedb / spark-snowflake

Snowflake Data Source for Apache Spark.
http://www.snowflake.net
Apache License 2.0
211 stars 98 forks source link

Bug with the number of Rows of a table in the Spark UI #522

Open jbergeroncoveo opened 1 year ago

jbergeroncoveo commented 1 year ago

I have a minor bug where I see that the number of rows of a table is doubled in the spark UI. To reproduce, simply create a snowflake table with any number of rows, then, with a spark session do the following:

val dataframe = spark
    .read
    .format(SNOWFLAKE_SOURCE_NAME)
    .options(...)
    .load()

dataframe.orderBy(rand()).collect()

If you look at the spark UI in the SQL / Dataframe tab and go into the latest collect row, you should see that the number of rows shown is doubled.