snowflakedb / spark-snowflake

Snowflake Data Source for Apache Spark.
http://www.snowflake.net
Apache License 2.0
213 stars 98 forks source link

Support trailing comments in passed SQL queries #580

Open sfc-gh-hachouraria opened 1 month ago

sfc-gh-hachouraria commented 1 month ago

To support trailing comments, particularly the inline types with -- or //, the query wrapping performed in the connector needs to ensure the query is placed into a new line of its own

Otherwise, passing such an inline comment will break the Snowflake SQL syntax.

Attempting any of the following, for example, fails today:

spark.sql("select * from table -- trailing comment").collect()
spark.read.[…].option("query", "select * from table -- trailing comment").load()

The failure occurs due to Spark wrapping around the query and executing a resulting invalid syntax:

select col from (select * from table -- trailing comment)

The change made here works by adjusting such wrapping to always, safely place provided statements into their own line:

select col from (
select * from table -- trailing comment
)

This allows for a valid syntax to be used despite users accidentally placing trailing comments at the end.

The tests added fail without the fixes added (tested via sbt test it:test).