Closed sundarshankar89 closed 1 month ago
394 tests ±0 367 :white_check_mark: ±0 4s :stopwatch: -1s 2 suites ±0 0 :zzz: ±0 2 files ±0 27 :x: ±0
For more details on these failures, see this check.
Results for commit 72eaa877. ± Comparison against base commit 884dd5a9.
closed as there are CI failures
LEFT
andRIGHT
were incorrectly being allowed as identifiers in join statements. These keywords are actually hard keywords in Snowflake SQL and must be escaped when used as column names. The solution implemented involves removingLEFT
from the list of non-reserved words and addingRIGHT
to this list in the SnowflakeParser.g4 file. Additionally, the commit introduces a more precise examination of the lookahead through a member function call when handling cases whereLEFT
orRIGHT
are used as column names without escaping, avoiding the use of semantic predicates which are generally discouraged. The tests for translating queries with LEFT JOIN and RIGHT JOIN have been updated to reflect these changes in the grammar. Furthermore, theignore
keyword has been removed from test cases that were previously marked asTODO
to be fixed, indicating that the tests are now expected to pass.read_data
function in thedatabricks.py
file and updating the installation queries for thereconcile
package. Theread_data
function has been enhanced to handle cases where thecatalog
variable is None more accurately by checking its existence before concatenating it to form the table name. The main table's FQN in the installation queries has been modified to remove the invalid null constraint and allow nullable values for thecatalog
field undersource_table
andtarget_table
STRUCT types. These changes, co-authored by Ganesh Dogiparthi and SundarShankar89, improve the flexibility and accuracy of the library, enhancing its overall functionality.IGNORE NULLS
andRESPECT NULLS
clauses in windowing functions to their equivalent functionality in Databricks SQL. In TSQL, certain windowing functions such as LEAD or LAG allow for the specification ofIGNORE NULLS
orRESPECT NULLS
, which influence their behavior. In Databricks SQL, equivalent functions take an optional trailing boolean parameter that indicates whether trailing nulls should be ignored. This update appends the boolean option to the Databricks windowing functions when theIGNORE NULLS
clause is specified, withRESPECT NULLS
being the default case. These changes are implemented in the TSqlExpressionBuilder.scala file and are tested using the test file for the TSQL function parser, with no modifications to other parts of the codebase.FOR
in TSQL option clauses without requiring escaping. This change involves expanding the ANTLR rule for parsing generic options in the TSqlParser.g4 file, allowing for the correct parsing of options containingFOR
as a keyword. This resolves issue #525 and ensures that TSQL statements like "SELECT * FROM t FOR XML RAW OPTION (OPTIMIZE FOR UNKNOWN)" are correctly parsed, even with the unescaped use of "FOR." The code now specifically handles T-SQL options containing theFOR
keyword in theOptionBuilder
class, eliding it and managing a few particular options such as "OPTIMIZE FOR UNKNOWN," which is now parsed asOPTIMIZE
with an optionalUNKNOWN
identifier. These updates enhance the consistency and accuracy of parsing T-SQL options within our codebase (Co-authored by Valentin Kasas)ReconcileConfig
in thedeployment.py
module of thedatabricks/labs/remorph/helpers
package. This configuration object is used to pass reconciliation configuration data to theDeployment
class and includes a new parameterrecon_config
in the__init__
method. The_job_recon_task
method has been updated to include the Oracle library in thelibraries
list only if the reconciliation source is Oracle. We have also added two new fixtures,oracle_recon_config
andsnowflake_recon_config
, in the test suite for different configurations of theReconcileConfig
class. Thetest_deploy_job
andtest_deploy_job_with_valid_state
tests have been updated to accept these fixtures as arguments and pass them to theJobDeployer
constructor. Additionally, thetest_deploy_job_in_gcp
test has been updated to set theis_gcp
attribute of theWorkspace
object toTrue
. These changes ensure the appropriate database library is used based on the provided configuration, improving the efficiency and reliability of the reconciliation job.