Closed nfx closed 1 month ago
394 tests ±0 351 :white_check_mark: ±0 4s :stopwatch: -1s 2 suites ±0 2 :zzz: ±0 2 files ±0 41 :x: ±0
For more details on these failures, see this check.
Results for commit f5381e03. ± Comparison against base commit 951638ba.
deployment.py
file in thedatabricks/labs/remorph/helpers
directory has been updated to add theojdbc8
library as aMavenLibrary
in the_job_recon_task
function, enabling the reconciliation process to access the Oracle Data source and pull data for reconciliation between Oracle and Databricks. TheJDBCReaderMixin
class in thejdbc_reader.py
file has also been updated to include the Oracle ojdbc8 dependent library for reconciliation during thereconcile
process. This involves installing thecom.oracle.database.jdbc:ojdbc8:23.4.0.24.05
jar as a dependent library and updating the driver class tooracle.jdbc.driver.OracleDriver
fromoracle
. A new dictionarydriver_class
has been added, which maps the driver name to the corresponding class name, allowing for dynamic driver class selection during the_get_jdbc_reader
method call. Thetest_read_data_with_options
unit test has been updated to test the Oracle connector for reading data with specific options, including the use of the correct driver class and specifying the database table for data retrieval, improving the accuracy and reliability of the reconciliation process.CommentBasedQueryExtractor
class, which accepts adialect
parameter and allows for easier configuration of the start and end comments for different SQL dialects. We have also updated the CommentBasedQueryExtractor for Snowflake and added two TSQL coverage tests to the generated report artifact to ensure that theQueryExtractor
is working correctly for TSQL queries. These changes will help ensure thorough testing and identification of TSQL and Snowflake queries during the CI/CD process.WITHIN GROUP
syntax and anOVER
clause, enabling more complex queries and data analysis. TheFixedArity
andVariableArity
classes have been updated with new methods for the supported functions, and appropriate examples have been provided to demonstrate their usage in SQL.l
in the string 'Hello world', starting the search from the second position. This feature is particularly relevant for software engineers working on data processing and analytics projects involving both Presto and Databricks SQL, as it ensures compatibility and consistent behavior between the two for string manipulation functions. The commit is part of issue #462, and the diff provided includes a new SQL file with test cases for the STRPOS function in Presto and Locate function in Databricks SQL. The test cases confirm if thehello
string is present in the greeting_message column of the greetings_table. This feature allows users to utilize the STRPOS function in Presto to determine if a specific substring is present in a string.compare.py
andexecute.py
files have been updated to include validation, and theQueryBuilder
andHashQueryBuilder
classes have new methods for validating join columns. TheSamplingQueryBuilder
,ThresholdQueryBuilder
, andrecon_capture.py
files have similar updates for validation and limiting rows for reports. Therecon_config.py
file now has a new return type for theget_join_columns
method, and a new methodtest_no_join_columns_raise_exception()
has been added in thetest_threshold_query.py
file. These changes aim to enhance data consistency, accuracy, and efficiency for software engineers.Permission Denied
exceptions. The logger messages have been updated for clarity and a new variable, README_RECON_REPO, has been created to reference the readme file for the recon_config repository. The ReconcileUtils class has been modified to handle scenarios where the recon_config file is not found or corrupted during loading, providing clear error messages and guidance for users. The unit tests for the install feature have been updated with permission checks for Catalog and Schema operations, ensuring robust handling of permission denied errors. These changes improve the system's error handling and provide clearer guidance for users encountering permission issues.recon
function in theexecute.py
file of thedatabricks.labs.remorph.reconcile
package has been updated to dynamically generate the secret name instead of hardcoding it as "secret_scope". This change utilizes the newget_key_form_dialect
function to create a secret name specific to the source dialect being used in the reconciliation process. Theget_dialect
function, along withDatabaseConfig
,TableRecon
, and the newly addedget_key_form_dialect
, have been imported fromdatabricks.labs.remorph.config
. This enhancement improves the security and flexibility of the reconciliation process by generating dynamic and dialect-specific secret names.report_types_visualisation.md
illustrates report generation for data, rows, schema, and overall reconciliation. No existing functionality was modified, ensuring the addition of valuable features for software engineers adopting this project.DashboardPublisher
class in thedashboard_publisher.py
module to streamline the process of creating and publishing dashboards in Databricks Workspace. This class simplifies dashboard creation by accepting an instance ofWorkspaceClient
andInstallation
and providing methods for creating and publishing dashboards with optional parameter substitution. Additionally, we've added a new JSON file, 'Remorph-Reconciliation-Substituted.lvdash.json', which contains a dashboard definition for a data reconciliation feature. This dashboard includes various widgets for filtering and displaying reconciliation results. We've also added a test file for the Lakeview Dashboard Publisher feature, which includes tests to ensure that theDashboardPublisher
can create dashboards using specified file paths and parameters. These new features and enhancements are aimed at improving the user experience and streamlining the process of creating and publishing dashboards in Databricks Workspace.databricks labs remorph reconcile
, has been added to initiate the Data Reconciliation process, loadingreconcile.yml
andrecon_config.json
configuration files from the Databricks Workspace. If these files are missing, the user is prompted to reinstall thereconcile
module and exit the command. The command then triggers theRemorph_Reconciliation_Job
based on the Job ID stored in thereconcile.yml
file. This simplifies the reconcile execution process, requiring users to first configure thereconcile
module and generate therecon_config_<SOURCE>.json
file usingdatabricks labs remorph install
anddatabricks labs remorph generate-recon-config
commands. The new CLI command has been manually tested and includes unit tests. Integration tests and verification on the staging environment are pending. This feature was co-authored by Bishwajit, Ganesh Dogiparthi, and SundarShankar89.Cache Maven packages
step is removed and replaced with two new steps:Run Unit Tests with Maven
and "Run Coverage Tests with Maven." The former executes unit tests and generates a test coverage report, while the latter downloads remorph-core jars as artifacts, executes coverage tests with Maven, and uploads coverage tests results as json artifacts. Thecoverage-tests
job runs after thetest-core
job and uses the same environment, checking out the code with full history, setting up Java 11 with Corretto distribution, downloading remorph-core-jars artifacts, and running coverage tests with Maven, even if there are errors. The JUnit report is also published, and the coverage tests results are uploaded as json artifacts, providing better test coverage and more reliable code for software engineers adopting the project.APPROX_PERCENTILE
function has been implemented in thepresto.py
file of thesqlglot.dialects.presto
package, allowing for approximate percentile calculations in Presto and Databricks SQL. A test file has been included for both SQL dialects, with queries calculating the approximate median of the height column in the people table. The new functionality enhances the compatibility and versatility of the remorph library in working with Presto databases and improves overall project functionality. Additionally, a new test file for Presto in the snowflakedriver project has been introduced to test expected exceptions, further ensuring robustness and reliability.ReconciliationException
, has been added as a child of theException
class, which takes two optional parameters in its constructor,message
andreconcile_output
. TheReconcileOutput
property has been created for accessing the reconcile output object. TheInvalidInputException
class now inherits fromValueError
, making the code more explicit with the type of errors being handled. A new method,_verify_successful_reconciliation
, has been introduced to check the reconciliation output status and raise aReconciliationException
if any table fails reconciliation. Thetest_execute.py
file has been updated to raise aReconciliationException
if reconciliation for a specific report type fails, and new tests have been added to the test suite to ensure the correct behavior of thereconcile
function with and without raising exceptions.USE
statements for selecting a catalog and schema has been removed in theget_sql_backend
function, thanks to the new feature provided by the lsql library. This enhancement improves code readability, maintainability, and enables better integration with the SQL backend. The commit also includes changes to the installation process for reconciliation metadata tables, providing more clarity and simplicity in the code. Additionally, several test functions have been added or modified to ensure the proper functioning of theget_sql_backend
function in various scenarios, including cases where a warehouse ID is not provided or when executing SQL statements in a notebook environment. An error simulation test has also been added for handlingDatabricksError
exceptions when executing SQL statements using theDatabricksConnectBackend
class.from dual
in from clause for oracle source (#464). In this release, we've added theget_key_from_dialect
function, replacing the previousget_key_form_dialect
function, to retrieve the key associated with a given dialect object, serving as a unique identifier for the dialect. This improvement enhances the flexibility and readability of the codebase, making it easier to locate and manipulate dialect objects. Additionally, we've modified the 'sampling_query.py' file to includefrom dual
in thefrom
clause for Oracle sources in a sampling query with a clause, enabling sampling from Oracle databases. The_insert_into_main_table
method in therecon_capture.py
file of thedatabricks.labs.remorph.reconcile
module has been updated to ensure accurate key retrieval for the specified dialect, thereby improving the reconciliation process. These changes resolve issues #458 and #464, enhancing the functionality of the sampling query builder and providing better support for various databases.yy
keyword. The DATEADD function is translated to the ADD_MONTHS function in Databricks SQL, with the number of months multiplied by 12. This ensures functional equivalence between T-SQL and Databricks SQL for date addition involving years. The tests are written as SQL scripts and are located in thetests/resources/functional/tsql/functions
directory, covering various scenarios and possible engine differences between T-SQL and Databricks SQL. The conversion process is documented, and future automation of this documentation is considered.NEXT VALUE FOR sequence
,CAST(col TO sometype)
,TRY_CAST(col TO sometype)
,JSON_ARRAY
, andJSON_OBJECT
. These functions support specialized syntax for handling data type conversions and JSON operations, including NULL value handling usingNULL ON NULL
andABSENT ON NULL
syntax. TheTSqlFunctionBuilder
class has been updated to accommodate these changes, and new test cases have been added to theTSqlFunctionSpec
test class in Scala. This enhancement enables SQL-based querying and data manipulation with increased functionality for T-SQL parser and function evaluations.DISTINCT
keyword in T-SQL for use in theSELECT
list and aggregate functions such asCOUNT
. When used in theSELECT
list,DISTINCT
ensures unique values of the specified expression are returned, and in aggregate functions likeCOUNT
, it considers only distinct values of the specified argument. This change aligns with the SQL standard and enhances the functionality of the T-SQL parser, providing developers with greater flexibility and control when usingDISTINCT
in complex queries and aggregate functions. The default behavior in SQL,ALL
, remains unchanged, and the parser has been updated to accommodate these improvements.danglingParentheses
preset option has been set to "false", removing dangling parentheses from the code. Additionally, theconfigStyleArguments
option has been set tofalse
under "optIn". These modifications to the configuration file are likely to affect the formatting and style of the Scala code in the project, ensuring consistent and organized code. This change aims to enhance the readability and maintainability of the codebase..github/ISSUE_TEMPLATE/bug.yml
file, new options for TranspileParserError, TranspileValidationError, and TranspileLateralColumnAliasError have been added to thelabel: Category of Bug / Issue
field. Additionally, a new option for ReconcileError has been included. Thefeature.yml
file in the.github/ISSUE_TEMPLATE
directory has also been updated, introducing a required dropdown menu labeled "Category of feature request." This dropdown offers options for Transpile, Reconcile, and Other categories, ensuring accurate classification and organization of incoming feature requests. The modifications aim to enhance clarity for maintainers in reviewing and prioritizing issue resolutions and feature implementations related to reconciliation functionality.uninstall
functionality of thedatabricks labs remorph
tool has been updated to align with the latest changes made to theinstall
refactoring. Theuninstall
flow now utilizes a newMockInstallation
class, which handles the uninstallation process and takes a dictionary of configuration files and their corresponding contents as input. Theuninstall
function has been modified to returnFalse
in two cases, either when there is no remorph directory or when the user decides not to uninstall. AMockInstallation
object is created for the reconcile.yml file, and appropriate exceptions are raised in the aforementioned cases. Theuninstall
function now uses aWorkspaceUnInstallation
orWorkspaceUnInstaller
object, depending on the input arguments, to handle the uninstallation process. Additionally, theMockPrompts
class is used to prompt the user for confirmation before uninstalling remorph.status
field in theDataReconcileOutput
andReconcileTableOutput
classes has been improved to comply with Python 3.11. Previously, a mutable default value was used, causing aValueError
issue. This has been addressed by implementing thedefault_factory
argument in thefield
function to ensure a new instance ofStatusOutput
is created for each class. Additionally,MismatchOutput
andThresholdOutput
classes now also utilizedefault_factory
for consistent and robust default value handling, enhancing the overall code quality and preventing potential issues arising from mutable default values.edit distance
feature for calculating the difference between two strings using the LEVENSHTEIN function. This has been achieved by adding a new method,anonymous_sql
, to theGenerator
class in thedatabricks.py
file. The method takes expressions of theAnonymous
type as arguments and calls theLEVENSHTEIN
function if thethis
attribute of the expression is equal to "EDITDISTANCE". Additionally, a new test file has been introduced for the anonymous user in the functional snowflake test suite to ensure the accurate calculation of string similarity using the EDITDISTANCE function. This change includes examples of using the EDITDISTANCE function with different parameters and compares it with the LEVENSHTEIN function available in Databricks. It addresses issue #500, which was related to testing the edit distance functionality.