search

open-metadata / OpenMetadata

OpenMetadata is a unified metadata platform for data discovery, data observability, and data governance powered by a central metadata repository, in-depth column level lineage, and seamless team collaboration.
https://open-metadata.org
Apache License 2.0
5.58k stars 1.05k forks source link

Table diff tests returns 'Login failed' error #17760

Open hanseliezer opened 2 months ago

hanseliezer commented 2 months ago

Affected module Does it impact the UI, backend or Ingestion Framework? Backend (?)

Describe the bug A clear and concise description of what the bug is.

Running a table diff test returns a Login failed for user '[username]' error according to the test logs. This is even when this diff test is comparing two tables which were part of the same database and ingestion connection, and part of a test suite with other table/column data quality tests, and the other tests were successfully run (i.e. there were no login failures for these other tests).

To Reproduce image

There are three data quality tests for this CUST table focusing on a single column called NPWP; the data diff test is "NPWP - consistency".

image

This CUST table is compared with another table called NEGATIVE_CUST which is part of the same database and ingestion (CONFINS Ingestion.CONFINS.dbo.).

An excerpt from the logs is as follows:

[2024-09-09T04:49:07.781+0000] {test_case_runner.py:105} DEBUG - Found 3 test cases for table CONFINS Ingestion.CONFINS.dbo.CUST
[2024-09-09T04:49:07.782+0000] {core.py:33} INFO - Executing test case NPWP - consistency for entity CONFINS Ingestion.CONFINS.dbo.CUST
[2024-09-09T04:49:08.082+0000] {tableDiff.py:105} ERROR - Unexpected error while running the table diff test: ('28000', "[28000] [Microsoft][ODBC Driver 18 for SQL Server][SQL Server]Login failed for user 'sa'. (18456) (SQLDriverConnect)")
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.10/site-packages/data_diff/databases/mssql.py", line 195, in create_connection
    connection = self._mssql.connect(**self._args)
pyodbc.InterfaceError: ('28000', "[28000] [Microsoft][ODBC Driver 18 for SQL Server][SQL Server]Login failed for user 'sa'. (18456) (SQLDriverConnect)")
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.10/site-packages/metadata/data_quality/validations/table/sqlalchemy/tableDiff.py", line 89, in run_validation
    return self._run()
  File "/home/airflow/.local/lib/python3.10/site-packages/metadata/data_quality/validations/table/sqlalchemy/tableDiff.py", line 117, in _run
    result = self.get_column_diff()
  File "/home/airflow/.local/lib/python3.10/site-packages/metadata/data_quality/validations/table/sqlalchemy/tableDiff.py", line 313, in get_column_diff
    changed = self.get_incomparable_columns()
  File "/home/airflow/.local/lib/python3.10/site-packages/metadata/data_quality/validations/table/sqlalchemy/tableDiff.py", line 170, in get_incomparable_columns
    ).with_schema()
  File "/home/airflow/.local/lib/python3.10/site-packages/data_diff/table_segment.py", line 153, in with_schema
    return self._with_raw_schema(self.database.query_table_schema(self.table_path))
  File "/home/airflow/.local/lib/python3.10/site-packages/data_diff/databases/base.py", line 1048, in query_table_schema
    rows = self.query(self.select_table_schema(path), list, log_message=path)
  File "/home/airflow/.local/lib/python3.10/site-packages/data_diff/databases/base.py", line 996, in query
    res = self._query(sql_code)
  File "/home/airflow/.local/lib/python3.10/site-packages/data_diff/databases/base.py", line 1248, in _query
    return r.result()
  File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 458, in result
    return self.__get_result()
  File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/usr/local/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/airflow/.local/lib/python3.10/site-packages/data_diff/databases/base.py", line 1253, in _query_in_worker
    raise self._init_error
  File "/home/airflow/.local/lib/python3.10/site-packages/data_diff/databases/base.py", line 1242, in set_conn
    self.thread_local.conn = self.create_connection()
  File "/home/airflow/.local/lib/python3.10/site-packages/data_diff/databases/mssql.py", line 198, in create_connection
    raise ConnectError(*error.args) from error
data_diff.databases.base.ConnectError: ('28000', "[28000] [Microsoft][ODBC Driver 18 for SQL Server][SQL Server]Login failed for user 'sa'. (18456) (SQLDriverConnect)")
[2024-09-09T04:49:08.083+0000] {tableDiff.py:113} DEBUG - ERROR: Unexpected error while running the table diff test: ('28000', "[28000] [Microsoft][ODBC Driver 18 for SQL Server][SQL Server]Login failed for user 'sa'. (18456) (SQLDriverConnect)")
[2024-09-09T04:49:08.087+0000] {core.py:33} INFO - Executing test case NPWP - uniqueness for entity CONFINS Ingestion.CONFINS.dbo.CUST
[2024-09-09T04:49:11.344+0000] {core.py:33} INFO - Executing test case NPWP - valid length for entity CONFINS Ingestion.CONFINS.dbo.CUST
[2024-09-09T04:49:14.099+0000] {ingestion_pipeline_mixin.py:52} DEBUG - Created Pipeline Status for pipeline CONFINS Ingestion.CONFINS.dbo.CUST.testSuite.f61c8c2c-2a04-4977-8321-5d96dab1ed91: runId='d3ff0e68-23e9-4e65-827a-a522ee5b8817' pipelineState=<PipelineState.success: 'success'> startDate=Timestamp(root=1725857346176) timestamp=Timestamp(root=1725857346176) endDate=Timestamp(root=1725857353969) status=IngestionStatus(root=[StepSummary(name='OpenMetadata', records=1, updated_records=0, warnings=0, errors=0, filtered=0, failures=None), StepSummary(name='Data Quality', records=4, updated_records=0, warnings=0, errors=0, filtered=0, failures=None), StepSummary(name='OpenMetadata', records=4, updated_records=0, warnings=0, errors=0, filtered=0, failures=None)])
[2024-09-09T04:49:14.100+0000] {logger.py:175} INFO - Statuses detailed info:
[2024-09-09T04:49:14.101+0000] {logger.py:175} INFO - OpenMetadata Status:
[2024-09-09T04:49:14.102+0000] {logger.py:175} INFO - {'failures': [],
 'filtered': [],
 'records': ['Tests for [CONFINS Ingestion.CONFINS.dbo.CUST]'],
 'source_start_time': 1725857346.5167072,
 'updated_records': [],
 'warnings': []}
[2024-09-09T04:49:14.102+0000] {logger.py:175} INFO - Data Quality Status:
[2024-09-09T04:49:14.103+0000] {logger.py:175} INFO - {'failures': [],
 'filtered': [],
 'records': ['CONFINS Ingestion.CONFINS.dbo.CUST.NPWP - consistency',
             'CONFINS Ingestion.CONFINS.dbo.CUST.NPWP.NPWP - uniqueness',
             'CONFINS Ingestion.CONFINS.dbo.CUST.NPWP - valid length',
             'NPWP - uniqueness,NPWP - valid length,NPWP - consistency'],
 'source_start_time': 1725857346.5223281,
 'updated_records': [],
 'warnings': []}

indicating the other two tests were successfully completed. This is confirmed by the first picture where the marker colour was red (fail), whereas the data diff test's colour was yellow (aborted).

Expected behavior A clear and concise description of what you expected to happen.

There is no Login failed error for the data diff test.

Version:

Additional context Add any other context about the problem here.

TeddyCr commented 1 month ago

cc @sushi30