search

calogica / dbt-expectations

Port(ish) of Great Expectations to dbt test macros
https://calogica.github.io/dbt-expectations/
Apache License 2.0
1.06k stars 130 forks source link

Table level expectations erroring out with Python code message #144

Closed vieiraAlexandre closed 2 years ago

vieiraAlexandre commented 2 years ago

Hi,

The following DBT expectations for table level data checks seemed to have stopped working without a clear cause yet:

When running the dbt test command it outputs a message seemingly from a Python context: 'str' object has no attribute 'database'

Could it be related to the way the macro reads the information received from Google BigQuery API? They did work until a certain point in time last week. On the other hand, the expectations for columns seem to keep working just fine.

Currently got these versions for dbt_expectations, dbt-bigquery and dbt. Would there be any (new) dependency clash? packages.yml

packages:
  - package: calogica/dbt_expectations
    version: 0.5.1
  - package: dbt-labs/dbt_utils
    version: 0.8.0

Pipfile

[packages]
dbt-bigquery = "==1.0.0"

DBT version installed version: 1.0.1

clausherther commented 2 years ago

Hi @vieiraAlexandre ! Thanks for flagging this. However, I can't reproduce this with Python 3.9.9 and dbt 1.0.1.

dbt-bigquery               1.0.0
dbt-core                   1.0.1
18:10:16  Installing calogica/dbt_date
18:10:16    Installed from version 0.5.2
18:10:16    Up to date!
18:10:16  Installing dbt-labs/dbt_utils
18:10:17    Installed from version 0.8.0
18:10:17    Up to date!

Here's an abbreviated output from out integration tests on BigQuery:

18:03:10  Running with dbt=1.0.1
18:03:12  Found 11 models, 93 tests, 0 snapshots, 0 analyses, 577 macros, 0 operations, 0 seed files, 0 sources, 0 exposures, 0 metrics
18:03:12
18:03:14  Concurrency: 16 threads (target='bq')
18:03:14
...
18:03:26  55 of 104 START test dbt_expectations_expect_table_column_count_to_be_between_data_test_1 [RUN]
18:03:26  56 of 104 START test dbt_expectations_expect_table_column_count_to_be_between_data_test_10 [RUN]
18:03:27  57 of 104 START test dbt_expectations_expect_table_column_count_to_be_between_data_test_10__1 [RUN]
18:03:27  58 of 104 START test dbt_expectations_expect_table_column_count_to_equal_data_test_7 [RUN]
18:03:27  59 of 104 START test dbt_expectations_expect_table_column_count_to_equal_other_table_data_test_ref_data_test_ [RUN]
18:03:28  60 of 104 START test dbt_expectations_expect_table_columns_to_contain_set_data_test_col_numeric_b__col_string_a [RUN]
18:03:28  56 of 104 PASS dbt_expectations_expect_table_column_count_to_be_between_data_test_10 [PASS in 1.60s]
18:03:28  61 of 104 START test dbt_expectations_expect_table_columns_to_match_ordered_list_data_test_idx__date_col__col_numeric_a__col_numeric_b__col_string_a__col_string_b__col_null [RUN]
18:03:28  55 of 104 PASS dbt_expectations_expect_table_column_count_to_be_between_data_test_1 [PASS in 1.91s]
18:03:28  62 of 104 START test dbt_expectations_expect_table_columns_to_match_set_data_test_idx__date_col__col_numeric_a__col_numeric_b__col_string_a__col_string_b__col_null [RUN]
18:03:29  57 of 104 PASS dbt_expectations_expect_table_column_count_to_be_between_data_test_10__1 [PASS in 1.66s]
18:03:29  58 of 104 PASS dbt_expectations_expect_table_column_count_to_equal_data_test_7.. [PASS in 1.77s]
18:03:29  60 of 104 PASS dbt_expectations_expect_table_columns_to_contain_set_data_test_col_numeric_b__col_string_a [PASS in 1.79s]
18:03:30  59 of 104 PASS dbt_expectations_expect_table_column_count_to_equal_other_table_data_test_ref_data_test_ [PASS in 2.19s]
18:03:30  61 of 104 PASS dbt_expectations_expect_table_columns_to_match_ordered_list_data_test_idx__date_col__col_numeric_a__col_numeric_b__col_string_a__col_string_b__col_null [PASS in 1.67s]
18:03:30  62 of 104 PASS dbt_expectations_expect_table_columns_to_match_set_data_test_idx__date_col__col_numeric_a__col_numeric_b__col_string_a__col_string_b__col_null [PASS in 1.82s]
18:03:35  82 of 104 START test dbt_expectations_expect_table_columns_to_match_ordered_list_timeseries_data_extended_date_day__row_value__row_value_log [RUN]
18:03:37  82 of 104 PASS dbt_expectations_expect_table_columns_to_match_ordered_list_timeseries_data_extended_date_day__row_value__row_value_log [PASS in 1.89s]
18:03:37  93 of 104 START test dbt_expectations_expect_table_columns_to_match_ordered_list_timeseries_data_date_day__row_value__row_value_log [RUN]
18:03:39  93 of 104 PASS dbt_expectations_expect_table_columns_to_match_ordered_list_timeseries_data_date_day__row_value__row_value_log [PASS in 1.67s]

18:05:08  Finished running 11 table models, 93 tests in 115.93s.
18:05:08
18:05:08  Completed successfully
18:05:08
18:05:08  Done. PASS=104 WARN=0 ERROR=0 SKIP=0 TOTAL=104
vieiraAlexandre commented 2 years ago

Hi @clausherther thanks for the feedback!

After upgrading the Python to version 3.9 the issue is indeed sorted out! I do wonder if it's anything specific to 3.8 Python version? Or at least would we have any idea why might it have happened?

Thanks!

clausherther commented 2 years ago

@vieiraAlexandre good to hear it got resolved. I have a feeling the version of Python is a red herring here, and the install of a new copy of Python did the trick.