The dbt-native data observability solution for data & analytics engineers. Monitor your data pipelines in minutes. Available as self-hosted or cloud service with premium features.
Describe the bug
When a dbt run skips a BigQuery materialized view model, the Elementary package fails with an error Value has type STRING which cannot be inserted into column rows_affected, which has type INT64.
Looking at the SQL job in BigQuery console, I have identified the issue to be that Elementary tries to insert a value '-1' (with quotes) to the rows_affected column, which is INT64.
Describe the bug When a dbt run skips a BigQuery materialized view model, the Elementary package fails with an error
Value has type STRING which cannot be inserted into column rows_affected, which has type INT64
.Looking at the SQL job in BigQuery console, I have identified the issue to be that Elementary tries to insert a value
'-1'
(with quotes) to therows_affected
column, which isINT64
.I have further identified that the value
'-1'
is not identified as number in theinsert_rows
macro on line{%- if value is number -%}
here: https://github.com/elementary-data/dbt-data-reliability/blob/0.16.1/macros/utils/table_operations/insert_rows.sql#L191Changing the line to
{%- if value is number or value == '-1' -%}
would fix the issue.However, there's a question of how to store "skipped" run results at all?
To Reproduce Steps to reproduce the behavior:
on_configuration_change = 'continue'
configdbt run
to create the viewdbt run
Expected behavior The dbt project and Elementary package should run without errors.
Screenshots
Error message:
Environment (please complete the following information):
n/a
0.16.1
1.8.7
bigquery (1.8.3)
python:3.11-slim-bullseye
Additional context
Here's the
run_results.json
from the run. As you can see, therows_affected
is set to"-1"
.Would you be willing to contribute a fix for this issue?
Possibly yes, but there are design decision needed on how to handle run results for skipped models.