search

elementary-data / elementary

The dbt-native data observability solution for data & analytics engineers. Monitor your data pipelines in minutes. Available as self-hosted or cloud service with premium features.
https://www.elementary-data.com/
Apache License 2.0
1.94k stars 165 forks source link

All anomaly checks failing after upgrading to 0.16.0 #1671

Closed henriklaurentz closed 3 months ago

henriklaurentz commented 3 months ago

Describe the bug All our anomaly checks are raising column "metric_type" does not exist in data_monitoring_metrics during dbt test after upgrading to dbt 0.16.0.

To Reproduce Steps to reproduce the behavior:

1.Apply some "model level" tests

    data_tests:
      - elementary.table_anomalies:
          tags: ["elementary"]
          severity: warn
      - elementary.dimension_anomalies:
          tags: ["elementary"]
          severity: warn
          dimensions:
            - tool
      - elementary.all_columns_anomalies:
          tags: ["elementary"]
          severity: warn
  1. Run dbt test
  2. Receive errors along the lines of: 
    [2024-08-13, 10:28:03 UTC] {pod_manager.py:447} INFO - [base]   Database Error in test elementary_all_columns_anomalies_slv_events_document_analytics_ (models/silver/events/production/events.yml)
[2024-08-13, 10:28:03 UTC] {pod_manager.py:447} INFO - [base]   column "metric_type" does not exist in data_monitoring_metrics

Expected behavior That the anomaly tests runs simply successfully, as on former versions.

Screenshots The columns in data_monitoring_metrics, closest to metric_type being metric_name

image

Environment (please complete the following information):

Additional context To clarify, dbt deps + dbt run -s elementary has been ran before running dbt test on version 0.16.0

Would you be willing to contribute a fix for this issue? Not sure if that would be the most pragmatic approach :)

haritamar commented 3 months ago

Hi @henriklaurentz ! Can you please clarify if the dbt run -s elementary runs were executed on the same dbt profile as your production environment? Because this should fix the issue (we indeed added a field to data_monitoring_metrics)

henriklaurentz commented 3 months ago

Oh my sweet lawd, my apologies, that was very sloppy of me. At least staging is 100%+ upgraded :)

Thanks for the reply and have a nice evening!