search

elementary-data / elementary

The dbt-native data observability solution for data & analytics engineers. Monitor your data pipelines in minutes. Available as self-hosted or cloud service with premium features.
https://www.elementary-data.com/
Apache License 2.0
1.83k stars 152 forks source link

From dbt-core 1.8 - Add support to load source freshness data as well #1425

Open b-per opened 5 months ago

b-per commented 5 months ago

Is your feature request related to a problem? Please describe. Until dbt-core 1.7, it is not possible to use on-run-end hooks to upload freshness data to the warehouse.

This merged PR provides this ability going forward (e.g. 1.8+)

There is also a flag that needs to be set in dbt_project.yml for the hook to be triggered

# dbt_project.yml
flags:
  source_freshness_run_project_hooks: true

Describe the solution you'd like Update the package so that source freshness data can be loaded to the warehouse as well

Maayan-s commented 2 months ago

@b-per this PR was actually contributed by @ofek1weiss from our team :) We are waiting for dbt to make the release official.

kokorin commented 2 months ago

@Maayan-s since 1.8 has been released, can we expect Elementary to use on-run-end hook to upload source freshness?

haritamar commented 2 months ago

Hi @kokorin ! Yes definitely, we'll likely prioritize this in the near future!

Shadowsong27 commented 1 week ago

Hello folks any updates on this one? We have been watching this since the release of 1.8 😃