dbt-labs / dbt-spark

dbt-spark contains all of the code enabling dbt to work with Apache Spark and Databricks
https://getdbt.com
Apache License 2.0
395 stars 221 forks source link

Implementation of metadata-based freshness #965

Closed mikealfare closed 3 months ago

mikealfare commented 9 months ago

resolves #890

Problem

The current implementation of source freshness requires querying the data and requires the user provide a datetime field. This is slower and more expensive than it should be. It doesn't scale across multiple models. And some models do not have an appropriate datetime field.

Solution

Use source metadata where available.

Checklist

github-actions[bot] commented 3 months ago

This PR has been marked as Stale because it has been open with no activity as of late. If you would like the PR to remain open, please comment on the PR or else it will be closed in 7 days.

github-actions[bot] commented 3 months ago

Although we are closing this PR as stale, it can still be reopened to continue development. Just add a comment to notify the maintainers.