brooklyn-data / dbt_artifacts

A dbt package for modelling dbt metadata. https://brooklyn-data.github.io/dbt_artifacts
Apache License 2.0
309 stars 113 forks source link

Add source freshness status to sources table #166

Open emilyjia opened 1 year ago

emilyjia commented 1 year ago

Thanks for the updates to version 1.0.0. Is there a way to access the status of source freshness tests in the existing models? I see the freshness criteria in the sources table, but not the freshness status. Alternatively, is it possible to add the status property (from sources.json) to the sources table?

NiallRees commented 1 year ago

Hey @emilyjia - this is not possible today while we wait for https://github.com/dbt-labs/dbt-core/issues/5609 but thank you for bringing it up!

rudeb0y commented 1 year ago

hi @NiallRees

I was looking at this as we also want to use the freshness status.

status is present in sources.yml, after dbt source freshness - and it does not get overwritten after dbt run

but when we upload the freshness only has criteria and no freshness status. Could we not add column for status if the key is present it is there, otherwise null? Is there a reason to wait for the issue mentioned?

glsdown commented 1 year ago

Hi @rudeb0y apologies for the delay in getting back to you here. dbt artifacts relies on the run result object provided via the on-run-end hook rather than the files created themselves. This doesn't include the source freshness status though unfortunately. The PR that Niall has referenced aims to make this data available though, which is why we are unable to use it until that has been implemented.

zs-gpillai commented 9 months ago

is this feature implemented or are there any timelines on this ? Since we are also looking for the same.

VDFaller commented 5 months ago

Just mentioning that https://github.com/dbt-labs/dbt-core/issues/5609 is merged.

aryan-tmg commented 3 months ago

Just pinging... Any update on this feature? https://github.com/dbt-labs/dbt-core/issues/5609 is merged

martindlarsson commented 1 month ago

This would be great. Any timeline when it can get implemented?