search

astronomer / astronomer-cosmos

Run your dbt Core projects as Apache Airflow DAGs and Task Groups with a few lines of code
https://astronomer.github.io/astronomer-cosmos/
Apache License 2.0
625 stars 159 forks source link

[Feature] Write-Audit-Publish pattern support #903

Open dwreeves opened 6 months ago

dwreeves commented 6 months ago

It should be possible to do something like this but with Cosmos's DbtOperators:

set -e
dbt run --target blue
dbt test --target blue
cp target/manifest.json build-artifacts/manifest.json
dbt clone --target green --state build-artifacts/manifest.json --full-refresh

Basically, a write-audit-publish pattern. Right now, DbtDags are able to conceptualize a node as a TaskGroup; here the task group would consist of, essentially run -> test -> clone.

This is mildly related to #895, as this should be a pattern that is reasonably achievable for users to implement via the middle API layer. I think though, in the longer-term, this should be explicitly supported as a feature in Cosmos.

dosubot[bot] commented 6 months ago

It seems like you've got this under control, if you want help or have specific questions, let me know what I can do for you!

#### About Dosu This response is meant to be useful and save you time. It is **not meant to be a precise solution**, but rather a starting point for your own research. Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant. If you want to continue the conversation, start your reply with @dosu-bot.