dagster-io / dagster

An orchestration platform for the development, production, and observation of data assets.
https://dagster.io
Apache License 2.0
11.17k stars 1.4k forks source link

[dagster-dbt] Only materialize views if there's a code change #23351

Open hello-world-bfree opened 1 month ago

hello-world-bfree commented 1 month ago

What's the use case?

When running dbt pipelines each day, I'd like to automate when views are materialized and when they don't need to be using code version changes. If the code version has not changed then even if the view is part of my selection, Dagster will not add it to the dbt build command. If it has changed then it does get added.

Allowing users to specify a dbt selection that includes views yet offload whether or not to include them at runtime to Dagster will be hugely beneficial to users and selling point for orchestrating dbt with Dagster.

Ideas of implementation

The implementation would be something similar to asset checks. Within _get_subset_selection_for_context of the DbtResource.cli method views would have their current and previous code version checked. If they match, don't include those views in the selected_dbt_resources. If they don't, include them. If the view has never ran before, materialize it.

Additional information

Message from the maintainers

Impacted by this issue? Give it a 👍! We factor engagement into prioritization.

garethbrickman commented 1 month ago

Check out this comment thread in our discussion about our upcoming Declarative Automation feature. Your input is welcome there!