A free to use dbt package for creating and loading Data Vault 2.0 compliant Data Warehouses (powered by dbt, an open source data engineering tool, registered trademark of dbt Labs)
Is your feature request related to a problem? Please describe.
AutomateDV assumes that the data being pointed at the macros is always deltas. In our architecture, we have a data lake where we create an event base history for each source table. We'd like to point macros at this (and can for the most part) but apply_source_filter is not across all of the macros.
Describe the solution you'd like
I'd like to have the apply_source_filter config available on all macros. This would allow idempotent builds over event base history tables from source where we can simply restart jobs, or full-refresh the data vault should issues arise. I would expect this to be available on all macros excluding possibly hubs and links.
Describe alternatives you've considered
The config is already there for the main sat macro, but I don't see any harm adding it to more as an option so there is the flexibility to limit the data considered by the macro rather than having to create a delta layer.
Is your feature request related to a problem? Please describe. AutomateDV assumes that the data being pointed at the macros is always deltas. In our architecture, we have a data lake where we create an event base history for each source table. We'd like to point macros at this (and can for the most part) but apply_source_filter is not across all of the macros.
Describe the solution you'd like I'd like to have the apply_source_filter config available on all macros. This would allow idempotent builds over event base history tables from source where we can simply restart jobs, or full-refresh the data vault should issues arise. I would expect this to be available on all macros excluding possibly hubs and links.
Describe alternatives you've considered The config is already there for the main sat macro, but I don't see any harm adding it to more as an option so there is the flexibility to limit the data considered by the macro rather than having to create a delta layer.
AB#5352