search

Datavault-UK / automate-dv

A free to use dbt package for creating and loading Data Vault 2.0 compliant Data Warehouses (powered by dbt, an open source data engineering tool, registered trademark of dbt Labs)
https://www.automate-dv.com
Apache License 2.0
505 stars 129 forks source link

[FEATURE] Support multi-source Effectivity Satellites #131

Open DVAlexHiggs opened 2 years ago

DVAlexHiggs commented 2 years ago

Is your feature request related to a problem? Please describe. In the case where multiple sources feed a Link, we will also need to track multiple sources in an Effectivity Satellite, which is not currently supported by dbtvault.

Describe the solution you'd like Allow multiple sources to be provided to the source_model parameter and handle the multi-sources in the eff_sat macro.

Describe alternatives you've considered Having a single effectivity satellite per Link could work but this is awkward and not ideal.

Additional context Highlighted by a user on Slack

AB#5374

philmaddocks commented 10 months ago

Are there any updates as to when this feature will be available in AutomateDV? (I've asked about it in the Slack Channel also)

We've got scenarios where we have multiple sources populating a link for the same business relationship. This is been due to data being migrated to new source systems which then become the master of the relationship

Example: Adviser data belonged in the core insurance platform but has since been migrated to Salesforce. This has meant the relationship between an adviser and relationship_manager has also moved to Salesforce. This is fine as the link can be loaded from multiple sources, but we aren't currently able to track effectivity across the link. We could create an eff sat per source which hangs off the link which will say in each source what is the effective relationship, but this isn't ideal.