search

Datavault-UK / automate-dv

A free to use dbt package for creating and loading Data Vault 2.0 compliant Data Warehouses (powered by dbt, an open source data engineering tool, registered trademark of dbt Labs)
https://www.automate-dv.com
Apache License 2.0
511 stars 131 forks source link

[FEATURE] Add an Effectivity Satellites for a Hub ? #250

Closed LocNguyend closed 1 week ago

LocNguyend commented 1 month ago

When transitioning to Data Vault, I also want to convert Satellites to SCD Type 2 so that the code in data science can remain unchanged. I have read about Effectivity Satellites, but they only apply to Links. During the incremental run, how can I automatically update the end_date and start_date columns to the Satellites? or I need to update it in the stage layer then satellites can capture it's in hashdiff or some different way ?

DVAlexHiggs commented 1 week ago

Hi, thanks for this question. Effectivity Satellites for Hubs are not part of the standards, and raw vault objects no longer have end dating in the Data Vault 2.0 standards since it became fully insert-only. Our implementation of Effectivity Satellites with 'auto-end-dating' functionality is a bit of an outlier and is opt-in because it's technically an application of a business rule, however it is there as a convenience for advanced users.

You shouldn't be working out effectivity on things in the Raw Vault, but instead applying logic downstream in the business vault. Further, if you want SCD Type 2, there are standards for this in Data Vault 2.0 as well.

This is an integral part of the Data Vault standards, so I suggest exploring this topic further to deepen your understanding. I'd encourage you to post this question on the Data Vault forums, where you can receive valuable insights from other industry professionals and prominent community members, such as Patrick Cuba. This could help deepen your understanding of the topic.