search

Datavault-UK / automate-dv

A free to use dbt package for creating and loading Data Vault 2.0 compliant Data Warehouses (powered by dbt, an open source data engineering tool, registered trademark of dbt Labs)
https://www.automate-dv.com
Apache License 2.0
511 stars 131 forks source link

[DOCS] Dependent child handling missing from links #228

Open koillinengit opened 8 months ago

koillinengit commented 8 months ago

Need to do dependent child link and satellite, but it seems to impossible to add link unhashed keys that are not described as hub's.

It should be possible to add for example invoice line numbers in link that has only invoice hub. Line number should be unhashed but should be included to link hashkey. Invoice line satellite should be linked to link and not hub.

I tried to add the line number as hashed key in link, but it seems that it requires always hub.

image

AB#5338

AB#5349

DVAlexHiggs commented 8 months ago

Hi! Thanks for this request. This should already be supported. A Link's src_fk field can be provided whatever column names you like as long as they are in the staging table your are sourcing from. Can i ask if you're getting any errors or if there's any further information to help us understand the issue please? Thanks

koillinengit commented 8 months ago

Yes, Alex you are right. The problem for us was that we tried to include the dependent child field directly into link's src_pk beside the src_fk. That was not the correct way of course. It should go into the hashed view and there link's business key list in hashed_columns section. You can close this one.

koillinengit commented 8 months ago

It could be beneficial if in the automateDV documentation would be some word about this dependent child structure handling because it is quite essential feature to get the DataVault model to work together with different source systems.

phillem15 commented 8 months ago

How could we achieve child dependent keys for link sats with the satellite macros? Would we use multi-active satellite? I don't see the src_fk for the sat macro, for example.

DVAlexHiggs commented 6 months ago

How could we achieve child dependent keys for link sats with the satellite macros? Would we use multi-active satellite? I don't see the src_fk for the sat macro, for example.

Satellite HK must be the same as the parent HK, so you would make the src_pk in the Sat equal to the src_pk in the parent link. You can then join between these to get the keys as needed. Additionally to avoid a join downstream you can use the src_extra_columns parameter to add any additional Link columns (including the fks) as needed. I'd be weary of doing this though.

Keeping this issue open as we want to add a section to the docs on how to represent dependent child relationships and will keep it open until we've got that in the docs. Thanks