search

Datavault-UK / automate-dv

A free to use dbt package for creating and loading Data Vault 2.0 compliant Data Warehouses (powered by dbt, an open source data engineering tool, registered trademark of dbt Labs)
https://www.automate-dv.com
Apache License 2.0
478 stars 114 forks source link

Multiple sources for a Satellite #214

Closed ssuri39 closed 8 months ago

ssuri39 commented 8 months ago

Is it possible to create a satellite that pulls data from multiple sources like we have the option in Hub. Example : I have a hub that is getting location data from multiple sources which is possible. Then I also need to create a satellite that captures information about that location like whether its active etc. This cannot be done by specifying the multiple sources like we can do in hub. please suggest here any alternate solution. Multiple satellites may not be feasible as all the sources capture same set of parameters, only thing is that we are getting location information from differerent sources

Thanks!

DVAlexHiggs commented 8 months ago

This is counter to the Data Vault approach and standards and it is not something we will support, now or in the future.

The answer is to create multiple satellites; the purpose of a Satellite is to contain information for a Hub from a single source system, by definition. If you have different sources of data for the hub, then you create a new satellite. Not all Satellites on one Hub need to contain the same fields (payload) and the intent is for all Satellites on a Hub to give us the full picture when combined and queried together downstream (either using a PIT table or 'manually' joining).

If you are unsure about this, I recommend you do further reading around the subject or you are welcome to reach out to us for more in-depth discussion.

Please see this previously closed issue with the same request, for more explanation #202

DatavaultUK commented 8 months ago

You can conform or merge satellites using a business rule to create a business satellite. For example - to generate a golden record.

That way you have a full audit trail.

Neil