search

Datavault-UK / automate-dv

A free to use dbt package for creating and loading Data Vault 2.0 compliant Data Warehouses (powered by dbt, an open source data engineering tool, registered trademark of dbt Labs)
https://www.automate-dv.com
Apache License 2.0
492 stars 121 forks source link

Built-in support for configuring NULL business key handling #148

Closed DVTimWilson closed 2 years ago

DVTimWilson commented 2 years ago

Built-in support for configuring NULL business key handling

More details here

Documentation PR47

DVTimWilson commented 2 years ago

@DVTimWilson @DVSiobhanStrange Great work!! Only gaps I can see in the testing are:

  • We should have one or two scenarios where SHA256 hashing is enabled
  • We should have a few scenarios where we load the -1, -2 records into Hubs, Links, and Satellites and make sure they are handled correctly (i.e. are actually loaded rather than ignored)
  • We should have a few scenarios where the replacement values are configured differently from the default (-3 instead of -1 on required keys for example) It would be good to have one of these scenarios be the main case we allowed this to be configurable for: Actually having a column which has the value -1

After these are complete I am happy to merge this PR.

Hi @DVAlexHiggs Alex, all good ideas, will do.

DVTimWilson commented 2 years ago

@DVTimWilson @DVSiobhanStrange Great work!! Only gaps I can see in the testing are:

  • We should have one or two scenarios where SHA256 hashing is enabled
  • We should have a few scenarios where we load the -1, -2 records into Hubs, Links, and Satellites and make sure they are handled correctly (i.e. are actually loaded rather than ignored)
  • We should have a few scenarios where the replacement values are configured differently from the default (-3 instead of -1 on required keys for example) It would be good to have one of these scenarios be the main case we allowed this to be configurable for: Actually having a column which has the value -1

After these are complete I am happy to merge this PR.

Hi @DVAlexHiggs Alex, all good ideas, will do.

I have pushed test scenarios for the first and third points. I am still working on test scenarios for the second point.