search

ScalefreeCOM / datavault4dbt

Scalefree's dbt package for a Data Vault 2.0 implementation congruent to the original Data Vault 2.0 definition by Dan Linstedt including the Staging Area, DV2.0 main entities, PITs and Snapshot Tables.
https://www.scalefree.com/
Apache License 2.0
134 stars 25 forks source link

Satellite v0 isn't updating with new ldts via SYSDATE in Redshift #173

Open tankim opened 4 months ago

tankim commented 4 months ago

Hey all. I'm realized that satellite v0 isn't updating with new ldts via SYSDATE in Redshift, so while appending rows to satellite v1, it doesn't add new ldts(current system timestamp) as a new row.

Let's say I'm trying to execute dbt run twice at 2024-05-05 00:00:00.000000, 2024-05-05 00:10:00.000000 The result of table that I'm expected at first execution is as below.

# Redshift Table: sat_wallet_v0
hk_wallet hd_wallet ldts ...
1 1 2024-05-05 00:00:00.000000
# Redshift Table: sat_wallet_v1
1 1 2024-05-05 00:00:00.000000

This is the result that I'm expected but it doesn't. Please check real and expected as below.

# Redshift Table: sat_wallet_v0
hk_wallet hd_wallet ldts ...
1 1 2024-05-05 00:00:00.000000
# Redshift Table: sat_wallet_v1
1 1 2024-05-05 00:00:00.000000
1 1 2024-05-05 00:00:00.000000 << real
1 1 2024-05-05 00:10:00.000000 << expected
tkirschke commented 1 month ago

Hi @tankim ,

Sorry for the late reply!

In your described scenario, there was no change in the second row, the hashdiff is the same. Therefore, no new row is loaded into the Satellite v0.

This is the expected Data Vault 2.0 behavior, let me know if I understood something wrong.

Best regards Tim