Open nnikolov84 opened 5 months ago
if your source table had unique records and it updated the record with new hash, will it be still an issue?
if your source table had unique records and it updated the record with new hash, will it be still an issue?
I'm not sure I understand what you mean. The data coming from the source is unique, but off course it is changing from delivery to delivery. This being SCD, it should show how a record changed in time, but it fails to do so, if a value was already delivered in the past. *see the table my first comment.
In addition to your input data you need to have an incremental model with customer_id
as unique key. And this new incremental model should be the source to your scd.
@nnikolov84 is correct. Take this example That's the source table.
And this is the output after calling scd.
As you can see, the new row is not shown in the output table because the hash value from the last row is exactly the same as the hash from the second row.
When using hash value comparison, new record is not loaded, if the same hash value exist in older record.
The third value in the table will not be processed. This is happening, because when generating "ids_to_update", the historical records are not filtered.
In order for the process to work properly, only most recent record per KEY should be compared to the input data. Possible solution is: