These are the two functions that we use to interact with and update the historical flagged data.
We need to verify that when we update the historical data it is being updated properly. There needs to be a check to make sure that we are not duplicating historical data. This can be as simple as removing duplicates at the end of the update join, but we still need to make sure that we are not keeping duplicate DTs with different data. This situation can occur because the data will go through the flagging process twice. The first time when it is first incoming, and the second time when it is used to generate statistics.
These are the two functions that we use to interact with and update the historical flagged data.
We need to verify that when we update the historical data it is being updated properly. There needs to be a check to make sure that we are not duplicating historical data. This can be as simple as removing duplicates at the end of the update join, but we still need to make sure that we are not keeping duplicate DTs with different data. This situation can occur because the data will go through the flagging process twice. The first time when it is first incoming, and the second time when it is used to generate statistics.