As part of the transformation work, we copy history data for summaries, and three one-to-many lookup tables.
Each of these history tables references the opportunity table. However because they're history records, they may actually be referring to the opportunity history table. If the opportunity itself has been deleted, it will not be found in the opportunity table, but can be found in the opportunity history table and marked as deleted.
We aren't currently pulling in opportunity history, so we can't import these records, and they'll currently error and we'll continuously re-process them. To avoid this, we want to instead recognize that these history records may be orphaned in our system and mark them as processed. However, if we later decide to maintain an opportunity history, we want to be able to import those records, so we want to add a column that lets us know why we skipped importing the record.
Acceptance criteria
[ ] Add a column to our staging tables that we can use to indicate "we skipped processing this for reason X"
[ ] For the history records without a corresponding opportunity (or summary), instead of erroring, mark this column, increment a metric, and log a warning.
[ ] Adjust the logic for when we skip orphaned CFDA records to also populate this new column
Summary
As part of the transformation work, we copy history data for summaries, and three one-to-many lookup tables.
Each of these history tables references the opportunity table. However because they're history records, they may actually be referring to the opportunity history table. If the opportunity itself has been deleted, it will not be found in the opportunity table, but can be found in the opportunity history table and marked as deleted.
We aren't currently pulling in opportunity history, so we can't import these records, and they'll currently error and we'll continuously re-process them. To avoid this, we want to instead recognize that these history records may be orphaned in our system and mark them as processed. However, if we later decide to maintain an opportunity history, we want to be able to import those records, so we want to add a column that lets us know why we skipped importing the record.
Acceptance criteria