In the current codebase, the reconcile_data and _get_mismatch_data functions are tightly integrated with HASH_COLUMN_NAME to identify the mismatch, missing_in_src and missing_in_tgt records; due to this, it can't be reused for any other use cases like Aggregate Data Reconciliation .
Proposed Solution
Need to update these as generic common functions, so that it can be reused for other use cases like Aggregate Data Reconciliation along with Hash Data Reconciliation.
Additional Context
For Aggregate Data Reconciliation, two new functions, join_aggregate_data and _get_mismatch_agg_data, are introduced. Once generic functions are developed, these two functions should be removed and integrated with the common functions.
vijaypavann-db
commented
2 months ago
Is there an existing issue for this?
Category of feature request
Reconcile
Problem statement
In the current codebase, the
reconcile_dataand_get_mismatch_datafunctions are tightly integrated withHASH_COLUMN_NAMEto identify themismatch,missing_in_srcandmissing_in_tgtrecords; due to this, it can't be reused for any other use cases likeAggregate Data Reconciliation.Proposed Solution
Need to update these as generic common functions, so that it can be reused for other use cases like
Aggregate Data Reconciliationalong withHash Data Reconciliation.Additional Context
For
Aggregate Data Reconciliation,two new functions,join_aggregate_dataand_get_mismatch_agg_data, are introduced. Once generic functions are developed, these two functions should be removed and integrated with the common functions.