Description:
Adds a new check to the elasticsearch_indexer and elasticsearch_indexer_for_spark commands when using the --process-deletes flag. This checks for transactions/awards that have been modified in the past 3 days and that have an action_date before FY2008 If any transactions/awards are found then we attempt to delete those records from Elasticsearch.
This is for a scenario where a transaction/award is create with an action_date on or after 2007-10-01, is then added to Elasticsearch and then the transaction/award's action_date is later changed to be before 2007-10-01. This causes the transaction/award record in Elasticsearch to contain the old data and not be updated. The only solution to this is a full reindex of the affected index.
Technical details:
Adds a new check to the elasticsearch_indexer and elasticsearch_indexer_for_spark commands when using the --process-deletes flag. This checks for transactions/awards that have been modified in the past 3 days and that have an action_date before FY2008 If any transactions/awards are found then we attempt to delete those records from Elasticsearch.
[x] Performance evaluation of affected (API | Script | Download)
[x] Before / After data comparison
Area for explaining above N/A when needed:
2. API documentation updated
No API documentation is affected by this change.
4. Matview impact assessment completed
Matviews are not affected by this change.
5. Frontend impact assessment completed
The frontend is not impacted by this change.
Description: Adds a new check to the
elasticsearch_indexer
andelasticsearch_indexer_for_spark
commands when using the--process-deletes
flag. This checks for transactions/awards that have been modified in the past 3 days and that have anaction_date
before FY2008 If any transactions/awards are found then we attempt to delete those records from Elasticsearch.This is for a scenario where a transaction/award is create with an
action_date
on or after 2007-10-01, is then added to Elasticsearch and then the transaction/award'saction_date
is later changed to be before 2007-10-01. This causes the transaction/award record in Elasticsearch to contain the old data and not be updated. The only solution to this is a full reindex of the affected index.Technical details: Adds a new check to the
elasticsearch_indexer
andelasticsearch_indexer_for_spark
commands when using the--process-deletes
flag. This checks for transactions/awards that have been modified in the past 3 days and that have anaction_date
before FY2008 If any transactions/awards are found then we attempt to delete those records from Elasticsearch.Requirements for PR merge:
Area for explaining above N/A when needed: