As sys admin, I want to be able to reparse datafile sets

[andrew-jameson]

Description: Implement ability to re-parse/refresh already-submitted data files and have a lifecycle for this generated data. This is a solution for #2870 and the re-indexing issues.

Acceptance Criteria:

• [ ] Manual step documented for triggering task to re-parse all old data files
• [ ] System Administrator can delete/re-parse a set of data files
• [ ] Admin should be able to update by quarter for all data types.
• [ ] Testing Checklist has been run and all tests pass
• [ ] README is updated, if necessary

Tasks:

• [ ] Build filter for data files submitted before certain date aka prior submission period
• [ ] Build out manual task trigger for orchestrating lifecycle, deployments, and re-parsing
• [ ] Run Testing Checklist and confirm all tests pass

Notes:

• @ADPennington has mentioned that prod data files from fy22 would be ideal candidates because those are not only un-parsed (submitted before parsing was in prod), but also may not adhere to current validation logic standards/guidance.

Supporting Documentation:

• Please see #2870
• https://elasticsearch-dsl.readthedocs.io/en/latest/search_dsl.html#delete-by-query

Open Questions:

• Do we want filtering/granularity for a given quarter, program type (tanf vs ssp), full year, etc.? since last reporting period? Alex gave guidance to focus on latest quarter in fy24.
• Do we need to implement drops/archives for all postgres data as well?

[andrew-jameson]

• [ ] Build DAC filter on datafiles by quarter
• [ ] Add DAC action for 're-parse'
• [ ] Implement base unit tests
• [ ] Based on list of datafile id's, iterate .delay() calls per id generating new celery tasks -- or do we want by quarter? how does admin input that? shell_plus w/ django cmd?
• [ ] Implement a orchestration func that:
  • [ ] a) cleans up postgres
  • [ ] b) cleans up ES
  • [ ] c) resets DFS
  • [ ] d) re-calls parse_datafile() w/ param(s)
• [ ] Have some check that all datafile ids were handled. how might we handle conflicts if data in db already exists?

[robgendron]

11 points remaining.

[robgendron]

This will be completed in #3004. Andrew's May 15 notes has become basis for #3004. Deemed closed.

[ADPennington]

@lfrohlich we will close this ticket and capture more details in #3004 cc: @robgendron @andrew-jameson