Re-parse Command

[jtimpe]

Summary of Changes

• Introduced new command that allows an admin user to re-parse a subset or all of the datafiles in the database.
  • Allows admin to specify whether to re-parse in place or to create new indices
  • Creates DB backup
• Created Django command that parametrizes the DB backup routine Pull request closes #3004

How to Test

cd tdrs-frontend && docker-compose up --build
cd tdrs-backend && docker-compose up --build

1. Open http://localhost:3000/ and sign in.
2. Submit some datafiles for different program types, fiscal years, and fiscal quarters.
3. Open http://localhost:9200/_cat/indices/?pretty&v&s=index and verify the elastic indices match the DAC
4. Execute the clean_and_reparse Django command a myriad of times with different values for the parameters.
   • Between each run of the command verify with the Elastic url above and the DAC that you see what you would expect.

Deliverables

More details on how deliverables herein are assessed included here.

Deliverable 1: Accepted Features

Checklist of ACs:

• [ ] Postgres record counts never change/Postgres always remains source of truth
• [ ] Elastic indices match with parameters passed to re-parse command
• [ ] App can archive/delete live data set
• [ ] Testing Checklist has been run and all tests pass
• [ ] README is updated, if necessary

Deliverable 2: Tested Code

• Are all areas of code introduced in this PR meaningfully tested?
  • [ ] If this PR introduces backend code changes, are they meaningfully tested?
  • [ ] If this PR introduces frontend code changes, are they meaningfully tested?
• Are code coverage minimums met?
  • [ ] Frontend coverage: [insert coverage %] (see CodeCov Report comment in PR)
  • [ ] Backend coverage: [insert coverage %] (see CodeCov Report comment in PR)

Deliverable 3: Properly Styled Code

• [ ] Are backend code style checks passing on CircleCI?
• [ ] Are frontend code style checks passing on CircleCI?
• [ ] Are code maintainability principles being followed?

Deliverable 4: Accessible

• [ ] Does this PR complete the epic?
• [ ] Are links included to any other gov-approved PRs associated with epic?
• [ ] Does PR include documentation for Raft's a11y review?
• [ ] Did automated and manual testing with iamjolly and ttran-hub using Accessibility Insights reveal any errors introduced in this PR?

Deliverable 5: Deployed

• [ ] Was the code successfully deployed via automated CircleCI process to development on Cloud.gov?

Deliverable 6: Documented

• [ ] Does this PR provide background for why coding decisions were made?
• [ ] If this PR introduces backend code, is that code easy to understand and sufficiently documented, both inline and overall?
• [ ] If this PR introduces frontend code, is that code easy to understand and sufficiently documented, both inline and overall?
• [ ] If this PR introduces dependencies, are their licenses documented?
• [ ] Can reviewer explain and take ownership of these elements presented in this code review?

Deliverable 7: Secure

• [ ] Does the OWASP Scan pass on CircleCI?
• [ ] Do manual code review and manual testing detect any new security issues?
• [ ] If new issues detected, is investigation and/or remediation plan documented?

Deliverable 8: User Research

Research product(s) clearly articulate(s):

• [ ] the purpose of the research
• [ ] methods used to conduct the research
• [ ] who participated in the research
• [ ] what was tested and how
• [ ] impact of research on TDP
• [ ] (if applicable) final design mockups produced for TDP development

[codecov[bot]]

Codecov Report

Attention: Patch coverage is 25.00000% with 3 lines in your changes missing coverage. Please review.

Project coverage is 92.97%. Comparing base (b5f8bd7) to head (728d1d0).

Additional details and impacted files

[](https://app.codecov.io/gh/raft-tech/TANF-app/pull/3041?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=raft-tech) ```diff @@ Coverage Diff @@ ## develop #3041 +/- ## ======================================== Coverage 92.97% 92.97% ======================================== Files 276 276 Lines 7316 7317 +1 Branches 646 647 +1 ======================================== + Hits 6802 6803 +1 - Misses 413 414 +1 + Partials 101 100 -1 ``` | [Flag](https://app.codecov.io/gh/raft-tech/TANF-app/pull/3041/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=raft-tech) | Coverage Δ | | |---|---|---| | [dev-backend](https://app.codecov.io/gh/raft-tech/TANF-app/pull/3041/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=raft-tech) | `93.01% <25.00%> (-0.02%)` | :arrow_down: | | [dev-frontend](https://app.codecov.io/gh/raft-tech/TANF-app/pull/3041/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=raft-tech) | `92.70% <ø> (+0.09%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=raft-tech#carryforward-flags-in-the-pull-request-comment) to find out more. | [Files](https://app.codecov.io/gh/raft-tech/TANF-app/pull/3041?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=raft-tech) | Coverage Δ | | |---|---|---| | [tdrs-backend/tdpservice/scheduling/parser\_task.py](https://app.codecov.io/gh/raft-tech/TANF-app/pull/3041?src=pr&el=tree&filepath=tdrs-backend%2Ftdpservice%2Fscheduling%2Fparser_task.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=raft-tech#diff-dGRycy1iYWNrZW5kL3RkcHNlcnZpY2Uvc2NoZWR1bGluZy9wYXJzZXJfdGFzay5weQ==) | `50.00% <25.00%> (-2.00%)` | :arrow_down: | ... and [1 file with indirect coverage changes](https://app.codecov.io/gh/raft-tech/TANF-app/pull/3041/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=raft-tech) ------ [Continue to review full report in Codecov by Sentry](https://app.codecov.io/gh/raft-tech/TANF-app/pull/3041?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=raft-tech). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=raft-tech) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://app.codecov.io/gh/raft-tech/TANF-app/pull/3041?dropdown=coverage&src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=raft-tech). Last update [b5f8bd7...728d1d0](https://app.codecov.io/gh/raft-tech/TANF-app/pull/3041?dropdown=coverage&src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=raft-tech). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=raft-tech).

[andrew-jameson]

Step 4 in the 'How to Test' section needs a little more verbosity and usage examples for QASP unless a markdown document is planned.