3065 -Sequential Reparse

[elipe17]

Summary of Changes

• Added sequential execution logic
• Added admin updates to better display file to reparse associations Pull request closes #3065

Screenshots of what the admin console will look like:

How to Test

List the steps to test the PR These steps are generic, please adjust as necessary.

cd tdrs-frontend && docker-compose up
cd tdrs-backend && docker-compose up

1. Open http://localhost:3000/ and sign in
2. Submit some datafiles
3. Reparse some datafiles
4. Ensure you cant execute reparse in parallel
5. Ensure DAC reflects what files have been reparsed and links work

Deliverables

More details on how deliverables herein are assessed included here.

Deliverable 1: Accepted Features

Checklist of ACs:

• [ ] Implement datetime logic if not already done
• [ ] Update clean_and_reparse to leverage that and print information messages if it cannot run
• [ ] Add integration testing for parallel clean_and_reparse commands
• [ ] Run Testing Checklist and confirm all tests pass

Deliverable 2: Tested Code

• Are all areas of code introduced in this PR meaningfully tested?
  • [ ] If this PR introduces backend code changes, are they meaningfully tested?
  • [ ] If this PR introduces frontend code changes, are they meaningfully tested?
• Are code coverage minimums met?
  • [ ] Frontend coverage: [insert coverage %] (see CodeCov Report comment in PR)
  • [ ] Backend coverage: [insert coverage %] (see CodeCov Report comment in PR)

Deliverable 3: Properly Styled Code

• [ ] Are backend code style checks passing on CircleCI?
• [ ] Are frontend code style checks passing on CircleCI?
• [ ] Are code maintainability principles being followed?

Deliverable 4: Accessible

• [ ] Does this PR complete the epic?
• [ ] Are links included to any other gov-approved PRs associated with epic?
• [ ] Does PR include documentation for Raft's a11y review?
• [ ] Did automated and manual testing with iamjolly and ttran-hub using Accessibility Insights reveal any errors introduced in this PR?

Deliverable 5: Deployed

• [ ] Was the code successfully deployed via automated CircleCI process to development on Cloud.gov?

Deliverable 6: Documented

• [ ] Does this PR provide background for why coding decisions were made?
• [ ] If this PR introduces backend code, is that code easy to understand and sufficiently documented, both inline and overall?
• [ ] If this PR introduces frontend code, is that code easy to understand and sufficiently documented, both inline and overall?
• [ ] If this PR introduces dependencies, are their licenses documented?
• [ ] Can reviewer explain and take ownership of these elements presented in this code review?

Deliverable 7: Secure

• [ ] Does the OWASP Scan pass on CircleCI?
• [ ] Do manual code review and manual testing detect any new security issues?
• [ ] If new issues detected, is investigation and/or remediation plan documented?

Deliverable 8: User Research

Research product(s) clearly articulate(s):

• [ ] the purpose of the research
• [ ] methods used to conduct the research
• [ ] who participated in the research
• [ ] what was tested and how
• [ ] impact of research on TDP
• [ ] (if applicable) final design mockups produced for TDP development

[codecov[bot]]

Codecov Report

Attention: Patch coverage is 85.71429% with 3 lines in your changes missing coverage. Please review.

Project coverage is 91.15%. Comparing base (1ea2d02) to head (c83ee6b). Report is 22 commits behind head on 3064-reparse-meta.

Files	Patch %	Lines
tdrs-backend/tdpservice/parsers/parse.py	81.81%	2 Missing :warning:
tdrs-backend/tdpservice/data_files/admin.py	85.71%	1 Missing :warning:

Additional details and impacted files

[](https://app.codecov.io/gh/raft-tech/TANF-app/pull/3136?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=raft-tech) ```diff @@ Coverage Diff @@ ## 3064-reparse-meta #3136 +/- ## ===================================================== - Coverage 91.17% 91.15% -0.02% ===================================================== Files 283 283 Lines 7694 7713 +19 Branches 702 702 ===================================================== + Hits 7015 7031 +16 - Misses 575 578 +3 Partials 104 104 ``` | [Flag](https://app.codecov.io/gh/raft-tech/TANF-app/pull/3136/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=raft-tech) | Coverage Δ | | |---|---|---| | [dev-backend](https://app.codecov.io/gh/raft-tech/TANF-app/pull/3136/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=raft-tech) | `90.94% <85.71%> (-0.02%)` | :arrow_down: | | [dev-frontend](https://app.codecov.io/gh/raft-tech/TANF-app/pull/3136/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=raft-tech) | `92.60% <ø> (ø)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=raft-tech#carryforward-flags-in-the-pull-request-comment) to find out more. | [Files](https://app.codecov.io/gh/raft-tech/TANF-app/pull/3136?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=raft-tech) | Coverage Δ | | |---|---|---| | [...nd/tdpservice/search\_indexes/admin/reparse\_meta.py](https://app.codecov.io/gh/raft-tech/TANF-app/pull/3136?src=pr&el=tree&filepath=tdrs-backend%2Ftdpservice%2Fsearch_indexes%2Fadmin%2Freparse_meta.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=raft-tech#diff-dGRycy1iYWNrZW5kL3RkcHNlcnZpY2Uvc2VhcmNoX2luZGV4ZXMvYWRtaW4vcmVwYXJzZV9tZXRhLnB5) | `100.00% <100.00%> (ø)` | | | [tdrs-backend/tdpservice/settings/common.py](https://app.codecov.io/gh/raft-tech/TANF-app/pull/3136?src=pr&el=tree&filepath=tdrs-backend%2Ftdpservice%2Fsettings%2Fcommon.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=raft-tech#diff-dGRycy1iYWNrZW5kL3RkcHNlcnZpY2Uvc2V0dGluZ3MvY29tbW9uLnB5) | `99.31% <100.00%> (+<0.01%)` | :arrow_up: | | [tdrs-backend/tdpservice/data\_files/admin.py](https://app.codecov.io/gh/raft-tech/TANF-app/pull/3136?src=pr&el=tree&filepath=tdrs-backend%2Ftdpservice%2Fdata_files%2Fadmin.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=raft-tech#diff-dGRycy1iYWNrZW5kL3RkcHNlcnZpY2UvZGF0YV9maWxlcy9hZG1pbi5weQ==) | `81.81% <85.71%> (+0.73%)` | :arrow_up: | | [tdrs-backend/tdpservice/parsers/parse.py](https://app.codecov.io/gh/raft-tech/TANF-app/pull/3136?src=pr&el=tree&filepath=tdrs-backend%2Ftdpservice%2Fparsers%2Fparse.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=raft-tech#diff-dGRycy1iYWNrZW5kL3RkcHNlcnZpY2UvcGFyc2Vycy9wYXJzZS5weQ==) | `83.72% <81.81%> (-0.19%)` | :arrow_down: | ------ [Continue to review full report in Codecov by Sentry](https://app.codecov.io/gh/raft-tech/TANF-app/pull/3136?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=raft-tech). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=raft-tech) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://app.codecov.io/gh/raft-tech/TANF-app/pull/3136?dropdown=coverage&src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=raft-tech). Last update [1ea2d02...c83ee6b](https://app.codecov.io/gh/raft-tech/TANF-app/pull/3136?dropdown=coverage&src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=raft-tech). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=raft-tech).

[raftmsohani]

@elipe17 could you resolve merge conflict

[elipe17]

Confirmed with Alex that it is not useful to see the RPM links in the datafiles table in the DAC.