2795 s1 duplicates

[elipe17]

Summary of Changes

• Added new class that manages duplicate detection while case cat4 is occurring
• Updated cat4 validation to leverage duplicate detector
• Updated parse logic to not create records in a case if it has any cat4 errors Pull request closes #2795

CPU and Memory Analysis

• Used Docker to track the web container stats during parsing of the largest file we have in the repo (~50MB) on the develop branch and this branch.
• The overall results can be found in super_big_file_results.txt. The data listed in that text file can be found in the accompanying files: develop_super_big_file.txt and dup_super_big_file.txt.
• The data was generated with: track_docker_stats.sh.txt (GitHub won't let you upload a pure shell file). super_big_file_results.txt dup_super_big_file.txt develop_super_big_file.txt track_docker_stats.sh.txt

How to Test

List the steps to test the PR These steps are generic, please adjust as necessary.

cd tdrs-frontend && docker-compose up
cd tdrs-backend && docker-compose up

1. Open http://localhost:3000/ and sign in.
2. Create a file for TANF section 1 - 4 that has one or more duplicate or partial duplicate records.
3. Submit file
4. Verify no records tied to that case are created and the error report reflects cat4 duplicate errors

Deliverables

More details on how deliverables herein are assessed included here.

Deliverable 1: Accepted Features

Checklist of ACs:

• [x] In section 1 files, exact and partial duplicate records are detected, not stored in db, and error message included in the feedback report.
• [x] In section 2 files, exact and partial duplicate records are detected, not stored in db, and error message included in the feedback report.
• [x] section 3 files, partial duplicate T6 records are detected, not stored in db, and error message included in the feedback report.
• [x] section 4 files, partial duplicate T7 records are detected, not stored in db, and error message included in the feedback report.
• [x] duplicate detection logic is applied consistently to TANF, SSP, and Tribal TANF files.
• [x] cat 4 records not stored in db
• [ ] Testing Checklist has been run and all tests pass
• [x] README is updated, if necessary

Deliverable 2: Tested Code

• Are all areas of code introduced in this PR meaningfully tested?
  • [x] If this PR introduces backend code changes, are they meaningfully tested?
  • [ ] If this PR introduces frontend code changes, are they meaningfully tested?
• Are code coverage minimums met?
  • [x] Frontend coverage: [insert coverage %] (see CodeCov Report comment in PR)
  • [x] Backend coverage: [insert coverage %] (see CodeCov Report comment in PR)

Deliverable 3: Properly Styled Code

• [x] Are backend code style checks passing on CircleCI?
• [x] Are frontend code style checks passing on CircleCI?
• [x] Are code maintainability principles being followed?

Deliverable 4: Accessible

• [ ] Does this PR complete the epic?
• [ ] Are links included to any other gov-approved PRs associated with epic?
• [ ] Does PR include documentation for Raft's a11y review?
• [ ] Did automated and manual testing with iamjolly and ttran-hub using Accessibility Insights reveal any errors introduced in this PR?

Deliverable 5: Deployed

• [x] Was the code successfully deployed via automated CircleCI process to development on Cloud.gov?

Deliverable 6: Documented

• [x] Does this PR provide background for why coding decisions were made?
• [ ] If this PR introduces backend code, is that code easy to understand and sufficiently documented, both inline and overall?
• [ ] If this PR introduces frontend code, is that code easy to understand and sufficiently documented, both inline and overall?
• [ ] If this PR introduces dependencies, are their licenses documented?
• [x] Can reviewer explain and take ownership of these elements presented in this code review?

Deliverable 7: Secure

• [x] Does the OWASP Scan pass on CircleCI?
• [ ] Do manual code review and manual testing detect any new security issues?
• [ ] If new issues detected, is investigation and/or remediation plan documented?

[codecov[bot]]

Codecov Report

Attention: Patch coverage is 92.75362% with 30 lines in your changes missing coverage. Please review.

Project coverage is 92.97%. Comparing base (14ca954) to head (c503e1d). Report is 1 commits behind head on develop.

Additional details and impacted files

[](https://app.codecov.io/gh/raft-tech/TANF-app/pull/2956?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=raft-tech) ```diff @@ Coverage Diff @@ ## develop #2956 +/- ## =========================================== - Coverage 93.10% 92.97% -0.13% =========================================== Files 275 276 +1 Lines 7047 7316 +269 Branches 601 646 +45 =========================================== + Hits 6561 6802 +241 - Misses 392 413 +21 - Partials 94 101 +7 ``` | [Flag](https://app.codecov.io/gh/raft-tech/TANF-app/pull/2956/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=raft-tech) | Coverage Δ | | |---|---|---| | [dev-backend](https://app.codecov.io/gh/raft-tech/TANF-app/pull/2956/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=raft-tech) | `93.03% <92.75%> (-0.16%)` | :arrow_down: | | [dev-frontend](https://app.codecov.io/gh/raft-tech/TANF-app/pull/2956/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=raft-tech) | `92.60% <ø> (ø)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=raft-tech#carryforward-flags-in-the-pull-request-comment) to find out more. | [Files](https://app.codecov.io/gh/raft-tech/TANF-app/pull/2956?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=raft-tech) | Coverage Δ | | |---|---|---| | [tdrs-backend/tdpservice/parsers/aggregates.py](https://app.codecov.io/gh/raft-tech/TANF-app/pull/2956?src=pr&el=tree&filepath=tdrs-backend%2Ftdpservice%2Fparsers%2Faggregates.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=raft-tech#diff-dGRycy1iYWNrZW5kL3RkcHNlcnZpY2UvcGFyc2Vycy9hZ2dyZWdhdGVzLnB5) | `98.00% <100.00%> (+0.08%)` | :arrow_up: | | [tdrs-backend/tdpservice/parsers/models.py](https://app.codecov.io/gh/raft-tech/TANF-app/pull/2956?src=pr&el=tree&filepath=tdrs-backend%2Ftdpservice%2Fparsers%2Fmodels.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=raft-tech#diff-dGRycy1iYWNrZW5kL3RkcHNlcnZpY2UvcGFyc2Vycy9tb2RlbHMucHk=) | `92.42% <100.00%> (+0.11%)` | :arrow_up: | | [tdrs-backend/tdpservice/parsers/row\_schema.py](https://app.codecov.io/gh/raft-tech/TANF-app/pull/2956?src=pr&el=tree&filepath=tdrs-backend%2Ftdpservice%2Fparsers%2Frow_schema.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=raft-tech#diff-dGRycy1iYWNrZW5kL3RkcHNlcnZpY2UvcGFyc2Vycy9yb3dfc2NoZW1hLnB5) | `94.78% <100.00%> (+0.13%)` | :arrow_up: | | [...s-backend/tdpservice/parsers/schema\_defs/ssp/m1.py](https://app.codecov.io/gh/raft-tech/TANF-app/pull/2956?src=pr&el=tree&filepath=tdrs-backend%2Ftdpservice%2Fparsers%2Fschema_defs%2Fssp%2Fm1.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=raft-tech#diff-dGRycy1iYWNrZW5kL3RkcHNlcnZpY2UvcGFyc2Vycy9zY2hlbWFfZGVmcy9zc3AvbTEucHk=) | `100.00% <100.00%> (ø)` | | | [...s-backend/tdpservice/parsers/schema\_defs/ssp/m2.py](https://app.codecov.io/gh/raft-tech/TANF-app/pull/2956?src=pr&el=tree&filepath=tdrs-backend%2Ftdpservice%2Fparsers%2Fschema_defs%2Fssp%2Fm2.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=raft-tech#diff-dGRycy1iYWNrZW5kL3RkcHNlcnZpY2UvcGFyc2Vycy9zY2hlbWFfZGVmcy9zc3AvbTIucHk=) | `100.00% <100.00%> (ø)` | | | [...s-backend/tdpservice/parsers/schema\_defs/ssp/m3.py](https://app.codecov.io/gh/raft-tech/TANF-app/pull/2956?src=pr&el=tree&filepath=tdrs-backend%2Ftdpservice%2Fparsers%2Fschema_defs%2Fssp%2Fm3.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=raft-tech#diff-dGRycy1iYWNrZW5kL3RkcHNlcnZpY2UvcGFyc2Vycy9zY2hlbWFfZGVmcy9zc3AvbTMucHk=) | `90.90% <100.00%> (-9.10%)` | :arrow_down: | | [...s-backend/tdpservice/parsers/schema\_defs/ssp/m4.py](https://app.codecov.io/gh/raft-tech/TANF-app/pull/2956?src=pr&el=tree&filepath=tdrs-backend%2Ftdpservice%2Fparsers%2Fschema_defs%2Fssp%2Fm4.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=raft-tech#diff-dGRycy1iYWNrZW5kL3RkcHNlcnZpY2UvcGFyc2Vycy9zY2hlbWFfZGVmcy9zc3AvbTQucHk=) | `100.00% <100.00%> (ø)` | | | [...s-backend/tdpservice/parsers/schema\_defs/ssp/m5.py](https://app.codecov.io/gh/raft-tech/TANF-app/pull/2956?src=pr&el=tree&filepath=tdrs-backend%2Ftdpservice%2Fparsers%2Fschema_defs%2Fssp%2Fm5.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=raft-tech#diff-dGRycy1iYWNrZW5kL3RkcHNlcnZpY2UvcGFyc2Vycy9zY2hlbWFfZGVmcy9zc3AvbTUucHk=) | `100.00% <100.00%> (ø)` | | | [...-backend/tdpservice/parsers/schema\_defs/tanf/t1.py](https://app.codecov.io/gh/raft-tech/TANF-app/pull/2956?src=pr&el=tree&filepath=tdrs-backend%2Ftdpservice%2Fparsers%2Fschema_defs%2Ftanf%2Ft1.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=raft-tech#diff-dGRycy1iYWNrZW5kL3RkcHNlcnZpY2UvcGFyc2Vycy9zY2hlbWFfZGVmcy90YW5mL3QxLnB5) | `100.00% <100.00%> (ø)` | | | [...-backend/tdpservice/parsers/schema\_defs/tanf/t2.py](https://app.codecov.io/gh/raft-tech/TANF-app/pull/2956?src=pr&el=tree&filepath=tdrs-backend%2Ftdpservice%2Fparsers%2Fschema_defs%2Ftanf%2Ft2.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=raft-tech#diff-dGRycy1iYWNrZW5kL3RkcHNlcnZpY2UvcGFyc2Vycy9zY2hlbWFfZGVmcy90YW5mL3QyLnB5) | `100.00% <100.00%> (ø)` | | | ... and [14 more](https://app.codecov.io/gh/raft-tech/TANF-app/pull/2956?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=raft-tech) | | ... and [1 file with indirect coverage changes](https://app.codecov.io/gh/raft-tech/TANF-app/pull/2956/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=raft-tech) ------ [Continue to review full report in Codecov by Sentry](https://app.codecov.io/gh/raft-tech/TANF-app/pull/2956?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=raft-tech). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=raft-tech) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://app.codecov.io/gh/raft-tech/TANF-app/pull/2956?dropdown=coverage&src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=raft-tech). Last update [4316a8f...c503e1d](https://app.codecov.io/gh/raft-tech/TANF-app/pull/2956?dropdown=coverage&src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=raft-tech). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=raft-tech).

[raftmsohani]

I did a test with ssp_section1_datafile.txtfile, and added another duplicate trying to mimic a user adding a line three times by mistake and the dup errors were handled correctly.

[jtimpe]

One last gut check @elipe17 - 36k lines were removed from tdrs-backend/tdpservice/parsers/test/data/ADS.E2J.NDM1.TS53_fake.rollback.txt, was that intentional?

[elipe17]

One last gut check @elipe17 - 36k lines were removed from tdrs-backend/tdpservice/parsers/test/data/ADS.E2J.NDM1.TS53_fake.rollback.txt, was that intentional?

@jtimpe, that is an intentional deletion. Since I changed the logic for rollback/record deletion it became very import for the test test_parse_big_s1_file_with_rollback to start running. But it was unnecessary to have as many lines in the file as we did to test the logic. So I removed all lines that were unnecessary to test the cases covered in that test.

[ADPennington]

@elipe17 below are my notes so far from qasp review. I've included test files as well for convenience.

---

• have not yet checked implementation for tribal and SSP. ℹ️
• expected logic is in-place for Section 1 :rocket:
• not seeing exact and part dup logic for Section 2 T5 records :question:
• not seeing dup logic for Section 3 T6 records :question:
• not seeing dup logic for Section 4 T7 records :question:
• Should i expect any records associated with cat 4 errors to not be in the db? or just dup-related records?
  • yes - is it possible to add the line number where the partial duplicate record was found in the error message? (see :grey_question:) - Partial duplicate record detected with record type T1 at line 5. Record is a partial duplicate of the record at line number 2. Duplicated fields causing error: record type, reporting month and year, and case number. :grey_question:

---

test files:

section1 2795_partdups_S1.txt 2795_exactdups_S1.txt

section2 2795_partdups_s2.txt 2795_exactdups_s2.txt

section3 2795_dups_s3.txt section4 2795_dups_s4.txt