2990 - stuck files notification

[jtimpe]

Summary of Changes

Pull request closes #2990

How to Test

1. Make sure the SENDGRID_API_KEY env var is set, reach out if you need a valid key
2. Spin up your environment

   cd tdrs-backend && docker compose up
   cd tdrs-frontend && docker compose up --build

3. Sign in and change your user role to OFA System Admin
4. Create some submissions. Using shell_plus, change the created_at to >1hr ago.
5. Reparse some submissions. Using shell_plus, change the timeout_at to some time in the past. Change finished and success to False.
6. Run docker compose exec web python manage.py find_pending_submissions.

Try a lot of combinations. If you can get it to fire in real life stuck-file scenarios, that's a plus.

Example of email

Deliverables

More details on how deliverables herein are assessed included here.

Deliverable 1: Accepted Features

Checklist of ACs:

• [x] Notification occurs if a newly submitted file has been in pending state for 1+ hours
• [x] Notification content matches spec Notification content must include:
  • [x] STT, Program Type, Fiscal Period, Section Name, Submission Date/time
  • [x] Dynamic link that passes data file ID to the django URL string
  • [x] Confirm these email events should be captured in logs
• [x] All sys-admin users receive the notification
• [x] lfrohlich and/or adpennington confirmed that ACs are met.

Deliverable 2: Tested Code

• Are all areas of code introduced in this PR meaningfully tested?
  • [x] If this PR introduces backend code changes, are they meaningfully tested?
  • [x] If this PR introduces frontend code changes, are they meaningfully tested?
• Are code coverage minimums met?
  • [x] Frontend coverage: [insert coverage %] (see CodeCov Report comment in PR)
  • [x] Backend coverage: [insert coverage %] (see CodeCov Report comment in PR)

Deliverable 3: Properly Styled Code

• [x] Are backend code style checks passing on CircleCI?
• [x] Are frontend code style checks passing on CircleCI?
• [x] Are code maintainability principles being followed?

Deliverable 4: Accessible

• [ ] Does this PR complete the epic?
• [ ] Are links included to any other gov-approved PRs associated with epic?
• [ ] Does PR include documentation for Raft's a11y review?
• [ ] Did automated and manual testing with iamjolly and ttran-hub using Accessibility Insights reveal any errors introduced in this PR?

Deliverable 5: Deployed

• [x] Was the code successfully deployed via automated CircleCI process to development on Cloud.gov?

Deliverable 6: Documented

• [x] Does this PR provide background for why coding decisions were made?
• [ ] If this PR introduces backend code, is that code easy to understand and sufficiently documented, both inline and overall?
• [ ] If this PR introduces frontend code, is that code easy to understand and sufficiently documented, both inline and overall?
• [ ] If this PR introduces dependencies, are their licenses documented?
• [x] Can reviewer explain and take ownership of these elements presented in this code review?

Deliverable 7: Secure

• [x] Does the OWASP Scan pass on CircleCI?
• [ ] Do manual code review and manual testing detect any new security issues?
• [ ] If new issues detected, is investigation and/or remediation plan documented?

Deliverable 8: User Research

Research product(s) clearly articulate(s):

• [ ] the purpose of the research
• [ ] methods used to conduct the research
• [ ] who participated in the research
• [ ] what was tested and how
• [ ] impact of research on TDP
• [ ] (if applicable) final design mockups produced for TDP development

[codecov[bot]]

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 92.66%. Comparing base (c15cb7c) to head (1f6756b). Report is 2 commits behind head on develop.

Additional details and impacted files

[](https://app.codecov.io/gh/raft-tech/TANF-app/pull/3195?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=raft-tech) ```diff @@ Coverage Diff @@ ## develop #3195 +/- ## ======================================== Coverage 92.66% 92.66% ======================================== Files 47 47 Lines 1009 1009 Branches 169 169 ======================================== Hits 935 935 Misses 42 42 Partials 32 32 ``` | [Flag](https://app.codecov.io/gh/raft-tech/TANF-app/pull/3195/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=raft-tech) | Coverage Δ | | |---|---|---| | [dev-frontend](https://app.codecov.io/gh/raft-tech/TANF-app/pull/3195/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=raft-tech) | `92.66% <ø> (ø)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=raft-tech#carryforward-flags-in-the-pull-request-comment) to find out more. ------ [Continue to review full report in Codecov by Sentry](https://app.codecov.io/gh/raft-tech/TANF-app/pull/3195?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=raft-tech). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=raft-tech) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://app.codecov.io/gh/raft-tech/TANF-app/pull/3195?dropdown=coverage&src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=raft-tech). Last update [fab247d...1f6756b](https://app.codecov.io/gh/raft-tech/TANF-app/pull/3195?dropdown=coverage&src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=raft-tech). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=raft-tech).

[ADPennington]

@jtimpe some unit tests are failing on this branch since the reparsing work was merged in.

[jtimpe]

@ADPennington should be fixed!

[ADPennington]

@jtimpe @victoriaatraft -- quick question -- was the design mockup here intended for this ticket or a follow-on?

[jtimpe]

@jtimpe @victoriaatraft -- quick question -- was the design mockup here intended for this ticket or a follow-on?

@ADPennington it was, but per this comment i modified the template to display a list of stuck files rather than a singular.

[ADPennington]

@jtimpe @victoriaatraft -- quick question -- was the design mockup here intended for this ticket or a follow-on?

@ADPennington it was, but per this comment i modified the template to display a list of stuck files rather than a singular.

ahh thank you! Appreciate the reference @jtimpe

[ADPennington]

testing update:

• received email notification with list of pending files.
• ran re-parsing command in qasp environment. no more pending files, so no email notification
• successfully triggered pending status via resubmitting large files in sequence. will re-deploy with new subject line tonight.