#3137 (+ #3138) - worker crash/timeout on csv export

[jtimpe]

Summary of Changes

Pull request closes #3137

• moves queryset export to a celery task and stores generated files in s3 (addressing #3138)
• implements queryset.iterator() to aid memory use for large querysets

How to Test

cd tdrs-frontend && docker compose up --build
cd tdrs-backend && docker compose up

1. Open http://localhost:3000/ and sign in. Navigate to the Django Admin Console.
2. Navigate to one of the search_indexes model pages. select any number of models and execute the "Export as csv" action.
3. Note the returned message stating the file location. The file can be retrieved locally by navigating to http://localhost:4566/tdp-datafiles-localstack/{returned path}
4. Inspect the LogEntry objects created in the admin console.

Deliverables

More details on how deliverables herein are assessed included here.

Deliverable 1: Accepted Features

Checklist of ACs:

• [x] Resolve worker crash exception preventing files from being downloaded
• [x] Resolve worker timeout exception preventing complete queryset from being exported
• [x] lfrohlich and/or adpennington confirmed that ACs are met.

Deliverable 2: Tested Code

• Are all areas of code introduced in this PR meaningfully tested?
  • [x] If this PR introduces backend code changes, are they meaningfully tested?
  • [x] If this PR introduces frontend code changes, are they meaningfully tested?
• Are code coverage minimums met?
  • [x] Frontend coverage: [insert coverage %] (see CodeCov Report comment in PR)
  • [x] Backend coverage: [insert coverage %] (see CodeCov Report comment in PR)

Deliverable 3: Properly Styled Code

• [x] Are backend code style checks passing on CircleCI?
• [x] Are frontend code style checks passing on CircleCI?
• [x] Are code maintainability principles being followed?

Deliverable 4: Accessible

• [ ] Does this PR complete the epic?
• [ ] Are links included to any other gov-approved PRs associated with epic?
• [ ] Does PR include documentation for Raft's a11y review?
• [ ] Did automated and manual testing with iamjolly and ttran-hub using Accessibility Insights reveal any errors introduced in this PR?

Deliverable 5: Deployed

• [x] Was the code successfully deployed via automated CircleCI process to development on Cloud.gov?

Deliverable 6: Documented

• [x] Does this PR provide background for why coding decisions were made?
• [ ] If this PR introduces backend code, is that code easy to understand and sufficiently documented, both inline and overall?
• [ ] If this PR introduces frontend code, is that code easy to understand and sufficiently documented, both inline and overall?
• [ ] If this PR introduces dependencies, are their licenses documented?
• [x] Can reviewer explain and take ownership of these elements presented in this code review?

Deliverable 7: Secure

• [x] Does the OWASP Scan pass on CircleCI?
• [ ] Do manual code review and manual testing detect any new security issues?
• [ ] If new issues detected, is investigation and/or remediation plan documented?

Deliverable 8: User Research

Research product(s) clearly articulate(s):

• [ ] the purpose of the research
• [ ] methods used to conduct the research
• [ ] who participated in the research
• [ ] what was tested and how
• [ ] impact of research on TDP
• [ ] (if applicable) final design mockups produced for TDP development

[codecov[bot]]

Codecov Report

Attention: Patch coverage is 28.07018% with 41 lines in your changes missing coverage. Please review.

Project coverage is 91.10%. Comparing base (febf97f) to head (e423b1b). Report is 25 commits behind head on develop.

Files with missing lines	Patch %	Lines
tdrs-backend/tdpservice/search_indexes/tasks.py	25.53%	35 Missing :warning:
...-backend/tdpservice/search_indexes/admin/mixins.py	40.00%	6 Missing :warning:

Additional details and impacted files

[](https://app.codecov.io/gh/raft-tech/TANF-app/pull/3162?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=raft-tech) ```diff @@ Coverage Diff @@ ## develop #3162 +/- ## =========================================== - Coverage 91.55% 91.10% -0.45% =========================================== Files 291 292 +1 Lines 8096 8153 +57 Branches 760 763 +3 =========================================== + Hits 7412 7428 +16 - Misses 601 642 +41 Partials 83 83 ``` | [Flag](https://app.codecov.io/gh/raft-tech/TANF-app/pull/3162/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=raft-tech) | Coverage Δ | | |---|---|---| | [dev-backend](https://app.codecov.io/gh/raft-tech/TANF-app/pull/3162/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=raft-tech) | `90.50% <28.07%> (-0.51%)` | :arrow_down: | | [dev-frontend](https://app.codecov.io/gh/raft-tech/TANF-app/pull/3162/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=raft-tech) | `95.34% <ø> (ø)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=raft-tech#carryforward-flags-in-the-pull-request-comment) to find out more. | [Files with missing lines](https://app.codecov.io/gh/raft-tech/TANF-app/pull/3162?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=raft-tech) | Coverage Δ | | |---|---|---| | [...-backend/tdpservice/search\_indexes/admin/mixins.py](https://app.codecov.io/gh/raft-tech/TANF-app/pull/3162?src=pr&el=tree&filepath=tdrs-backend%2Ftdpservice%2Fsearch_indexes%2Fadmin%2Fmixins.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=raft-tech#diff-dGRycy1iYWNrZW5kL3RkcHNlcnZpY2Uvc2VhcmNoX2luZGV4ZXMvYWRtaW4vbWl4aW5zLnB5) | `62.16% <40.00%> (+16.26%)` | :arrow_up: | | [tdrs-backend/tdpservice/search\_indexes/tasks.py](https://app.codecov.io/gh/raft-tech/TANF-app/pull/3162?src=pr&el=tree&filepath=tdrs-backend%2Ftdpservice%2Fsearch_indexes%2Ftasks.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=raft-tech#diff-dGRycy1iYWNrZW5kL3RkcHNlcnZpY2Uvc2VhcmNoX2luZGV4ZXMvdGFza3MucHk=) | `25.92% <25.53%> (ø)` | | ------ [Continue to review full report in Codecov by Sentry](https://app.codecov.io/gh/raft-tech/TANF-app/pull/3162?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=raft-tech). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=raft-tech) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://app.codecov.io/gh/raft-tech/TANF-app/pull/3162?dropdown=coverage&src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=raft-tech). Last update [db12f08...e423b1b](https://app.codecov.io/gh/raft-tech/TANF-app/pull/3162?dropdown=coverage&src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=raft-tech). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=raft-tech).

[ADPennington]

when you get some time, can you fill in the ACs for this @jtimpe ?

[jtimpe]

when you get some time, can you fill in the ACs for this @jtimpe ?

@ADPennington apologies - the bug tickets didn't have any AC so i added some generic ones. the implementation for each is noted in the description

[ADPennington]

when you get some time, can you fill in the ACs for this @jtimpe ?

@ADPennington apologies - the bug tickets didn't have any AC so i added some generic ones. the implementation for each is noted in the description

@jtimpe no trouble; mostly just trying to get a better understanding of any possible constraints when testing. will keep you posted. thank you!

[ADPennington]

first round of testing results below cc: @jtimpe :

• generated 3 exports with 3 unique s3 file paths (evidence from logentries below)

• only the most recent one comes up in the list of S3 contents:

  apennington@HHSLBDSWL73 MINGW64 /
  $ aws s3 ls  s3://<bucket-name> --recursive  --profile TDP_qasp_datafilesS3_20240228 | grep -e  'exports/*'
  2024-09-18 15:28:11    3035675 exports/search_indexes.ssp_m1_FY2023Q4_180924-19-25-25.csv.gz

I'll continue testing tmor.

[ADPennington]

2nd round of testing results cc: @jtimpe

• retried the export of TANF_T2 records attempted yesterday (i.e. Select all 1,091,996 TANF T2s)
• logentries re-affirm that the export began but did not observe another entry for export completing.
• backend logs show that worker was lost, which makes sense given the very large number of records:

  2024-09-19T08:43:38.34-0400 [APP/PROC/WEB/0] ERR [2024-09-19 12:43:38 +0000] [567] [DEBUG] Closing connection.
  2024-09-19T08:43:41.21-0400 [APP/PROC/WEB/0] ERR [2024-09-19 12:43:41,219: ERROR/MainProcess] Process 'ForkPoolWorker-10' pid:625 exited with 'signal 9 (SIGKILL)'
  2024-09-19T08:43:41.23-0400 [APP/PROC/WEB/0] ERR [2024-09-19 12:43:41,231: ERROR/MainProcess] Task handler raised error: WorkerLostError('Worker exited prematurely: signal 9 (SIGKILL) Job: 9.')
  2024-09-19T08:43:41.23-0400 [APP/PROC/WEB/0] ERR Traceback (most recent call last):
  2024-09-19T08:43:41.23-0400 [APP/PROC/WEB/0] ERR File "/home/vcap/deps/1/python/lib/python3.10/site-packages/billiard/pool.py", line 1265, in mark_as_worker_lost
  2024-09-19T08:43:41.23-0400 [APP/PROC/WEB/0] ERR raise WorkerLostError(
  2024-09-19T08:43:41.23-0400 [APP/PROC/WEB/0] ERR billiard.exceptions.WorkerLostError: Worker exited prematurely: signal 9 (SIGKILL) Job: 9.

will continue testing.

[jtimpe]

top or htop while ssh'd into machine during csv export (no other concurrent operations) - if mem climbs, leak somewhere - celery DEBUG is on