DAC CSV Export - Githubissues

elipe17 commented 6 months ago

Summary of Changes

Added support for multi-selection list filters
Added new filters for STT code, and fiscal year
Added support for CSV export
Made Records read only in DAC
Updated Newest filter to display the newest records associated to each STT instead of to the latest datafile. Pull request closes #2908

How to Test

cd tdrs-frontend && docker-compose up
cd tdrs-backend && docker-compose up -- build

Open http://localhost:3000/ and sign in as a user with admin privileges and permissions to search indexes.
Submit some files as different STTs
View the records and verify that records for the latest datafile for the respective STTs are shown
Export some or all of the records as CSV
Verify all records are in CSV, STT has its own column, and you can freely navigate DAC while the download is occurring

Deliverables

More details on how deliverables herein are assessed included here.

Deliverable 1: Accepted Features

Checklist of ACs:

[x] DIGIT team/OFA system admin/OFA admin can export parsed data via DAC to CSV
[x] CSV export includes field names and STT code
[x] CSV export only includes latest parsed data for any/all STTs
[x] anyone with access to search_indexes cannot edit/delete parsed data via DAC
[x] filtering capability built to abovementioned specs
- in-place but needs refinement
[x] Testing Checklist has been run and all tests pass
[x] README is updated, if necessary

Deliverable 2: Tested Code

Are all areas of code introduced in this PR meaningfully tested?
- [x] If this PR introduces backend code changes, are they meaningfully tested?
- [x] If this PR introduces frontend code changes, are they meaningfully tested?
Are code coverage minimums met?
- [x] Frontend coverage: [insert coverage %] (see CodeCov Report comment in PR)
- [x] Backend coverage: [insert coverage %] (see CodeCov Report comment in PR)

Deliverable 3: Properly Styled Code

[x] Are backend code style checks passing on CircleCI?
[x] Are frontend code style checks passing on CircleCI?
[x] Are code maintainability principles being followed?

Deliverable 4: Accessible

[ ] Does this PR complete the epic?
[ ] Are links included to any other gov-approved PRs associated with epic?
[ ] Does PR include documentation for Raft's a11y review?
[ ] Did automated and manual testing with iamjolly and ttran-hub using Accessibility Insights reveal any errors introduced in this PR?

Deliverable 5: Deployed

[x] Was the code successfully deployed via automated CircleCI process to development on Cloud.gov?

Deliverable 6: Documented

[x] Does this PR provide background for why coding decisions were made?
[ ] If this PR introduces backend code, is that code easy to understand and sufficiently documented, both inline and overall?
[ ] If this PR introduces frontend code, is that code easy to understand and sufficiently documented, both inline and overall?
[ ] If this PR introduces dependencies, are their licenses documented?
[x] Can reviewer explain and take ownership of these elements presented in this code review?

Deliverable 7: Secure

[x] Does the OWASP Scan pass on CircleCI?
[ ] Do manual code review and manual testing detect any new security issues?
[ ] If new issues detected, is investigation and/or remediation plan documented?

Deliverable 8: User Research

Research product(s) clearly articulate(s):

[ ] the purpose of the research
[ ] methods used to conduct the research
[ ] who participated in the research
[ ] what was tested and how
[ ] impact of research on TDP
[ ] (if applicable) final design mockups produced for TDP development

codecov[bot] commented 6 months ago

Codecov Report

Attention: Patch coverage is 49.06832% with 82 lines in your changes are missing coverage. Please review.

Project coverage is 92.31%. Comparing base (a662286) to head (ca3c6db). Report is 1 commits behind head on develop.

Additional details and impacted files

[![Impacted file tree graph](https://app.codecov.io/gh/raft-tech/TANF-app/pull/2915/graphs/tree.svg?width=650&height=150&src=pr&token=BA04YXPAL9&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=raft-tech)](https://app.codecov.io/gh/raft-tech/TANF-app/pull/2915?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=raft-tech) ```diff @@ Coverage Diff @@ ## develop #2915 +/- ## =========================================== - Coverage 93.35% 92.31% -1.05% =========================================== Files 270 271 +1 Lines 6323 6442 +119 Branches 535 555 +20 =========================================== + Hits 5903 5947 +44 - Misses 324 399 +75 Partials 96 96 ``` | [Flag](https://app.codecov.io/gh/raft-tech/TANF-app/pull/2915/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=raft-tech) | Coverage Δ | | |---|---|---| | [dev-backend](https://app.codecov.io/gh/raft-tech/TANF-app/pull/2915/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=raft-tech) | `92.25% <49.06%> (-1.24%)` | :arrow_down: | | [dev-frontend](https://app.codecov.io/gh/raft-tech/TANF-app/pull/2915/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=raft-tech) | `92.62% <ø> (ø)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=raft-tech#carryforward-flags-in-the-pull-request-comment) to find out more. | [Files](https://app.codecov.io/gh/raft-tech/TANF-app/pull/2915?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=raft-tech) | Coverage Δ | | |---|---|---| | [...drs-backend/tdpservice/search\_indexes/admin/ssp.py](https://app.codecov.io/gh/raft-tech/TANF-app/pull/2915?src=pr&el=tree&filepath=tdrs-backend%2Ftdpservice%2Fsearch_indexes%2Fadmin%2Fssp.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=raft-tech#diff-dGRycy1iYWNrZW5kL3RkcHNlcnZpY2Uvc2VhcmNoX2luZGV4ZXMvYWRtaW4vc3NwLnB5) | `100.00% <100.00%> (ø)` | | | [...rs-backend/tdpservice/search\_indexes/admin/tanf.py](https://app.codecov.io/gh/raft-tech/TANF-app/pull/2915?src=pr&el=tree&filepath=tdrs-backend%2Ftdpservice%2Fsearch_indexes%2Fadmin%2Ftanf.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=raft-tech#diff-dGRycy1iYWNrZW5kL3RkcHNlcnZpY2Uvc2VhcmNoX2luZGV4ZXMvYWRtaW4vdGFuZi5weQ==) | `100.00% <100.00%> (ø)` | | | [...-backend/tdpservice/search\_indexes/admin/tribal.py](https://app.codecov.io/gh/raft-tech/TANF-app/pull/2915?src=pr&el=tree&filepath=tdrs-backend%2Ftdpservice%2Fsearch_indexes%2Fadmin%2Ftribal.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=raft-tech#diff-dGRycy1iYWNrZW5kL3RkcHNlcnZpY2Uvc2VhcmNoX2luZGV4ZXMvYWRtaW4vdHJpYmFsLnB5) | `100.00% <100.00%> (ø)` | | | [...-backend/tdpservice/search\_indexes/admin/mixins.py](https://app.codecov.io/gh/raft-tech/TANF-app/pull/2915?src=pr&el=tree&filepath=tdrs-backend%2Ftdpservice%2Fsearch_indexes%2Fadmin%2Fmixins.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=raft-tech#diff-dGRycy1iYWNrZW5kL3RkcHNlcnZpY2Uvc2VhcmNoX2luZGV4ZXMvYWRtaW4vbWl4aW5zLnB5) | `47.27% <47.27%> (ø)` | | | [...backend/tdpservice/search\_indexes/admin/filters.py](https://app.codecov.io/gh/raft-tech/TANF-app/pull/2915?src=pr&el=tree&filepath=tdrs-backend%2Ftdpservice%2Fsearch_indexes%2Fadmin%2Ffilters.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=raft-tech#diff-dGRycy1iYWNrZW5kL3RkcHNlcnZpY2Uvc2VhcmNoX2luZGV4ZXMvYWRtaW4vZmlsdGVycy5weQ==) | `32.91% <32.91%> (ø)` | | ------ [Continue to review full report in Codecov by Sentry](https://app.codecov.io/gh/raft-tech/TANF-app/pull/2915?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=raft-tech). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=raft-tech) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://app.codecov.io/gh/raft-tech/TANF-app/pull/2915?dropdown=coverage&src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=raft-tech). Last update [1f6010f...ca3c6db](https://app.codecov.io/gh/raft-tech/TANF-app/pull/2915?dropdown=coverage&src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=raft-tech). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=raft-tech).

elipe17 commented 5 months ago

@ADPennington answering your remaining questions and comments here:

Excel has limit of 1mil records. this will likely impact the export of Section 1, T3 records by fiscal period. Does that limit impact how many records can be exported to csv? It's unlikely we would try to open the csv in excel; rather, the more common scenario is to import the csv into SAS, Python, etc. is it feasible to add a test about exporting >1mil records to csv?

Creating a csv file with more than 1 million rows will not cause problems in terms of file size supported by csv. It can be as big as our memory budget will allow. Excel truncates the file and only show 1M rows to avoid memory issues and user experience issues. I tested this locally with ~400k records and the file size was ~50MB for reference. I don't think we have a feasible way to generate a test that has 1M records as of right now. I know @jtimpe has researched this idea a bit. Perhaps he can comment?

STT filter option is a bit strange. is it feasible to have an STT combobox like we have elsewhere (e.g. user profile) but with the added capability of multiple selections?

It is feasible, but I think that will scope creep this ticket. Perhaps we can write an enhancement ticket to address the format?

Can we add the STT type filter to all search indexes? only tribes submit tribal TANF record types, for example.

The STT filter is enabled on all search indexes.

The fiscal period filter seems to be arbitrarily set relative to today's date, but should be consistent with frontend fiscal period options. I dont think the fiscal filtering options should change relative to today's date.

I updated the year generation algorithm to be a one to one match with the frontend's algorithm

If we have fiscal period filter, we do not also need the RPT_MONTH_YEAR field to filter by.

I removed all the erroneous field filters

raft-tech / TANF-app

DAC CSV Export #2915