Kibana Dashboard MVP

[elipe17]

Summary of Changes

• Added support for automated Kibana object upload
• Created four dashboards corresponding to the Families section in this document.
• Updated Terraform to specify the Elastic version and allow for a long timeout because on average it takes longer than the default timeout of 15 minutes for cloud.gov/AWS to stand up the instance. Pull request closes #2883

Note: I did not create dashboards/visualizations for every single table in the linked document. If that is what we want, I can do so. Just didn't want to do too much.

How to Test

cd tdrs-frontend && docker-compose up --build
cd tdrs-backend && docker-compose up --build

1. Open http://localhost:3000/ and sign in as a Sys admin.
2. In an incognito window: Open http://localhost:3000/ and sign in as a data analyst with a different STT.
3. For Both STTs submit TANF S1 and S3 data for FY 2021 Quarter 1. For Section 1, use ADS.E2J.FTP1.TS06 and for Section 3, use the attached file below. Note, this file is a modified version of the original: ADS.E2J.FTP3.TS06. ADS.E2J.FTP3.TS06.txt
4. [Local only] Make sure to run source commands.sh and then execute the new command: upload_kibana_objs
5. As the sys admin user, navigate to Kibana->Stack Management->Saved Objects and verify you see have several new saved objects.
6. From there, navigate through the Visualizations and Dashboards. Make sure your time range includes the datetime that you submitted your S1/S3 files or there will be no data to see.

Deliverables

More details on how deliverables herein are assessed included here.

Deliverable 1: Accepted Features

Checklist of ACs:

• [ ] Create necessary queries for reports
• [ ] Create dashboards mirroring: this
• [ ] Objects are automatically loaded into Kibana on deployment

Deliverable 2: Tested Code

• Are all areas of code introduced in this PR meaningfully tested?
  • [ ] If this PR introduces backend code changes, are they meaningfully tested?
  • [ ] If this PR introduces frontend code changes, are they meaningfully tested?
• Are code coverage minimums met?
  • [ ] Frontend coverage: [insert coverage %] (see CodeCov Report comment in PR)
  • [ ] Backend coverage: [insert coverage %] (see CodeCov Report comment in PR)

Deliverable 3: Properly Styled Code

• [ ] Are backend code style checks passing on CircleCI?
• [ ] Are frontend code style checks passing on CircleCI?
• [ ] Are code maintainability principles being followed?

Deliverable 4: Accessible

• [ ] Does this PR complete the epic?
• [ ] Are links included to any other gov-approved PRs associated with epic?
• [ ] Does PR include documentation for Raft's a11y review?
• [ ] Did automated and manual testing with iamjolly and ttran-hub using Accessibility Insights reveal any errors introduced in this PR?

Deliverable 5: Deployed

• [ ] Was the code successfully deployed via automated CircleCI process to development on Cloud.gov?

Deliverable 6: Documented

• [ ] Does this PR provide background for why coding decisions were made?
• [ ] If this PR introduces backend code, is that code easy to understand and sufficiently documented, both inline and overall?
• [ ] If this PR introduces frontend code, is that code easy to understand and sufficiently documented, both inline and overall?
• [ ] If this PR introduces dependencies, are their licenses documented?
• [ ] Can reviewer explain and take ownership of these elements presented in this code review?

Deliverable 7: Secure

• [ ] Does the OWASP Scan pass on CircleCI?
• [ ] Do manual code review and manual testing detect any new security issues?
• [ ] If new issues detected, is investigation and/or remediation plan documented?

Deliverable 8: User Research

Research product(s) clearly articulate(s):

• [ ] the purpose of the research
• [ ] methods used to conduct the research
• [ ] who participated in the research
• [ ] what was tested and how
• [ ] impact of research on TDP
• [ ] (if applicable) final design mockups produced for TDP development

[codecov[bot]]

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 92.99%. Comparing base (b5f8bd7) to head (599f628).

Additional details and impacted files

[](https://app.codecov.io/gh/raft-tech/TANF-app/pull/2993?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=raft-tech) ```diff @@ Coverage Diff @@ ## develop #2993 +/- ## =========================================== + Coverage 92.97% 92.99% +0.01% =========================================== Files 276 277 +1 Lines 7316 7320 +4 Branches 646 646 =========================================== + Hits 6802 6807 +5 Misses 413 413 + Partials 101 100 -1 ``` | [Flag](https://app.codecov.io/gh/raft-tech/TANF-app/pull/2993/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=raft-tech) | Coverage Δ | | |---|---|---| | [dev-backend](https://app.codecov.io/gh/raft-tech/TANF-app/pull/2993/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=raft-tech) | `93.03% <100.00%> (+<0.01%)` | :arrow_up: | | [dev-frontend](https://app.codecov.io/gh/raft-tech/TANF-app/pull/2993/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=raft-tech) | `92.70% <ø> (+0.09%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=raft-tech#carryforward-flags-in-the-pull-request-comment) to find out more. | [Files](https://app.codecov.io/gh/raft-tech/TANF-app/pull/2993?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=raft-tech) | Coverage Δ | | |---|---|---| | [tdrs-backend/tdpservice/parsers/transforms.py](https://app.codecov.io/gh/raft-tech/TANF-app/pull/2993?src=pr&el=tree&filepath=tdrs-backend%2Ftdpservice%2Fparsers%2Ftransforms.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=raft-tech#diff-dGRycy1iYWNrZW5kL3RkcHNlcnZpY2UvcGFyc2Vycy90cmFuc2Zvcm1zLnB5) | `100.00% <100.00%> (ø)` | | | [...ations/0029\_alter\_tribal\_tanf\_t4\_closure\_reason.py](https://app.codecov.io/gh/raft-tech/TANF-app/pull/2993?src=pr&el=tree&filepath=tdrs-backend%2Ftdpservice%2Fsearch_indexes%2Fmigrations%2F0029_alter_tribal_tanf_t4_closure_reason.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=raft-tech#diff-dGRycy1iYWNrZW5kL3RkcHNlcnZpY2Uvc2VhcmNoX2luZGV4ZXMvbWlncmF0aW9ucy8wMDI5X2FsdGVyX3RyaWJhbF90YW5mX3Q0X2Nsb3N1cmVfcmVhc29uLnB5) | `100.00% <100.00%> (ø)` | | | [...backend/tdpservice/search\_indexes/models/tribal.py](https://app.codecov.io/gh/raft-tech/TANF-app/pull/2993?src=pr&el=tree&filepath=tdrs-backend%2Ftdpservice%2Fsearch_indexes%2Fmodels%2Ftribal.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=raft-tech#diff-dGRycy1iYWNrZW5kL3RkcHNlcnZpY2Uvc2VhcmNoX2luZGV4ZXMvbW9kZWxzL3RyaWJhbC5weQ==) | `100.00% <100.00%> (ø)` | | ... and [1 file with indirect coverage changes](https://app.codecov.io/gh/raft-tech/TANF-app/pull/2993/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=raft-tech) ------ [Continue to review full report in Codecov by Sentry](https://app.codecov.io/gh/raft-tech/TANF-app/pull/2993?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=raft-tech). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=raft-tech) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://app.codecov.io/gh/raft-tech/TANF-app/pull/2993?dropdown=coverage&src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=raft-tech). Last update [b5f8bd7...599f628](https://app.codecov.io/gh/raft-tech/TANF-app/pull/2993?dropdown=coverage&src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=raft-tech). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=raft-tech).

[robgendron]

3 points remaining.

[robgendron]

Finishing creation of dashboards. Aiming to publish PR today - 2 point remaining.

[elipe17]

Note, we can't deploy this again unless we update the elastic instance to 7.10.x manually. I did that once, but it seems other deployments reverted it back and the terraform refuses to update it in the pipeline.

[raftmsohani]

Note, we can't deploy this again unless we update the elastic instance to 7.10.x manually. I did that once, but it seems other deployments reverted it back and the terraform refuses to update it in the pipeline.

@elipe17 You might have to add the version as a variable to terraform

[elipe17]

Note, we can't deploy this again unless we update the elastic instance to 7.10.x manually. I did that once, but it seems other deployments reverted it back and the terraform refuses to update it in the pipeline.

@elipe17 You might have to add the version as a variable to terraform

@raftmsohani the terraform does specify the version, but it bombs out when it tries to update elastic in the pipeline. I could try deleting the elastic instance and see if it recreates it all together. But that would kill the qasp environment too for a little while.

[elipe17]

Adding this reference to the PR for an annoying feature of Date based aggregations. I also added it in the comments of the scripted field I created for RPT_MONTH_YEAR.