EKS Cluster Upgrade `vets-api` Testing Discovery

[pjhill]

Description

In order to have immediate impact when the EKS cluster upgrade vets-api testing is ready to begin, Platform Reliability Team members who will be contributing need to discover and document key attributes. In particular, we will be looking to discover details and synthesize information for the below acceptance criteria.

Acceptance Criteria

• [x] We have analyzed and documented how many existing Datadog synthetics can be borrowed or adapted for our use.
• [x] We have analyzed and documented how much coverage we can expect from existing synthetics.
• [x] We have an analyzed what percentage of existing synthetics can be adapted for our use on this project as well as how we can adapt them
• [x] We have identified best practices for contributing new synthetics via Terraform

[IGallupSoCo]

I'm currently working my way through the DSVA repos I have access to - we have several set up for datadog analytics and configuration, so I believe it's a matter of finding the main Terraform file to get started with assessing the currently running Synthetics tests.

[IGallupSoCo]

It looks like at least some of the API tests for Synthetics are based here: https://github.com/department-of-veterans-affairs/terraform-aws-lighthouse-datadog/blob/main/modules/api-http-synthetics-test/main.tf

[IGallupSoCo]

At least some of the tests are kept here: https://github.com/department-of-veterans-affairs/devops/tree/master/terraform/environments/datadog

[IGallupSoCo]

[IGallupSoCo]

I've accounted for 25 of the 75 Synthetics tests explicitly flagged as api tests in the devops repo. My goal remains to locate the tests in our repos for closer consideration by the Product team. From there, we can adapt useful test scripts that already exist and engineer new scripts that fill out an appropriate level of test coverage for vets-api overall.

[IGallupSoCo]

I'm now building out a doc here: https://vfs.atlassian.net/wiki/spaces/PPT/pages/3150118913/EKS+DataDog+test+coverage

[IGallupSoCo]

So far, I have identified the following endpoint coverages, at least in part, within existing tests:

• appeals
• benefits_forms
• profile
• /v0/backend_statuses
• /v0/feature_toggles
• /v0/forms
• /flipper/features
• /v0/higher_level_reviews
• TIC GWN/GWW/GWS - not sure what this is yet
• /services/claims/v1/ - not in the api roster?
• /services/address_validation/ - not in the api roster?

[IGallupSoCo]

So that leaves us with these endpoint schemes that currently have zero test coverage: For v0 -

• benefits_info
• form_526
• prescriptions
• health_records
• secure_messaging
• gi_bill_institutions
• in_progress_forms
• claim_status_tool
• medical_copays
• notice_of_disagreements
• appointments
• contact_us
• benefits_reference_data
• claim_letters
• coe
• debts
• dependents_applications
• efolder
• financial_status_reports
• default
• form_1095_b
• mdot
• search_click_tracking
• search_typeahead
• central_mail_claim
• virtual_agent

For v1 -

• ivc_champva_forms
• higher_level_reviews
• income_limits
• benefits_status
• notice_of_disagreements
• supplemental claims

[pjhill]

Reading through TF guidance here -- https://vfs.atlassian.net/wiki/spaces/OT/pages/2075721739/Terraform+Standard

[pjhill]

I created a TestRail project here.

Per Isaac's analysis, I created stubs for --

• Each existing available synthetic monitor
• Each endpoint that does not currently have a monitor / synthetic defined

We end up with a total of 42 possible endpoint tests.

[pjhill]

Additional work about how to adapt the synthetics will be conducted under a new ticket next sprint. As a result of this ticket we discovered several potential issues with the testing technique we hope to use. We will be creating a spike/prototype of a parameterized synthetic before adapting all of them.