EKS Upgrade Testing Plan: Step 5: Endpoint Testing

[jennb33]

The VA Platform Product team is undergoing an EKS upgrade. As part of the process, it is necessary to conduct comprehensive testing on various components of Vets-API to ensure seamless functionality and performance during the upgrade.

From the EKS Testing Proposal:

As part of the current EKS upgrade process, comprehensive testing on various components of Vets API is essential in order to ensure seamless functionality and performance. Upgrading EKS can introduce changes that affect the behavior of deployed applications. Testing verifies that deployed applications remain compatible with any EKS changes. It ensures that all aspects of Vets API continue to operate as expected, without introducing regressions or unexpected behavior that could disrupt services for users.

Because of future goals to conduct upgrades on a 3-6 month cadence, it is the Product Platform team’s goal to maximize the automation of these upgrade testing processes. By automating these tests, verification of successful upgrades is potentially expedited and the right amount of time for necessary adjustments can be allocated. The Vets API components that require testing are below, along with the potential automation strategies. The objective is to standardize testing methodologies across the platform, thereby eliminating the inconsistency of testing solutions across EKS applications.

Part Five of Testing plan - Endpoint testing

Tasks

• [x] Identify endpoints that are to be tested in Vets-API
• [x] Basic endpoints like API status, welcome, v0/search
• [x] Aid in identifying appropriate endpoints for both auth and non-auth flows
• [x] Document process of testing and expected versus actual results.
• [x] Document the steps that will need to be followed by the person doing the upgrade testing in this Confluence document

Other Things To Consider

• [x] Need to verify that this can be automated; this can be automated perhaps by: (This will be automated by a Postman test suite running from inside the cluster on deployed to a postman container)
• [x] Datadog Synthetic
• [x] Discovery: Deploying a postman container to the cluster
• [x] Discuss plan to alert on any failures

Ideas

• Can we get a domain for the Vets Next Cluster?
• How Vets-api domain was created: 2022-06-06 New domain to serve vets-api traffic goes through TIC f5 device F5 Team: OIT DSO IO NETWORK NOC Network Edge Operations NOCNetworkEdgeOperations@va.gov we can reach out to them on new domain creation 220607 - Domain Procurement Process - Cloud Isolation
• Talked with Kyle later and we might not need to do this. According to Kyle it can be Dynamic configuration in K8s. since we don't want vets next to be publicly accessible
• http://api-vetstest-next-eks.vfs.va.gov is already pointed at the cluster,
• it'll require at least an ingressRoute pointing to vets, and if vets is still using a list of "Safe hosts" then further work will be needed.
• There is a port forwarding solution, but could potentially be automated with Datadog synthetic
• We have created a collection of manual functional test cases to execute on our example Preview Environment instance that can be reused (all or most of these) for this EKS upgrade test.
• We will test from the inside and the outside
• We can turn this into DataDog synthetic tests in the DD monitors (Terraform for DataDog?)
• Eventually these manual tests could be translated into synthetic monitors that we can point at an arbitrary environment.
• What is the solution around around dynamically updating the DNS for the given cluster we are testing

Success Metrics

• [x] Endpoints are identified for testing
• [x] There is a strategy for inside and outside testing (Internal testing is being prioritized as the ask here is scoped to Vets API and external testing would be testing additional parts of the stack)

Acceptance Criteria

• [x] Endpoints are identified
• [x] The Vets next application is healthy
• [x] Inside and outside testing strategies are discussed and confirmed
• [x] The documentation is updated with the necessary steps and instructions for this step. (updated decision log)

NOTE: This work can be done concurrently with tickets 74644 and 74647

[jennb33]

@LindseySaari This is the ticket/phase that the Reliability team/ @JoeTice @pjhill will get involved with the testing and QA process. It is important that we not let this ticket slide.

[jennb33]

7/10 - There is a Postman collection of the endpoints that are to be tested from Reliability. See meeting notes here to contextualize. This is more of a support ticket, as Reliability is doing the major testing.

[LindseySaari]

We now have the building blocks in place to move forward with the internal cluster testing. The Vets Next application is now healthy. Peters team is working through the postman collection and a solution to get a postman container running from inside the cluster. We've moved forward with Postman vs Datadog synthetic, which would require some initial automated setup for the private locations necessary for internal testing via Datadog synthetic.

For the most part though, we are acting as a support to the reliability teams portion of the EKS testing for the endpoints. We've also aided in identifying test paths and the best endpoints for executing both authenticated and non-authenticated flows.

[LindseySaari]

Todo Monday: Ping Peter & Isaac on the redis check

[LindseySaari]

For the scope of this effort, internal testing is top priority. External testing really isn't testing the application, but the Platform itself, which is a separate effort and Helm chart. Since the Vets API Vets Next is in a healthy state, we can close this ticket as we've completed the prep work for the endpoint testing. As always, we are available as a resource for any needs that arise from the Reliability team. I've updated the decision log with this information