EKS Upgrade Testing Plan: Step 6: Sidekiq Functionality

[jennb33]

The VA Platform Product team is undergoing an EKS upgrade. As part of the process, it is necessary to conduct comprehensive testing on various components of Vets-API to ensure seamless functionality and performance during the upgrade.

From the EKS Testing Proposal:

As part of the current EKS upgrade process, comprehensive testing on various components of Vets API is essential in order to ensure seamless functionality and performance. Upgrading EKS can introduce changes that affect the behavior of deployed applications. Testing verifies that deployed applications remain compatible with any EKS changes. It ensures that all aspects of Vets API continue to operate as expected, without introducing regressions or unexpected behavior that could disrupt services for users.

Because of future goals to conduct upgrades on a 3-6 month cadence, it is the Product Platform team’s goal to maximize the automation of these upgrade testing processes. By automating these tests, verification of successful upgrades is potentially expedited and the right amount of time for necessary adjustments can be allocated. The Vets API components that require testing are below, along with the potential automation strategies. The objective is to standardize testing methodologies across the platform, thereby eliminating the inconsistency of testing solutions across EKS applications.

Part Six of Testing plan - Sidekiq Functionality

Tasks

• [x] Find a basic Sidekiq job to use for testing
• [ ] Create Datadog monitor to check for errors on sidekiq startup on the new cluster.
• [ ] Document the steps that will need to be followed by the person doing the upgrade testing in this Confluence document.

Success Metrics

• [ ] Sidekiq functionality is tested and it runs
• [ ] Confluence doc is updated

Acceptance Criteria

• [ ] Sidekiq functionality is comprehensively tested during the upgrade.
• [ ] The documentation is updated with the necessary steps and instructions for this step.

[jennb33]

9/18/2024 update: need to check if the sidekiq UI is present. @LindseySaari verified it and asked @rmtolmach to do the 0-status. Might also test it by kicking off a basic job to run.

[rmtolmach]

We have a check to make sure Sidekiq's redis is working. That code is here and the output goes to the /v0/status endpoint which will be automatically checked with the postman tests Isaac has written.

To further check Sidekiq's functionality, we can check to make sure the sidekiq pods are healthy. We can do this by creating a log monitor in Datadog that catches initial sidekiq pod failure.

And as an additional check, we can kick off a sidekiq job in the console. I put instructions for that in the upgrade test doc.

[jennb33]

This ticket was duped for Sprint 11, as 93586 to complete the work. Closing this ticket for Sprint 10. cc: @rmtolmach