Document and/or Automate all Aspects of Checking an Openshift Cluster and Supporting Services

[wmhutchison]

Describe the issue

This is a top-level EPIC that aims to act as an umbrella for reliably tracking and ensuring that every single component of a vendor-supplied Openshift cluster is being accurately tracked during maintenance activities, and also ensure when required that application owners are engaged to QA work when maintenance is completed and when automation has not yet been created to confirm these additional services via Canary applications and other checks.

EPICs that will nest under this EPIC will be promoted as appropriate as Sprint Goals, and tickets under each EPIC will be created for Platform Operations as well as targeted individuals from Platform Services as needed to ensure all involved stakeholders have separate tickets that accurately reflect their required time Estimate and setup Blocker dependencies if necessary.

Definition of Done

• [ ] Identify all components needed to be tracked before/during/after each maintainance activity.
• [ ] Ensure all identified components are covered off in some fashion, be it via automation or contacting the product owners of the involved technologies.

[wmhutchison]

Apart from DB Canary apps, we need a recurring (manual or automated) method to perform master fail-overs of the DB technology to ensure that works. Canary monitoring will then detect issues or failures for this.

Will document as a requirement for now during regular OCP maintenance, but definitely want this automated, but probably during business hours.

[wmhutchison]

Fail-over tests should be as "mean" as possible to the Patroni cluster in question. Either stand-alone or integration into Kraken is to be considered.

[wmhutchison]

Closing this EPIC as no longer required. Said monitoring is now done via n8n which is managed by the Platform Services Team.