Silver Cluster Post outage check-list

[ShellyXueHan]

Describe the issue Action items listed by Sal:

• [x] Shelly and Ian will fix the Cerberus monitoring for SILVER #4327
• [x] Shelly will fix the Cerberus monitoring for EMERALD #4333
• [x] Ian will improve the monitoring for all the clusters (Long-term)
• [x] Cailey will be adding a banner to uptime with a resetting disclaimer. Cailey will reset the monitoring for SILVER. Mention at the community meetup how "history and incidents" work on uptime.com
• [x] Paton, Cam will review Incident Management emails and find out why some people didn't get the resolution email (Sal/Olena/Shelly) A separate dist list has been created by the DXC Situation manager for Platform Team leadership (Olena, Luke, Shelly, Cailey, Nick, and Platform Services team Mailbox) to ensure the email reaches everyone it need to
• [x] Cailey will take incident management's template and put it in GitHub for future use in email and Rocket.Chat Communication Doc. Situation Management is the group to "bug" when an incident is underway for updates. Plat Admin to report out to community using the template above when an outage is happening. Plat Admins are optional to join the bridge call using the situation management email. All notifications should be identical in Rocket.Chat and Uptime.com. Notifications should be provided every hour.
• [ ] Sal to work with App Dev team to move the Registry and Mautic to GOLD cluster (long-term) #1094
• [x] Cailey to finish moving Artifcatory to GOLD cluster #3832
• [x] Clint McNichol will send Sal RCA findings. Sal will draft a follow up email to the community. Include Dan in the review of the email.
• [x] Cailey will create an outage checklist. Work with Barre, Steven and review his existing checklist #4378

Definition of done

• [ ] review with the team and make sure the above are completed

[IanKWatts]

Tickets for improvement of monitoring include:

• 4327 general cluster monitoring improvements

• 4381 argocd monitoring improvements

• 4017 vault agent injector HA

• 4529 monitor vault test app

[lukegonis]

Closing this epic, one outstanding item remains (Registry to Gold/GoldDR)