Closed WadeBarnes closed 5 months ago
Backup containers
Databases
S2I Builds
Email verification services specifically
BC Registries FDW Database
Others
Monitored Applications Affected:
aries-endorser-db
imageOthers Affected
I've spun the application pods back up and reviewed the environments for any other containers that were spun down. Next step is to review and identify what can be done to update the affected application pods.
Closing this. The investigation is complete. Addressing the issues is covered by https://github.com/bcgov/DITP-DevOps/issues/158
Platform services has started running jobs that scale down any pods that have not been updated (rolled out) in over a year. These scripts will be run every Tuesday from now on.
The idea is to eliminate any abandoned projects and free the associated resources as well as attempt to encourage best practices around pod/application maintenance.
The best practice set forth is to rebuild and redeploy application pods at least once a month in order to pick updates and patches performed to the base image(s). This will have knock-on effects in some of our projects such as those dependent on aca-py images.
As a workaround, application pods can be rolled out, this updates the resource manifests to include the current date.
For now we want to review the pods that did get scaled down and identify what's needed to updated them. We also want to identify what other pods may have been scaled down since there are some services in the
tools
and deployment environments we don't activity monitor.A separate ticket will be opened to discuss and design the update strategy moving forward.