Maintain Applications (new page)

[mtspn]

Maintain Applications new page.

Definition of done

• [x] First draft
• [x] SME review
• [x] Approval

[mtspn]

Notes from team day on topics that could be included on this page: Monitor an Application

• Sysdig content from 201, uptime docs,
• ArgoCD notifications, post in RC or send email when sync fails etc.
• Improve your app: move secrets to vault, using Artifactory more, review your ACS results, improve your Database backups, configure a HPA, configure a PDB
• Sidecar container for monitoring (link to Sid’s sidecar app https://github.com/bcgov/elmsd-nodejs/tree/main/packages/openshift/templates/efk-stack)
• Using S3 for long term storage

[Pilargit12]

Draft outline Google Doc: https://docs.google.com/document/d/1Gc8SfaDjYeH_HTnt3IvEO6VbNCfyJOwr2EZmny8-M7A/edit

Each author will be writing their own outline to draft this document, we will then review it together and vote for the best outline and focus on this new page creation.

[Pilargit12]

Final outline has been established and division of sections.

@IanKWatts - Reliability and Resiliency | Security (August 9 - 10 ) @ShellyXueHan - Monitoring and Alerting | Communitation (August 9 - 10) @caggles - Images | Personnel (August 4)

[ShellyXueHan]

@Pilargit12 copy pasting the markdown content in google doc doesn't work very well as the format breaks. So I'm just putting them here for now, it's easier to put it into a PR for review later when needed!

Monitoring and Alerting

Application monitoring is a critical aspect of maintaining a healthy and efficient application environment. It allows you to proactively manage performance, detect and resolve issues quickly, and make informed decisions to enhance the overall user experience and business outcomes.

Once your application is running in OpenShift, you can use Sysdig to monitor app healthiness and performance via Kubernetes metrics. Here are a list of steps to follow:

• Onboard to Sysdig and setup the OpenShift project set access to monitor your applications.
• Create Sysdig monitoring dashboards to gathers important application metrics there, such as resource utilization and service Golden Signals.
• Having just monitoring dashboards is not enough as you will not be looking at them all day long. Instead, it's suggested to setup alerts for monitoring metrics, this way you will get notified when symptoms start to appear and resolve issues in early stage. Follow the guide on how to setup Sysdig alerts via Rocket.Chat.
• You can setup comprehensive service monitoring with PromQL in Sysdig that includes multiple metrics and complex logics for alerting. If you'd like to expose application specific metrics, here are steps on how to create custom/user defined monitoring.

If service availability is important to you, leverage Uptime.com for uptime monitoring and public service status pages. The Platform Services Team uses Uptime.com to share the status of OpenShift clusters as well as shared services. You can check out the SaaS service catalog to explore more about Uptime.com.

With comprehensive monitoring and alerts setup for your application can greatly reduce service downtime or disaster, but this does not guarantee the application will be up and running 100% without any issues. In the event of an issue or downtime, you can:

• follow this guide to determine the source of outage, whether that be a platform wide issue or your application specific issue, and how to start troubleshooting your application issues. In addition, it's a good idea to prepare a checklist for your application that details what you need to check to ensure that everything is up and running after an outage.
• leverage application logs for troubleshooting. OpenShift integrates with Kibana for log aggregation. This allows you to collect, index, and visualize container logs in a centralized location. Here are the training material for application logging with Kibana.
• nonetheless, it's crucial for your team to have the knowledge base of your application and relevant components to efficiently troubleshoot and reduce the impact of the downtime.

Communication

The OpenShift platform is targeted to provide zero downtime during planned maintenance activities, such as version upgrades. However, this does not mean that your application is all set once it's running in production. Platform maintenance do impact your workloads in various ways, especially major version upgrades and deprecation of outdated features and APIs. Thus, it is important for your team to stay up-to-date with the platform changes and keep an active communication with Platform Services Team and the community.

Before any scheduled platform and shared services changes, Platform Services Team will reach out to all Product Owners and Technical Leads registered from the Product Registry. To ensure your team gets notified of upcoming events in advance, please make sure the contact information for POs and TLs are accurate and updated all time on the Product Registry.

In addition to that, it's recommended for all team members to stay connected with the community via the following channels:

• subscribe to Platform Services Newsletters so that you will receive email notifications for upcoming changes, as well as the Platform Community MeetUp Event every 3 weeks for platform updates and technical demos.
• monitor Rocket.Chat #devops-alerts channel for service alerts and status updates. Rocket.Chat serves as the main communication method for teams on the platform to get help, share knowledge and information, and build up the community.
• check out more Platform activities and resources from time to time, such as OpenShift training events, StackOverflow knowledge base, and so on.

[Pilargit12]

Github PR can be found:

https://github.com/bcgov/platform-developer-docs/pull/182

[Pilargit12]

New PR to fix broken links: https://github.com/bcgov/platform-developer-docs/pull/184