Monitoring and Telemetry of the Release Infrastructure

[andriipatsula]

Goals & Motivation

Currently we don't have any mechanism to measure the health and performance of the Release Infrastructure or to monitor infrastructure-related issues. With each build of the Staging and Release pipelines, numerous errors and warnings arise. We need to implement a solution that allows us to track these issues, identify their severity and measure execution time. By establishing proper monitoring, we can ensure the stability and reliability of our Release Infrastructure.

Business Objectives

One of the primary business objectives of the .NET Release Infrastructure Adoption is to establish telemetry and alerting mechanisms, enabling us to effectively monitor the release infrastructure.

• [x] We have identified and defined metrics, adapting our processes effectively. This allows us to establish a well-defined mechanism for comparing the release process over time.
• [x] We can measure and visualize the performance of the Release Infrastructure, enabling us to consistently enhance the infrastructure and effectively showcase the improvements.

Goals

1. We aim to quantify the number of manual steps performed during the release process.
2. We aim to quantify the number of issues detected during the release process.
3. Our goal is to monitor the time spent by the Release Infrastructure and .NET Release team on issue resolution during the release process.
4. Our goal is to monitor the number of errors and warnings encountered during the release process, comparing each build over time.
5. By introducing the ability to rerun parts of the Staging pipeline, our goal is to detect bottlenecks and calculate how much time and effort it saves.
6. We aim to quantify the time needed for the end-to-end process of releasing the .NET product, from acquiring a coherent build to the release of the final bits.
7. We aim to quantify the time needed to prepare an "emergency" release (to be compliant with the K hours SLA).

Milestones

• [x] Explore the metrics and methods for measuring the health and performance of the Release infrastructure.
• [x] Explore the inclusion of extra steps and stages for reporting metrics.
• [x] Create a Proof of Concept (PoC) that sends telemetry (Events, Metrics) from the Staging pipeline to Application Insights.
• [x] Write a one-pager where Metrics are identified and described.
• [x] Introduce templates for production issues and release audit.
• [x] Write documentation that outlines how we integrate metrics into the release processes.
• [x] Find a way to store Metrics data and choose how to display it visually.
• [ ] Follow the new processes to prepare and supervise a release in December.

One pager

https://dev.azure.com/dnceng/internal/_git/dotnet-release?path=/documentation/OnePagers/telemetry-and-monitoring.md

Release Issue template

https://github.com/dotnet/release/blob/main/.github/ISSUE_TEMPLATE/release-issue-template.yml

[andriipatsula]

PR: https://dev.azure.com/dnceng/internal/_git/dotnet-release/pullrequest/35165

[andriipatsula]

The new Release Issue template was introduced: https://github.com/dotnet/release/blob/main/.github/ISSUE_TEMPLATE/release-issue-template.yml

[tkapin]

What needs to be done to close this yet? Small things form my perspective: move the dashboard from "apatsula" folder to "release-infra" folder, ensure the dashboard json is backed up / stored properly, send the woohoo email with description of what has been done and agreed on.

[andriipatsula]

I'm closing this task. A 'Woohoo' email has been sent out. Additionally, I've established a backup for the dashboard and set up automatic deployment.