Rollout template (updated)

[tkapin]

Purpose

This issue tracks the arcade-services repository rollout. On top of the Rollout instructions described on the wiki, it provides the person responsible for the rollout checklist of the steps that should to be performed to rollout services in this repository. All relevant information, including the rollout PR, issues encountered during the rollout and steps taken to resolve them should be linked or added to this issue to keep full audit trail of changes rolled out to production.

Process

Build status check (Monday)

• [ ] Check the status of the dotnet-arcade-services-weekly pipeline
• [ ] Ensure the arcade-services-internal-ci pipeline is green
• [ ] In case there is a problem with the CI build, notify the Rollout channel

Rollout preparation (Tuesday)

• [ ] Wait for the vendor to prepare the rollout. This includes a thread on the Rollout channel, rollout issue in AzDO and the rollout PR
• [ ] Link the rollout PR to the "Rollout data" section of this issue
• [ ] Double-check that the release notes contain all information
• [ ] Merge the already prepard rollout PR
• [ ] Ensure the build is green and stops on the "Approval" phase

Rollout day (Wednesday)

• [ ] Approve the rollout build (that has been already started the day before)
• [ ] Monitor the rollout build for failures. The Maestro exceptions query might help in diagnosing issues.
• [ ] Keep track of any issues encountered during the rollout either directly in this issue, or in a dedicated issue linked to this issue
• [ ] Update the rollout stats in the "Stats" section below
• [ ] Notify the Rollout channel
• [ ] Close this issue with closing comment describing a high-level summary of issues encountered during the rollout

Rollback

In case the services don't work as expected after the rollout, it's necessary to roll back.

• [ ] Announce the issues on the Rollout channel, rollout issue in AzDO
• [ ] Notify the partners that we'll be rolling back
• [ ] Rollback as described on the Rollback / Hotfix wiki page
• [ ] Validate the rolled-back services are running as expected
• [ ] Announce successful rollout on the Rollout channel
• [ ] Notify the partners that the rollback has been finished (as reply on the original email)

Rollout data

Rollout PRs

• The main PR: <TO BE FILLED>
• Rollback PRs: <TO BE FILLED, IF APPICABLE>

Rollout times

Use the following Kusto query to gather data about rollout times:

Pre-Approval run time: <TO BE FILLED> Post-Approval run time: <TO BE FILLED>

Useful links

• AzDO pipelines
  • arcade-services-internal-ci
  • dotnet-arcade-services-weekly
• Rollout channel
• Maestro exceptions
• Deployment Policy