Rollout 2024-07-11

[mmitche]

Purpose

This issue tracks the arcade-services repository rollout. On top of the Rollout instructions described on the wiki, it provides the person responsible for the rollout checklist of the steps that need to be performed to rollout services in this repository. All relevant information, including the rollout PR, issues encountered during the rollout and steps taken to resolve them should be linked or added to this issue to keep full audit trail of changes rolled out to production.

Process

Build status check

• [x] Check the status of the dotnet-arcade-services-weekly pipeline
• [x] Rotate any secrets that need manual rotation
• [x] Check the status of the arcade-services-internal-ci pipeline. Try to fix issues, if any, so that we have a green build before the rollout day.
• [x] Check the Rollout column in the Product Construction board - move any issues rolled-out last week into Done

Rollout preparation

• [x] Create the rollout PR:
  • Find a commit on main that you want to rollout
  • Create a branch named rollout/YYYY-MM-DD from that commit
  • Create a PR on GitHub from the rollout/YYYY-MM-DD branch to production
  • Name the PR [Rollout] Production rollout YYYY-MM-DD
  • Link this issue in the PR description
• [x] Link the rollout PR to the Rollout PRs section of this issue
• [x] Merge the prepared rollout PR (⚠️ DO NOT SQUASH)
• [x] Link the rollout build to the Rollout build section of this issue
• [x] Verify that Maestro opened a production => main PR in arcade-services with the rollout merge commit (example). There should be no changes in the PR to any files. Do not merge the PR yet.
• [x] Ensure the build is green and stops at the Approval phase

Rollout

• [x] Approve the Approval stage of the rollout build.
• [x] Monitor the rollout build for failures.
  • Note: this Maestro exceptions query might help in diagnosing issues.
• [x] Keep track of any issues encountered during the rollout either directly in this issue, or in a dedicated issue linked to this issue
• [x] When finished, update the rollout stats in the Stats section below. The statistics will be available in Kusto a few minutes after the build was finished
• [x] Merge the production => main PR in arcade-services (⚠️ DO NOT SQUASH)
• [x] Move rolled-out issues/PRs in the Rollout column of the Product Construction board into Done. Verify that PRs have a reference to the release at the bottom (example). If needed, manually add a comment with the reference (example)
• [x] Close this issue with closing comment describing a high-level summary of issues encountered during the rollout
• In case of rollback, uncomment the Rollback section below and follow the steps there

Rollout data

Rollout PRs

• The main PR: https://github.com/dotnet/arcade-services/pull/3721

Rollout build

• Rollout AzDO build: https://dev.azure.com/dnceng/internal/_build/results?buildId=2493203&view=results

Rollout times

Use the following Kusto query to gather data about rollout times:

• Pre-Approval run time: 16:39
• Post-Approval run time: 01:13

Useful links

• AzDO pipelines
  • arcade-services-internal-ci
  • dotnet-arcade-services-weekly
• Rollout channel
• Maestro exceptions
• Deployment Policy

[dkurepa]

The rollout required a hotfix to unblock https://github.com/dotnet/arcade-services/pull/3723. This is because the previous Maestro version had a broken darc add-build-to-channel command (which was the original reason for a rollout), so we had to pin darc to a previous working version

[dkurepa]

After this, we had some issues with publishing, but this was an arcade issue, which was fixed

[dkurepa]

After this the rollout succeeded without any issues