Rollout 2024-05-22

[oleksandr-didyk]

Purpose

This issue tracks the arcade-services repository rollout. On top of the Rollout instructions described on the wiki, it provides the person responsible for the rollout checklist of the steps that need to be performed to rollout services in this repository. All relevant information, including the rollout PR, issues encountered during the rollout and steps taken to resolve them should be linked or added to this issue to keep full audit trail of changes rolled out to production.

Process

Build status check (Monday)

• [x] Check the status of the dotnet-arcade-services-weekly pipeline
• [x] Rotate any secrets that need manual rotation
• [x] Check the status of the arcade-services-internal-ci pipeline. Try to fix issues, if any, so that we have a green build before the rollout day.
• [x] Check the Rollout column in the Product Construction board - move any issues rolled-out last week into Done

Rollout preparation (Tuesday)

• [x] Check that the vendor prepared the rollout:
  • Thread on the Rollout channel
  • Rollout issue in AzDO
  • Rollout PR in arcade-services
• [x] In case there is a problem with the CI build, notify the Rollout channel
• [x] Link the rollout PR to the Rollout PRs section of this issue
• [x] Double-check that the release notes contain all information
• [x] Merge the already prepared rollout PR (⚠️ DO NOT SQUASH)
• [x] Link the rollout build to the Rollout build section of this issue
• [x] Verify that Maestro opened a production => main PR in arcade-services with the rollout merge commit (example). There should be no changes in the PR to any files. Do not merge the PR yet.
• [x] Ensure the build is green and stops at the Approval phase

Rollout day (Wednesday)

• [x] Approve the Approval stage of the rollout build (that has been already started the day before)
• [x] Monitor the rollout build for failures.
  • Note: this Maestro exceptions query might help in diagnosing issues.
• [x] Keep track of any issues encountered during the rollout either directly in this issue, or in a dedicated issue linked to this issue
• [x] Update the rollout stats in the Stats section below. The statistics will be available in Kusto a few minutes after the build was finished
• [x] Notify the Rollout channel
• [x] Merge the production => main Maestro PR in arcade-services (⚠️ DO NOT SQUASH)
• [x] Move rolled-out issues in the Rollout column of the Product Construction board into Done. Add a link in to this rollout issue in the comments before closing them (example)
• [x] Close this issue with closing comment describing a high-level summary of issues encountered during the rollout
• In case of rollback, uncomment the Rollback section below and follow the steps there

Rollout data

Rollout PRs

• The main PR: https://github.com/dotnet/arcade-services/pull/3544

Rollout build

• Rollout AzDO build: https://dev.azure.com/dnceng/internal/_build/results?buildId=2456993&view=results

Rollout times

Use the following Kusto query to gather data about rollout times:

• Pre-Approval run time: 00:48
• Post-Approval run time: 03:28

Useful links

• AzDO pipelines
  • arcade-services-internal-ci
  • dotnet-arcade-services-weekly
• Rollout channel
• Maestro exceptions
• Deployment Policy

[oleksandr-didyk]

Rollout finished successfully.

Maestro deployment had to be restarted once due to timeout in the Deploy Service Fabric Application (Maestro) job:

##[error]8:56:39 AM - The deployment 'MaestroApplicationType.1.0.42.20240521.1-1.1.1' failed with error(s).
Status Message: Resource operation failed. Operation: CreateOrUpdate. Error details: {
  "FailureReason": "UpgradeDomainTimeout",
  "Details": null
} (Code: ClusterChildResourceOperationFailed)