search

microsoft / mu_devops

Project Mu Developer Operations
https://microsoft.github.io/mu/
Other
26 stars 22 forks source link
continuous-integration firmware projectmu uefi uefi-development

readme

=================================================== Project Mu Developer Operations (DevOps) Repository

|Latest Mu DevOps Release Version (latest SemVer)| |Commits Since Last Release| |Sync Mu DevOps Files to Mu Repos| |Containers Build|

.. |Latest Mu DevOps Release Version (latest SemVer)| image:: https://img.shields.io/github/v/release/microsoft/mu_devops?label=Latest%20Release :target: https://github.com/microsoft/mu_devops/releases/latest

.. |Commits Since Last Release| image:: https://img.shields.io/github/commits-since/microsoft/mu_devops/latest/main?include_prereleases :target: https://github.com/microsoft/mu_devops/releases

.. |Sync Mu DevOps Files to Mu Repos| image:: https://github.com/microsoft/mu_devops/actions/workflows/FileSyncer.yml/badge.svg :target: https://github.com/microsoft/mu_devops/actions/workflows/FileSyncer.yml

.. |Containers Build| image:: https://github.com/microsoft/mu_devops/actions/workflows/Build-Containers.yml/badge.svg?branch=main :target: https://github.com/microsoft/mu_devops/actions/workflows/Build-Containers.yml

This repository is part of Project Mu. Please see Project Mu for details https://microsoft.github.io/mu

This repository is used to manage files related to build, continuous integration (CI), and continuous deployment (CD) for other Project Mu repositories.

Many of these files are generic YAML templates that can be combined together to compose a fully functional pipeline.

Python based code leverages edk2-pytools to support cross platform building and execution.

You can find a high-level summary of the latest changes since the last release by viewing the latest draft release_.

.. _latest draft release: https://github.com/microsoft/mu_devops/releases

Table of Contents

  1. Project Mu Developer Operations (DevOps) Repository_

  2. Table of Contents_

  3. Continuous Integration (CI)_

    • Code Coverage_

  4. Conventions_

  5. Containers_

  6. GitHub Automation Workflow Overview_

    • Leaf Workflows & Reusable Workflows_

    • Reusable Workflows_

    • Leaf Workflows_

  7. GitHub Automation Workflow Summary_

    • Auto Merge_

    • Auto Approver_

    • File Synchronization_

    • Initial Issue Triage_

    • Issue Assignment_

    • Label Automation_

    • Pull Request Validator_

    • Release Drafting_

    • Scheduled Maintenance_

    • Stale Detection_

  8. GitHub Action Summary_

    • Submodule Release Updater_

  9. Links_

Continuous Integration (CI)

There are two broad categories of CI - Core CI and Platform CI. You may see these terms used in the repo.

Code Coverage

mu_devops provides azure pipeline templates for uploading code coverage to the pipline currently executing, or directly to codecov.io for public github repositories that use azure pipelines. No matter the yaml file you are using, whether it be MuDevOpsWrapper.yml, Jobs/PrGate.yml, Steps/PrGate.yml or Steps/UploadCodeCoverage.yml itself, the only a action that needs to be taken is to set an environment variable coverage_upload_target to ado or codecov.

Conventions

Containers

This repo maintains containers used throughout Project Mu projects. Containers provide well-defined, ready-to-go images and result in improved performance, portability, and consistency of build environments. Project Mu leverages containers for both server-side builds (e.g. pull requests and continuous integration) and for local developer builds.

At this time, containers are only offered for Linux. If you want to get started quickly and receive the smoothest build experience, it is recommended to use containers where available.

The Containers directory contains the actual dockerfiles for building the containers. The containers are actually built (in pull requests to dockerfiles and merges to the main branch) in the .github/workflows/Build-Containers.yml workflow. On any change to a dockerfile a new container is built and pushed to the Microsoft GitHub container registry as a container package associated with this repo. The latest Project Mu container builds are available in the Packages - Mu DevOps container feed and more information is available in the Container Readme file.

.. _Container Readme file: https://github.com/microsoft/mu_devops/blob/main/Containers/Readme.md .. _Packages - Mu DevOps: https://github.com/orgs/microsoft/packages?repo_name=mu_devops

GitHub Automation Workflow Overview

This repository also drives automation of Project Mu GitHub repositories.

Leaf Workflows & Reusable Workflows

Workflows are split into two categories (1) leaf and (2) reusable.

The main reason for reusable workflows is to consolidate the main logic for the workflow to a single file and allow the leaf workflow to be present in repositories that opt into what the reusable workflow provides. Leaf workflows can provide any repo-specific input to a reusable workflow (if necessary). Leaf workflows can be considered minimal wrappers around reusable workflows.

Reusable Workflows

These workflow are only designed to be called from other workflows. The files are maintained in the .github/workflows directory of this repository. This is mandatory as GitHub only allows leaf workflows to call reusable workflows located in this directory.

Review reusable workflow files to understand what they do and what input parameters are available.

Leaf Workflows

These workflow are only designed to call reusable workflows. They should not directly invoke GitHub Actions. The actual GitHub Actions used by Project Mu are centrally tracked/updated in the single-copy reusable workflow files in the Mu DevOps repo. This allows dependabot to update the actions here at once.

GitHub Automation Workflow Summary

Following is a brief summary of the actual workflows in the repository.

Auto Merge

As automated bots pick up mundane tasks like syncing PIP module updates, submodules, files, and so on, an increasing number of pull requests can accumulate that essentially update dependencies we expect to be updated over time. In most cases, we simply care that the new update passes CI checks.

Therefore, Project Mu repos auto merge certain pull requests to reduce human burden of approving these requests in all of the Project Mu repos. Individual repos can opt out of this functionality by removing the leaf workflow sync to their repo, however, it is recommended to keep this flow enabled for consistency across all repos.

To see more about this flow look in these files:

A Project Mu repo simply needs to sync .sync/workflows/leaf/auto-merge.yml to their repo in Files.yml and the auto merge workflow will run in the repo.

Auto Approver

Auto approves pull requests from allowed bot accounts. As part of reducing dependency overhead, this workflow first approves pull requests that are then auto merged after CI status checks complete. If a CI status check (e.g. build) fails, the pull request will not be merged.

Note: This is currently disabled in most Project Mu repos.

To see more about this flow look in these files:

A Project Mu repo simply needs to sync .sync/workflows/leaf/auto-approve.yml to their repo in Files.yml and the auto approve workflow will run in the repo.

File Synchronization

Because Project Mu is distributed over many repositories, a need arises to sync common files across all of the repos. This is done via the .github/workflows/FileSyncer.yml workflow in Mu DevOps. It determines how to map files from Mu DevOps to any repo with the configuration file .sync/Files.yml.

The configuration file can map any file in Mu DevOps to any file path in a destination repo. Flexibility is provided to map the same file to different file paths in different repos, not map the file to some repos, etc. Whole directories can also be synced as well.

The file sync operation automatically runs anytime a file in the .sync/ directory of Mu DevOps is updated.

The file modification flow should be as follows:

  1. Developer updates a synced file in Mu DevOps
  2. Once PR for (1) is merged all mapped repos get a PR with the change
  3. Reviewers in each repo review and approve the PR
  4. The file is now in sync across all repos

File synchronization PRs are created by the Project Mu UEFI Bot_ account.

The file synchronization process will use the original commit title and message when syncing the change if it is triggered on a single commit. Therefore, it is recommended to make changes to sync files one file per commit at a time. If more than one file is modified, the PR is simply a single commit with a generic message containing both changes.

.. _Project Mu UEFI Bot: https://github.com/uefibot

Initial Issue Triage

This repo syncs GitHub issue form templates_ to many Project Mu repos. Part of initial triage for incoming issues involves parsing data in the issue form to apply the appropriate labels so the issue is ready for triage by a human.

Issues need to be triaged by a human when the state:needs-triage label is present. This workflow can parse details provided in issue forms to apply additional labels. For example, the state:needs-owner label is applied if the user indicates they are not fixing the issue, the urgency:<level> label is applied based on user selection in the urgency dropdown, etc.

A Project Mu repo simply needs to sync .sync/workflows/leaf/triage-issues.yml to their repo and the issue triage workflow will run in the repo.

.. _GitHub issue form templates: https://github.com/microsoft/mu_devops/tree/main/.sync/github_templates/ISSUE_TEMPLATE

This workflow works in concert with other issue workflows such as .sync/workflows/leaf/issue-assignment.yml to automate labels in issues based on the state of the issue.

Issue Assignment

A generic workflow that contains actions applied when GitHub issues are assigned. Currently, the workflow removes labels from the issue that are no longer relevant after it is assigned.

To see more about this flow look in these files:

Label Automation

Labels are automated from this repo in two main ways:

  1. Automatically synchronize labels across all Project Mu repos
  2. Automatically apply labels to issues and PRs

(1) is provided via the .github/workflows/LabelSyncer.yml reusable workflow with the labels defined in the file .github/Labels.yml.

(2) is provided via the .github/workflows/Labeler.yml reusable workflow with the labeling configuration defined in .sync/workflows/config/label-issues.

Labels are synced to all repos on a regular schedule that is the same for all repos.

Labels are automatically applied to issues and pull request on creation/modification and can be applied based on file paths modified a pull request or content in the body of the issue or pull request.

Pull Request Validator

Validates pull request formatting against requirements defined in the workflow. This workflow is not intended to strictly validate exact formatting details but provide hints when simple, broad changes are needed to enhance the quality of pull request verbiage.

Release Drafting

In order to ensure semantic versioning is followed based on well-defined labels used in Project Mu pull requests, the release drafting process is automated. On every PR merge, a draft release is updated that contains the PR change entry categorized according to the labels with the semantic version of the draft release updated according to the semantic version specification.

This means, that the details for an upcoming release are always available, the release format is consistent across Project Mu repos, and semantic versioning is followed consistently.

The draft release should be converted to an actual release any time the minor or major version is updated by a change.

To see more about this flow look in these files:

A Project Mu repo simply needs to sync .sync/workflows/leaf/release-draft.yml and the config file .sync/workflows/config/release-draft/release-draft-config.yml to their repo and adjust any parameters needed in the sync process (like repo default branch name) and the release draft workflow will run in the repo.

Scheduled Maintenance

Performs regularly scheduled maintenance-related tasks such as closing pull requests and issues marked stale. Similar tasks can be added to the workflow over time.

The leaf workflow contains the primary implementation and is directly synced to subscribed repos:

Stale Detection

Stale issues and pull requests are automatically labeled and closed after a configured amount of time.

This is provided by the .github/workflows/Stale.yml reusable workflow.

Individual repositories can control the label and time settings but it is strongly recommended to use the default values defined in the reusable workflow for consistency.

GitHub Action Summary

Following is a brief summary of the GitHub Actions maintained in the repository.

Submodule Release Updater

A GitHub Action and leaf workflow that automatically create a pull request for any submodule in a repo that has a new GitHub release available. The leaf workflow can easily be synced to repos and wraps around the GitHub action.

Links