Add Rust Support [Rebase & FF]

makubacki commented 1 year ago

This pull request adds Rust support to mu_devops with patterns that extend to Rust adoption in additional Project Mu repos.

The changes can be grouped into three high-level parts:

Sync Rust environment files to repos
Add pipeline support to build and test Rust code
Update the build container to incorporate Rust support

These changes extend existing Rust efforts in Project Mu and improve user experience outside the CI environment.

Commit/Change Overview

.azurepipelines: Add YAML to cache Cargo tools

These files are added so dedicated pipelines can download the latest release from each tools GitHub repository and push the binaries to the pipeline artifacts.

Project Mu pipelines dependent on Rust tools can access the tools directly from the Project Mu pipeline artifacts which is faster than building and more reliable than depending on infrastructure outside the pipelines.

.sync/Files.yml: Move dev container section

Moves the section higher in the file to maintain alphabetic ordering of sections.

.sync: Sync common Rust files

Syncs the following files to repos that support Rust builds.

Makefile.toml - Cargo makefile configuration. Repos will no longer extend from MU_BASECORE.
rustfmt.toml - The Rust formatting settings for the repo.

Jobs/PrGate.yml: Give the linux container a reliable name

Passes the --name parameter as a container creation option in pipelines so steps that may need to operate on the container instance have a fixed name to refer to the container by.

The name is: mu_devops_build_container

Steps: Add Rust templates

Adds two new step templates.

RustSetupSteps.yml - Prepare Cargo for Rust builds including build during the edk build process and perform other actions available during pre-firmware build.
RustCargoSteps.yml.yml - Steps to run common Cargo commands on the workspace. Cargo should be installed and the workspace source cloned before running these steps.

.sync/azure_pipelines/MuDevOpsWrapper.yml: Add rust support

Allows repos that extend the YAML template to pass a new parameter rust_build. By default, the option is false. If true, then the repo will be set up to build for Rust and Cargo commands will be run at the workspace level to build and test code.

Dockerfile: Add Rust support

Updates the build container image to be able to build Rust code (including Rust UEFI code).

The following additional applications are installed to assist with setting up Rust dependencies:

curl, jq, and unzip

The CARGO_HOME and RUSTUP_HOME environment variables are set in the image and the cargo directory is added to the system path.

The version of Rust specified in Version.njk is installed alongside support for other tools used in Project Mu Rust builds such as cargo-make and cargo-tarpaulin. The latest release binaries are pulled directly from their GitHub releases to minimze container size impact.

The resulting container images can be used for local development and within pipelines.

Notes

The Rust toolchain version is defined in Version.njk. The version is used in:
- The Dockerfile to ensure the container version is in sync
- The YAML pipeline files to ensure (Windows) pipelines are in sync
- The rust-toolchain.toml file to ensure local builds in repos are in sync
The Cargo make makefile is no longer defined in MU_BASECORE and extended to repos. It is synced in whole to repos. Extending from MU_BASECORE unnecessarily complicated pipeline logic and required pulling MU_BASECORE through a submodule (or worse, an external git dependency) on an agent increasing build time.
The container image build is structured to provide a ready-to-go Rust dev environment while also minimizing space. The previous build container size was ~1.94GB. Adding the Rust tools initially increased size to 3.41GB. With a variety of adjustments, the size now is ~2.58GB.
After this PR completes, a docker container build will happen and a follow up PR will update the Version.njk file to use the new container build. After that PR is completed, repos dependent on the container will be able to access it.

File sync tested in the following PRs:

makubacki commented 1 year ago

acpica.org appears to be broken now which is breaking the container build. Will put up a separate PR to get iasl from our external mu-iasl feed first and then rebase this PR on that.

makubacki commented 1 year ago

acpica.org appears to be broken now which is breaking the container build. Will put up a separate PR to get iasl from our external mu-iasl feed first and then rebase this PR on that.

PR with the change to move away from acpica.org: https://github.com/microsoft/mu_devops/pull/227

makubacki commented 1 year ago

acpica.org appears to be broken now which is breaking the container build.

PR with the change to move away from acpica.org: #227

Rebase done

microsoft / mu_devops