pulumi / ci-mgmt

Configuration for all things CI
Apache License 2.0
10 stars 4 forks source link

Pulumi CI Management

Purpose

This repository contains code to manage CI/CD for the many Pulumi providers in a consistent and (mostly) automated manner. The repo's intended audience are Pulumi Corp engineers, but its contents may also serve as a helpful example for Pulumi community members looking to maintain their own providers with a similar CI/CD process to Pulumi Corp.

Pulumi providers use GitHub Actions for CI/CD. Because we maintain a long list of providers, we use this repository to:

Usage

This repository has the following components:

Prerequisites

The following tools are required for generating and deploying GitHub Actions workflows:

Building

After checking out the code, run the following command:

cd provider-ci && make clean && make -j

Common commands:

Adding generated workflows to your local provider repository

Sometimes, you want those changes NOW rather than having to wait for a GitHub PR. This example command will generate workflows for pulumi-datadog, and place them in the specified --out directory. Adjust for your provider and filesystem.

./bin/provider-ci generate --name pulumi/pulumi-datadog --template bridged-provider --config ./providers/datadog/config.yaml --out ../../pulumi-dtadog

Adding a New Bridged Provider

To add a new provider:

  1. A new provider needs a top-level .ci-mgmt.yaml in its own repository with the following basic configuration:

    # Required values:
    provider: foo # substitute the name of your provider, without the pulumi- prefix
    env: # A map of required configuration for any integration tests, etc.
     AN_OPTION: value
     ANOTHER_OPTION: true
     # etc.
    lint: true # Linting should be true in most cases, unless failing rules in the upstream provider makes this impractical.
    
    # Optional values:
    docker: true # Whether the provider's tests use Docker to run. If set to true, a file `testing/docker-compose.yml` must be present in the provider repository.
    setup-script: testing/setup.sh # Path to a script that's used for testing bootstraps

    ci-mgmt will read your provider's .ci-mgmt.yaml and generate the standard set of CI files from templates. You can override every one of the default values in your .ci-mgmt.yaml file.

  2. Add your provider to provider-ci/providers.json in alphabetical order. This ensures your provider receives regular updates and maintenance.

  3. Add your provider to .github/ISSUE_TEMPLATE/0-ecosystem-providers.md to ensure manual rollouts track your provider.

  4. Commit the changes and open a pull request.

  5. To receive a pull request with your new config files, you can run the Update Workflows, Single Bridged Provider workflow run, using your provider name as the input. Another option is to wait for the nightly cronjob to send this pull request automatically.

  6. If you would like to manually generate the configuration to get started right away, you can do so in your provider repository root:

    go run github.com/pulumi/ci-mgmt/provider-ci@master generate \
      --name pulumi/pulumi-$(PROVIDER_NAME) \
      --out . \
      --template bridged-provider \
      --config .ci-mgmt.yaml

    The generated files will be writen to your current directory.

Updating All Bridged Providers

The Update GH Workflows, ecosystem providers Workflow runs on a nightly schedule. You may trigger this Workflow manually; however be aware that this causes a lot of GitHub Actions to run at the same time, which may cause rate limiting across the org. Plan ahead and do this at a low-traffic time.

Updating GitHub workflow schema

Fetch the latest JSON Schema then re-generate type definitions:

make discovery

Automatically editing source code across provider repositories

You can apply ad-hoc source edits across provider repositories even on files that are not managed or generated by ci-mgmt. For example, you might need to automatically update example code to use a newer Pulumi SDK major version dependency or a newer version of the underlying infrastructure such as .NET Framework. This can be done with sourcemigrator: