search

emmeowzing / dynamic-continuation-orb

CircleCI orb for directory-targeted dynamically continued pipelines
https://circleci.com/developer/orbs/orb/bjd2385/dynamic-continuation
BSD 3-Clause "New" or "Revised" License
22 stars 12 forks source link
github probot

readme

CircleCI dynamic continuation orb

CircleCI

This orb is a merger of CircleCI's path-filtering and continuation orbs. It enables engineers to create configs under .circleci/ that only run when there are code changes in coinciding directory paths. Directory-targeted dynamically-executed pipelines offer engineers reduced execution time, and by extension, reduced CI costs.

example

This orb is based on a published example of advanced configuration with continuations from CircleCI.

See my article on Medium about dynamic continuations for a simple introduction to using this orb.

For users familiar with GitHub actions, this orb is analogous to the paths-filter action.

Usage

Get up-and-running with dynamically continued pipelines in these 4 steps:

  1. Add this orb, a continue job to your CI config (.circleci/config.yml), and the setup keyword, such as

    setup: true

orbs:
 dynamic: bjd2385/dynamic-continuation@<version>

workflows:
 on-commit:
   jobs:
     - dynamic/continue:
         base-revision: main
         context: circleci

    Note: pipeline config files under .circleci/ are automatically detected by default (cf. auto-detect in the documentation).

  2. Enable setup workflows in your project under Advanced Settings.

    dynamic config

  3. The circleci context in your organization must have two environment variables set for the orb to reference, including

    • CIRCLE_ORGANIZATION (in my case, this is set to bjd2385), and
    • CIRCLE_TOKEN, which contains your CircleCI API token.

  4. Move workflows to their new configs in .circleci/<config-name>.yml. Configs should be complete and pass a circleci config validate-check.

How it works

The orb will automatically detect and run a workflow (we'll call it <module>) if any of the following conditions are met.

  1. If .circleci/<module>.yml changes (this is configurable, enabled by default).
  2. If changes have been detected within the <module>/'s directory on your branch against the repository's default branch (defaults to main). See below on how to filter out CI runs from specific changed files.
  3. If, following merge to the default branch, there are changes to .circleci/<module>.yml or under <module>/, when diffing against the former commit (you must perform a merge commit for this to work properly).

These conditions can be overridden, and all workflows forced to run, if the force-all parameter is set to true on the continue job.

Examples

Basic directory layout

Show If you have a directory layout ```shell .circleci/config.yml terraform/ scripts/ src/ ``` with the addition of this orb, a user could define targeted configs ```shell .circleci/config.yml .circleci/terraform.yml # targets changes under the 'terraform/' directory .circleci/scripts.yml # targets changes under the 'scripts/' directory .circleci/src.yml # targets changes under the 'src/' directory terraform/ scripts/ src/ ``` The `dynamic/continue` workflow would look like ```yaml setup: true orbs: dynamic: bjd2385/dynamic-continuation@ workflows: on-commit: jobs: - dynamic/continue: context: circleci modules: | /terraform /scripts /src ``` Once again, the workflows will only execute if any code changes are introduced to the containing "module". For example: if no changes are made on your branch within the `terraform/` directory, the `.circleci/terraform.yml` CI config will not be executed.

Nested directories

Show Let's build off of the directory layout above, but add some environments. ```shell .circleci/config.yml terraform/ development/ production/ staging/ scripts/ src/ pkg1/ pkg2/ ``` We may write further targeted configs ```shell .circleci/config.yml .circleci/terraform.development.yml # target specific changes under 'terraform/development/' .circleci/terraform.production.yml # target specific changes under 'terraform/production/' .circleci/terraform.staging.yml # target specific changes under 'terraform/staging/' .circleci/scripts.yml .circleci/src.pkg1.yml # target specific changes under 'src/pkg1/' .circleci/src.pkg2.yml # target specific changes under 'src/pkg2/' ``` with a corresponding `dynamic/continue` workflow in our standard `config.yml` ```yaml setup: true orbs: dynamic: bjd2385/dynamic-continuation@ workflows: on-commit: jobs: - dynamic/continue: context: circleci modules: | /terraform/development /terraform/production /terraform/staging /scripts /src/pkg1 /src/pkg2 ``` Note that the filenames denote additional directory structure with dots `.`, whereas our modules may contain dots `.` or slashes `/`. Thus, the following list of modules is also valid, albeit potentially harder to follow. ```yaml modules: | terraform.development terraform.production terraform.staging scripts src.pkg1 src.pkg2 ```

Filtering, ignoring and including file changes

Show At times, there may be files that change in modules that should _not_ cause workflows to run. These could include, as an example, updated markdown or README-like files. To solve this problem, the orb has the ability to read an optional `.gitignore`-like filter on each module, named `.circleci/.ignore`, to prevent detected changed files on your PR from enabling workflows.
Ignore file changes
Show Starting with the same directory layout as above, we could add `.gitignore`-like files ```shell .circleci/config.yml .circleci/terraform.yml .cirlceci/terraform.ignore # optionally ignore changes under 'terraform/' directory .circleci/scripts.yml .cirlceci/scripts.ignore # optionally ignore changes under 'scripts/' directory .circleci/src.yml .cirlceci/src.ignore # optionally ignore changes under 'src/' directory terraform/ scripts/ src/ ``` These files are automatically referenced, and do not need to be explicitly specified, with a job as ```yaml workflows: on-commit: jobs: - dynamic/continue: context: circleci # auto-detect: true ``` or, exactly the same as above.
Include additional file changes
Show We can create additional pipeline file change dependencies throughout the repo within the same `*.ignore`-files. For example, suppose we have a subdirectory `scripts/terraform/`, and we want changes to files under this subdirectory to enable the pipeline defined in `.circleci/terraform.yml`; we can add ```text !scripts/terraform/* ``` to `.circleci/terraform.ignore`, as described in the [Git documentation](https://git-scm.com/docs/gitignore#_pattern_format).

Specify an alternate workflow for your repository's root directory

Show It is possible to run a workflow targeting the root of a repository's directory structure, offering overlapping workflows and more flexibility on file changes when paired with the above strategies. We can accomplish this by specifying `.` or `/` as a module. For example, ```yaml workflows: on-commit: jobs: - dynamic/continue: context: circleci auto-detect: true root-config: app # Defaults to 'app.yml' and 'app.ignore' under .circleci/, should the orb detect a '.'- or '/'-root module ``` Note that this requires you define an `app.yml` (though this root config's name is configurable), at minimum, under `.circleci/`, for the orb to process.

Config validation with pre-commit

Standard CircleCI config validation pre-commit hooks will only validate the default config at .circleci/config.yml. Please use this project's pre-commit hook to validate any additional configs you've created.

Append the following to your .pre-commit-config.yaml -

- repo: https://github.com/bjd2385/dynamic-continuation-orb
    rev: v<version>
    hooks:
      - id: circleci-config-validate

You must have the circleci CLI installed.

Development

This orb has been developed in unpacked form. You may view its packed source with

yarn orb:pack  # creates a file 'orb.yml'

Validate an orb definition with

yarn orb:validate

When you're done with development, you may clean up the packed source with

yarn orb:clean

pre-commit

This repository uses pre-commit to uphold certain code styling and standards. You may install the hooks listed in .pre-commit-config.yaml with

yarn install:pre-commit-hooks