CDK Express Pipeline

• Introduction
• Features
• How does it work?
• Deployment Order
• Selective Deployment
• Installation
  • TS
  • Python
• Usage
• Legacy Usage
• Builds System Templates/Examples
  • Local
  • GitHub Workflows
  • GitLab
  • Any other build system
• Demo Projects
• Docs

Introduction

CDK Express Pipelines is a library that allows you to define your pipelines in CDK native method. It is built on top of the AWS CDK and is an alternative to AWS CDK Pipelines that is build system agnostic.

Sponsors

Features

• Works on any system for example your local machine, GitHub, GitLab, etc.
• Uses the cdk deploy command to deploy your stacks
• It's fast. Make use of concurrent/parallel Stack deployments
• Stages and Waves are plain classes, not constructs, they do not change nested Construct IDs (like CDK Pipelines)
• Supports TS and Python CDK

How does it work?

This library makes use of the fact that the CDK CLI computes the dependency graph of your stacks and deploys them in the correct order. It creates the correct dependency graph between Waves, Stages and Stacks with the help of the native .addDependency method of the CDK Stack. The cdk deploy '**' command will deploy all stacks in the correct order.

Deployment Order

The Wave, Stage and Stack order is as follows:

• Waves are deployed sequentially, one after the other.
• Stages within a Wave are deployed in parallel.
• Stacks within a Stage are deployed in order of stack dependencies within a Stage.

For example, the following definition of Waves, Stages and Stacks as in CDK Express Pipelines:

Will create a dependency graph as follows:

When used with cdk deploy '**' --concurrency 10, it will deploy all stacks in parallel, 10 at a time, where possible while still adhering to the dependency graph. Stacks will be deployed in the following order:

Selective Deployment

Leverages a consistent and predictable naming convention for Stack IDs. A Stack ID consists of the Wave, Stage and original Stack ID. This enables us to target Waves, Stages or individual stacks for deployment. For example, given the following stack IDs:

Wave1_Stage1_StackA
Wave1_Stage1_StackB
Wave1_Stage1_StackC
Wave1_Stage2_StackD

Wave2_Stage1_StackE
Wave2_Stage1_StackF

It makes targeted deployments easy:

• Deploy Wave1: cdk deploy 'Wave1_*' deploys all stacks in Wave1
• Deploy Wave1 Stage1: cdk deploy 'Wave1_Stage1_*' deploys all stacks in Wave1_Stage1
• Deploy Wave1 Stage1 StackA: cdk deploy 'Wave1_Stage1_StackA' deploys only Wave1_Stage1_StackA

[!IMPORTANT] When targeting specific stacks be sure to pass the --exclusively flag to the cdk deploy command to only deploy the specified stacks and not its dependencies.

Benefits of selecting a specific Wave, Stage or Stack over the all '**' method:

• While developing, you can speed up deployments from your local machine by deploying only what you are working on.
• When deploying with a CI/CD system, you can have additional logic between them. For example, you can place a manual approval step between Wave1 and Wave2.

Installation

TS

npm install cdk-express-pipelines

Then import the library in your code:

import { CdkExpressPipeline } from 'cdk-express-pipelines';

Python

pip install cdk-express-pipelines

Then import the library in your code:

from cdk_express_pipelines import CdkExpressPipeline

Usage

The ExpressStack extends the cdk.Stack class and has a very similar signature, only taking an extra stage parameter. There are multiple ways to build your pipeline, it involves creating the Pipeline, adding Waves, Stages and Stacks to your Stages and then calling .synth() on the Pipeline. See the alternative expand sections for other methods.

Stack Definition:

  class StackA extends ExpressStack {
  constructor(scope: Construct, id: string, stage: ExpressStage, stackProps?: StackProps) {
    super(scope, id, stage, stackProps);

    new cdk.aws_sns.Topic(this, 'MyTopic');
    // ... more resources
  }
}

class StackB extends ExpressStack {
  //... similar to StackA
}

class StackC extends ExpressStack {
  //... similar to StackA
}

1️⃣ Pipeline Definition:

const app = new App();
const expressPipeline = new CdkExpressPipeline();

// === Wave 1 ===
const wave1 = expressPipeline.addWave('Wave1');
// --- Wave 1, Stage 1---
const wave1Stage1 = wave1.addStage('Stage1');

const stackA = new StackA(app, 'StackA', wave1Stage1);
const stackB = new StackB(app, 'StackB', wave1Stage1);
stackB.addExpressDependency(stackA);

// === Wave 2 ===
const wave2 = expressPipeline.addWave('Wave2');
// --- Wave 2, Stage 1---
const wave2Stage1 = wave2.addStage('Stage1');
new StackC(app, 'StackC', wave2Stage1);
expressPipeline.synth([
  wave1,
  wave2,
]);

The stack deployment order will be printed to the console when running cdk commands:

ORDER OF DEPLOYMENT
🌊 Waves  - Deployed sequentially
🔲 Stages - Deployed in parallel, all stages within a wave are deployed at the same time
📄 Stack  - Dependency driven, will be deployed after all its dependent stacks, denoted by ↳ below it, is deployed

🌊 Wave1
  🔲 Stage1
    📄 StackA (Wave1_Stage1_StackA)
    📄 StackB (Wave1_Stage1_StackB)
        ↳ StackA
🌊 Wave2
  🔲 Stage1
    📄 StackC (Wave2_Stage1_StackC)

Legacy Usage

The CdkExpressPipelineLegacy class can be used when you do not want/can not use the ExpressStack class and have to stick to the CDK Stack class.

[!WARNING] Always use non-legacy classes for greenfield projects. Only use the Legacy classes if you have no other choice.

The following features are not available when using the Legacy classes:

• Enforcing Wave, Stage and Stack names do not include the separator character.
• Enforcing that a Stack in Stage 1 can not depend on a Stack in Stage 2.
• Printing stack dependencies within a Stage. Since we do not know what stage a stack belongs to, it's not possible to print the dependencies of stacks of only that stage and not others.
• If a consistent naming convention has not been followed for Stacks, it might not be possible to target all stacks in a stage or a wave. Deployment will have to always target all stacks with "**".
• Stack ids are not changed and have to be unique across all stacks in the CDK app, whereas with the non-legacy classes, stack ids only have to be unique within a Wave.

Stack Definition:

class StackA extends cdk.Stack {
  constructor(scope: Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    new cdk.aws_sns.Topic(this, 'MyTopicA');
    // ... more resources
  }
}

class StackB extends cdk.Stack {
  // ... similar to StackA
}

class StackC extends cdk.Stack {

}

1️⃣ Pipeline Definition:

const app = new App();
const expressPipeline = new CdkExpressPipelineLegacy();

/* === Wave 1 === */
/* --- Wave 1, Stage 1--- */
const stackA = new StackA(app, 'StackA');
const stackB = new StackB(app, 'StackB');
stackB.addDependency(stackA);

// === Wave 2 ===
/* --- Wave 2, Stage 1--- */
const stackC = new StackC(app, 'StackC');

expressPipeline.synth([
  {
    id: 'Wave1',
    stages: [{
      id: 'Stage1',
      stacks: [
        stackA,
        stackB,
      ],
    }],
  },
  {
    id: 'Wave2',
    stages: [{
      id: 'Stage1',
      stacks: [
        stackC,
      ],
    }],
  },
]);

The stack deployment order will be printed to the console when running cdk commands:

ORDER OF DEPLOYMENT
🌊 Waves  - Deployed sequentially
🔲 Stages - Deployed in parallel, all stages within a wave are deployed at the same time
📄 Stack  - Dependency driven

🌊 Wave1
  🔲 Stage1
    📄 StackA
    📄 StackB
🌊 Wave2
  🔲 Stage1
    📄 StackC

Builds System Templates/Examples

Local

These examples all assume a project created with the default structure of the CDK CLI command cdk init app --language typescript.

These example are taken from the demo TS project: https://github.com/rehanvdm/cdk-express-pipeline-demo-ts

Diff commands

# Diffs all stacks
cdk diff '**' --profile YOUR_PROFILE
# Diffs only specific stacks in a Wave
cdk diff 'Wave1_*' --profile YOUR_PROFILE --exclusively
# Diffs only specific stacks of a Stage in a Wave
cdk diff 'Wave1_Stage1_*' --profile YOUR_PROFILE --exclusively
# Diffs only a specific stack
cdk diff 'Wave1_Stage1_StackA' --profile YOUR_PROFILE --exclusively

Deploy commands

# Deploys all stacks in correct order
cdk deploy '**' --profile YOUR_PROFILE --concurrency 10 --require-approval never
# Deploys only specific stacks in a Wave in correct order
cdk deploy 'Wave1_*' --profile YOUR_PROFILE --exclusively --concurrency 10 --require-approval never
# Deploys only specific stacks of a Stage in a Wave in correct order
cdk deploy 'Wave1_Stage1_*' --profile YOUR_PROFILE --exclusively --concurrency 10 --require-approval never
# Deploys only a specific stack
cdk deploy 'Wave1_Stage1_StackA' --profile YOUR_PROFILE --exclusively --concurrency 10 --require-approval never

GitHub Workflows

These examples all assume a project created with the default structure of the CDK CLI command cdk init app --language typescript.

These example are taken from the demo TS project: https://github.com/rehanvdm/cdk-express-pipeline-demo-ts

GitLab

TODO...

Any other build system

...

Demo Projects

• CDK Express Pipeline Demo TS
• CDK Express Pipeline Demo Python

Docs

• FAQ
• Projen Sacrifices
• API