(pipelines): view `cdk diff` output in CodePipeline

[blimmer]

It would be awesome to be able to see the results of cdk diff in the CodePipeline UI along with a manual approval step before infrastructure changes are deployed by CDK Pipelines.

Use Case

CDK Pipelines are great. I love that there's an officially supported way to deploy CDK from a CI-environment vs. people's local machines. In fact, I created a system for a client that does this with Circle CI before pipelines were available. Eventually, it would be great to move clients over to the officially supported pipelines module from the custom solution.

The one thing that's missing from the current solution, IMO, is the ability to view the cdk diff output in CodePipeline before mutating infrastructure.

Imagine that I'm a developer working on a CDK-managed RDS cluster. Some CloudFormation properties, when updated, require replacement of the Database instance and cannot be performed in-place. Imagine a developer gets their PR approved and merges without running a cdk diff locally, first. Suddenly, they could be in for a big surprise when the RDS cluster is being recreated by CloudFormation.

This situation could be prevented if the developer would have checked out the cdk diff before mutating the infrastructure. The red REQUIRES REPLACEMENT message would have tipped them off that they should look into other ways of accomplishing their update.

This is just one example. It's always a good idea to check out the cdk diff before updating your infrastructure.

Proposed Solution

One way to accomplish this would be to create a CodePipeline stage for running cdk diff. It could use CodeBuild to execute cdk diff, just like it does to run cdk synth. Then, a manual approval step could optionally be added, linking to the CodeBuild output with the cdk diff. If the diff looks OK, the user approves and the applicationStage is deployed. If something looks wrong, the user could reject and fix the problem.

Other

The CodePipeline would look something like this:

Source
|
Build
|
Diff
|-- optional manual approval
|
Update Pipeline
|
Application Stage

• [x] :wave: I may be able to implement this feature request
• [ ] :warning: This feature might incur a breaking change

---

This is a :rocket: Feature Request

[rix0rrr]

We totally want to do something like this eventually. The problem is, where do we put this diff, and what are we diffing? These are not easy things to answer and any answer we pick will leave some folks unhappy.

• Keep in mind that a pipeline can have N stages in them (could be 1, 2, 5, 20).
• Keep in mind that a pipeline can have multiple changes going through it at once.

So:

Do we put an approval gate before every stage? Most accurate but kinda defeats the purpose of CI/CD. People WILL forget to hit that button, your pipeline WILL stall.

Do we put a single approval gate at the start? But then what are we diffing against what? Just the first stage? All stages? If all stages, what about changes that are still percolating through the pipeline, the diff will look wrong. What if looking up a diff for a single stage fails (for whatever reason). Maybe it's the last stage in your pipeline of 20. Are we now not going to deploy anywhere?

And then there's the question of how we're going to publish a UI for this. CodePipeline doesn't really give us a way to host a good UI so we're going to have to do that somewhere else, maybe a serverless website. Now we need to do auth to that website. What are we going to auth with?

---

All this is to say, it's definitely on the radar but there are some questions that need answering here, and they're all pretty tough.

If you have any clever ideas, I'd love to hear them!

[simon-dk]

@blimmer @rix0rrr Stumpled upon this request today as I have been working with pipelines as well. Maybe I am a bit daft, but it does not seem possible to run "cdk diff" locally either, or is there something I am completely missing?

Cdk diff only shows me the diff for the pipeline, not for each stack (if you know a way to show the diffs for each stack, please let me know!), but I agree with @blimmer that a way to add a "diff stage" in pipelines would be very useful for teams.

[blimmer]

@Simon-SDK - to see the diffs for your stages, you have to run something like:

> cdk diff -a cdk.out/assembly-MyPipeline-MyStage

It'll be different based on the names in your project. This is also reported in #8676, which is marked for GA.

[mrpackethead]

Just adding my 5c to this problem. Its a wee bit of a stumbling block for one of my clients, who woudl really like to see whats going to change. before they commit to it.

cdk diff provides the information thats needed.

I have a couple of use cases.. The simple one is what i'd call an 'infrastructure' change... An example being that I'd typically deploy a VPC via cdk for an account.. It doesnt have a 'dev/test/prod' type set up.. So in effect its only one stage, but my stages could well have multiple stacks across accounts.

A second use case is deploying some kind of applicaiton that may have a dev/test/prod.. Those have multiple stages.. and and a few different ways of being progressed through the pipeline.. Some will have manual approvals, and some will need to pass some automated testing..

I agree with @rix0rrr comment above,.. 'These are not easy things to answer and any answer we pick will leave some folks unhappy.'

My own needs require several different approaches, depending on what i'm doing.. Whatever the approach is, its got to be very flexible to cover all the use-cases....

The first problem to resolve might just be a way to to create 'external' ( to the pipeline ) approvals.. something to replace the manual approval.

[mrpackethead]

Did some more thinking since the last comment. :-) ( and sleeping zzzzzz ). It probably helps. Firstly i'm not sure that this is a problem that can be universally solved, for everyone with a L3 type construct. I think we've learned that with ec2.VPC, that there are times that such highly opinionated constructs actually lead to more problems than they solve. ( thats not a criticism, thats just learning we are all making. ).

I'm going to have a go and trying to resolve this for my own situation..

I've been using Manual Approvals for some time. However what i did'nt really appreicate is that 'manual' approvals dont' need to be 'manual' at all. ( and that leads to the uestion why are they called 'Manual Approvals'.. should they just be 'Approval' ? ).

Currently i'm doing this, where i need to ( sorry I am a python guy, but hopefully its reasonably clear to the 74% of CDK users who are not )

this_stage.add_manual_approval_action(
    action_name = 'Release_this_Stage',
    run_order = 1
)

This just adds a simple 'push the button'.. While it might stop an 'accidental' deployment if you accepted a PR that you really did'tn mean to, it does not provide any useful info..

So, I'm thinking i need some smarts. It turns out that theres a an 'add_actions' method for stages, which lets me add codepipeline actions. Lets use a 'ManualApprovalAction'.. This will send an SNS message..

this_stage.add_actions(
     codepipeline_actions.ManualApprovalAction(
         action_name = 'CreateDiffForApproval',
         notification_topic = creatediff    # this is an sns topic. 
) 

Receving the sns message can be a lambda.. The lambdas function essentially would be to

(a) Create a diff using cdk diff.. The result would likely be placed in an S3 bucket. (b) notify the approvers that a change is needing approval. This coudl be email/slack/teams what ever... They would provide a link to an API...

the API ( and a bit of web-based smarts or maybe it will just be CLI) ( which would need some authentication ) would display the diff, and then provide an approve button... The approve button triggers another lambda which then uses the PutApprovalResult API call ( codepipeline ) to 'approve' the stage and it can move on.

Depending on the need, you could set up differnet approval rules.. For example, if you are making some kind of changes that change some security senstive thigns, you might need a 'security team' approval.... This of course really gets very specific to your situation.

This will take a bit of experimentation, but it looks fesible..

Credit I got a bit of inspiration from this blog, which is not cdk specific, but describes the codepipeline "manualApproval' https://www.eliasbrange.dev/posts/aws-auto-approve-codepipeline/

[polothy]

The way I went about doing this is using GitHub actions because I wanted to see the CloudFormation changes at the PR, before merging.

See https://github.com/blackboard-innersource/gh-action-cdk-diff (use at your own risk, I'll freely modify this to suite our needs).

It's all bash scripting stuff, so you could easily do the same via CodeBuild on PR (or any other build system).

The tl;dr of the action:

• Run synth on the main branch.
• Run synth on the PR branch.
• Diff the two cdk.out directories.
• Post the diff to a comment on the PR (the comment uses GH collapsable sections and diff syntax highlighting).

I realize this isn't the same as doing cdk diff against the deployed CloudFormation stack, but it seems to do the trick for us.

[ahammond]

I think that having an option to "generate the changeset then wait for manual approval" would be Good Enough. Ideally, I'd want to set it at the wave level:

So I'd be able to write something like

const manualApprovalAfterCreateChangeSet = new pipelines.ManualApprovalStep('Approve Changeset', {...});
const myWave = myPipeline.addWave('MyWaveName', { manualApprovalAfterCreateChangeSet });
// add stages to wave

And... almost certainly this should be part of an interface which is common to wave and stage.

[ahammond]

@TheRealAmazonKendra how is this a p2? We have a ManualApproval step with only the coarsest visibility into what a user is approving. That's a pretty major missing feature for enterprise adoption and use.

[MrArnoldPalmer]

Hey @ahammond , p2 is relative of course and since this is a feature request and we have a lot of other issues including bugs, this is just an indicator that the CDK core team may not get to implementing this ourselves as part of our immediate work. We definitely agree that this is an important feature for CDK pipelines though so of course PRs are welcome in the interim.

[ahammond]

@MrArnoldPalmer totally see where you are coming at with this. Perhaps it's an argument for more staffing?

[ahammond]

Also, related, https://github.com/time-loop/clickup-projen has a cdk-diff solution that works at the GitHub PR level. That has mitigated a lot of the pain here.

[github-actions[bot]]

This issue has received a significant amount of attention so we are automatically upgrading its priority. A member of the community will see the re-prioritization and provide an update on the issue.

[jk2l]

if anyone want some example here is one (Python CDK)

SEE NEXT COMMENT

what i am having issues and trying to overcome is to have figure how to pass down URL parameter to approval step. and having CDK generate HTML diff for python (which I can't find a way). there is cdk-pretty-diff but it is base on nodejs and don't think it can integrate with Python CDK.

if i can do the two above, i would have the URL link under approval step point to html diff. atm i just view it via codebuild output log

[jk2l]

latest invention, this will generate html with color diff. the rest is you put it up to whatever S3 or any storage you can link with then you will complete with a pre approval diff view in same stage

        stage_name = f"{env_name}"
        account_id = self.node.try_get_context(f"account:{env_name}")
        diff_step = pipelines.CodeBuildStep(
            "Diff",
            input=input,
            install_commands=[
                "npm install -g aws-cdk",
                "pip install -r requirements.txt",
                "pip install ansi2html",
                "mkdir diff.out"
            ],
            commands=[
                f"cdk diff --ci {pipeline_name}-{env_name}/{stage_name}/* | tee diff.out/result.log",
                "cat diff.out/result.log | ansi2html > diff.out/result.html"
            ],
            env={
                "FORCE_COLOR": "1"
            },
            primary_output_directory="diff.out",
            role_policy_statements=[
                iam.PolicyStatement(
                    actions=["sts:AssumeRole"],
                    resources=[
                        f"arn:aws:iam::{account_id}:role/cdk-hnb659fds-lookup-role-{account_id}-ap-southeast-2"
                    ],
                    conditions={
                        "StringEquals": {
                            "iam:ResourceTag/aws-cdk:bootstrap-role": "lookup"
                        }
                    }
                )
            ]
        )
        approve_step = pipelines.ManualApprovalStep("Deploy Infrastructure")
        approve_step.add_step_dependency(diff_step)
        application_pipeline.add_stage(
            PipelineStage(
                self,
                stage_name,
                env=cdk.Environment(
                    account=self.node.try_get_context(f"account:{env_name}"),
                    region=self.node.try_get_context(f"region:{env_name}"),
                ),
                env_name=env_name,
            ),
            pre=[
                diff_step,
                approve_step
            ],
        )

[alexbaileymembr]

@Simon-SDK - to see the diffs for your stages, you have to run something like:

> cdk diff -a cdk.out/assembly-MyPipeline-MyStage

It'll be different based on the names in your project. This is also reported in #8676, which is marked for GA.

This solution seems relatively simple. I wonder if a 'step in the right direction' with CDK pipelines and providing a 'diff' functionality would simply be to add a code build step that ran this for a stage if the user wanted to.

For example:

myStage.addDiffApproval();

Which could:

• Run the above command cdk diff -a cdk.out/assembly-MyPipeline-MyStage
• Add an approval step for the stage to deploy.

There could also be a separate method like pipeline.addDiffApprovals() or a property for the pipeline synth step which defaults to false and allows diffs to be generated for all stages and an approval step to be created straight after the synth and diff is generated?

This might not solve every scenario and every context but seems like it would provide a more confidence when deploying through CDK pipelines.