Flexible pipeline stages

[thiezn]

Hi,

We are using ADF pipelines for almost all of our multi-account deployments but are increasingly finding scenario's where it would be beneficial to be able to define the various pipeline stages ourselves.

For instance, we would like to build a pipeline for managing containers in an ECR repository. We would like to roll out the ECR instance using CloudFormation and have a codebuild step next to it using the docker image that builds and uploads the image to the ECR repository.

Here's an example deployment_map showing what the interface could possibly look like:

pipelines:
  - name: myfreefullyflexiblepipeline
    type: cc-freeforall 
    params:
      - SourceAccountId: 342596870
      - NotificationEndpoint: hello@moorspots.com
      - RestartExecutionOnUpdate: true
    stages:
      - type: adf-codebuild
        # defaults to python container and buildspec for parsing params
      - type: codebuild
        # This can be anything you want to do yourself after the adf-codebuild
        image: docker
        buildspec: buildspec2.yml
        targets:
          - path: 123456789
            regions: eu-west-1
      - approval
      - type: cloudFormation
        # templatefile defaults to template.yml 
        # templateparams defaults to params/ folder
        targets:
          - path: 123456789
            regions: eu-west-1
      - type: CloudFormation
        templatefile: template2.yml
        templateparams: params2/
        targets:
          - path: 123456789
            regions: eu-west-1

I suspect it's going to be quite difficult to make a change like this without breaking backwards compatibility. Also, I read somewhere that you're thinking of switching out jinja2 in favour of AWS Cloud Development Kit (CDK) which might provide other options to accomplish this.

I'm not sure if I fully like my proposed solution yet as it can get a bit unwieldy. Perhaps there is a better way of accomplishing this.

Having flexibility like this should also resolve issues like #89

Cheers, Thiezn

[klontje85]

Hello Thiezn,

you can achieve this in the current version with defining your own pipeline type using the Jinja2 templates. As a base you could take the cc-cloudformation.yml.j2 in the pipeline_types folder as part of your deployment accounts CodeCommit repository. In the future this will change if we make the switch to CDK based CodePipeline generation.

Kind regards,

Koen