feature: support a template engine

[zaro0508]

For many simple use cases where we want to dynamically generate cloudformation templates we prefer to generate them using a template engine (we use jinja[1]). We can use a full programming language to translate to CFN (i.e. CDK or trophosphere) however templating is enough to get the job done in most of our use cases. It would be great if org-formation can integrate with a templating engine. I think something like nunjucks[2] would work really well. Here's a nice article comparing a few javascript templating engines: https://strongloop.com/strongblog/compare-javascript-templates-jade-mustache-dust/

here's an example: I want to create one security group that contains multiple ingress ports.

securitygroup.yaml:

  InstanceSecurityGroup:
    Type: 'AWS::EC2::SecurityGroup'
    Properties:
      VpcId: 'vpc-1234ABC'
      SecurityGroupIngress:
      {% for port in IngressPorts %}
        - CidrIp: "0.0.0.0/0"
          FromPort: {{ port }}
          ToPort: {{ port }}
          IpProtocol: tcp
      {% endfor %}

passed in parameter:

IngressPorts:
  - 22
  - 80

result:

  InstanceSecurityGroup:
    Type: 'AWS::EC2::SecurityGroup'
    Properties:
      VpcId: 'vpc-1234ABC'
      SecurityGroupIngress:
        - CidrIp: "0.0.0.0/0"
          FromPort: 22
          ToPort: 22
          IpProtocol: tcp
        - CidrIp: "0.0.0.0/0"
          FromPort: 80
          ToPort: 80
          IpProtocol: tcp

[1] https://jinja.palletsprojects.com/en/2.11.x/ [2] https://mozilla.github.io/nunjucks/

[OlafConijn]

i am trying to figure out how this would work....

... would this be an additional attribute of update-stacks task? something like TemplateTransform: { Data: xs:any } or would this be an additional task type?

would certain data be available by default?

[eduardomourar]

I would be in favor of Jinja because the AWS Proton team already is using it: https://docs.aws.amazon.com/proton/latest/adminguide/svc-pipeline.html

Or even better use CDK when we have everything cloudformation natively: https://github.com/aws/aws-proton-public-roadmap/issues/21

[zaro0508]

@eduardomourar jinja would be great! I only suggested nunjucks because it's javascript based. Jinja is written in python so I just wasn't sure how well it can be integrated.

@OlafConijn i could see it working as a new org-formation type (i.e. update-templatized-stack) with a special parameter (i.e TemplatizedData) to allow passing in variables. for example..

org-formation-tasks.yaml

AppSecurityGroup:
  Type: update-templatized-stack
  Template: ./securitygroup.yaml
  StackName: app-security-group
  TemplatizedData:
     IngressPorts:
       - 22
       - 80  
  MaxConcurrentStacks: 10
  DefaultOrganizationBinding:
    IncludeMasterAccount: true
    Account: '*'
    Region: us-east-1

securitygroup.yaml:

  InstanceSecurityGroup:
    Type: 'AWS::EC2::SecurityGroup'
    Properties:
      VpcId: 'vpc-1234ABC'
      SecurityGroupIngress:
      {% for port in TemplatizedData.IngressPorts %}
        - CidrIp: "0.0.0.0/0"
          FromPort: {{ port }}
          ToPort: {{ port }}
          IpProtocol: tcp
      {% endfor %}

[OlafConijn]

@zaro0508 i would think about this as something we could add to update-stacks as opposed to creating a new type. current update-stacks tasks could then start to use templating by adding templating instructions from a certain version onwards.

is there a specific reason you suggest introducing a different type?

[zaro0508]

no specific reason @OlafConijn. using update-stacks to templatize would be fine as well, although I think that would lock org-formation into one specific template engine. The only advantage to creating a new type is to support multiple engines and maybe have some code separation? But really, i'm not sure we ever really need to support multiple template engines as long as we choose one that supports logic.

[zaro0508]

@eduardomourar nunjucks is based on jinja and is kinda of compatible with jinja

[OlafConijn]

my suggestion for implementation would be to:

1) add a this capability to the existing update-stacks task type. 2) add an attribute TextTemplatingParameters to the task type. 3) if TextTemplatingParameters is undefined not do anything 4) if TextTemplatingParameters is defined use this object to run through the text templater. 5) when passing down the TextTemplatingParameters from UpdateStacksBuildTaskProvider all the way down to cfn-binder you can take a look at how parameters are passed down. 6) i would start out not allows these TextTemplatingParameters to be passed as CLI arguments, that adds another thing to worry about. lets first make sure it works with tasks. 7) i would worry about Validate and Print commands. possibly write integration tests for those (ping me to get started on this!) 8) i would implement the call to the templater in the cfn-binder after cfnTemplate.createTemplateBodyAndResolve (and before this.calculateHash). at this point you have the entire template for that target.

Q on point 8: maybe templating should happen cfn functions are processed? maybe immediately when loading the file? open to that too. Q on point 2: please help me choose a better name here :).