[Request]: Support for using existing CloudFormation templates and CFN parameters in Proton templates

[clareliguori]

Currently, Proton injects values into a CloudFormation template using Jinja. For example:

Resources:
  TaskDefinition:
    Type: AWS::ECS::TaskDefinition
    Properties:
      Cpu: {{service_instance.cpu}}
      Memory: {{service_instance.memory}}
      ExecutionRoleArn: '{{environment.ECSTaskExecutionRoleArnOutput}}'
...

For organizations with existing CloudFormation templates that already use Parameters to inject values at deploy time, these templates must currently be updated to inject values using Jinja in order to use the templates with Proton, instead of using CFN Parameters. This request proposes adding support to Proton for using existing CloudFormation templates in Proton templates without modification, with Proton providing the parameter values for the CFN stack at deploy time. For example:

Parameters:
  ContainerCpu:
    Type: Number
  ContainerMemory:
    Type: Number
  TaskExecutionRoleArn:
    Type: String
Resources:
  TaskDefinition:
    Type: AWS::ECS::TaskDefinition
    Properties:
      Cpu: !Ref 'ContainerCpu'
      Memory: !Ref 'ContainerMemory'
      ExecutionRoleArn: !Ref 'TaskExecutionRoleArn'
...

[clareliguori]

One challenge I'll call out here (and would love feedback on!) is how to differentiate between the variety of sources for the values that Proton can inject into the template and how they should be mapped to CFN parameters in the template. With Jinja, Proton currently namespaces the Jinja variables in order to differentiate between values coming from two different sources. From the Proton documentation:

• Inputs or outputs that the Proton environment provides are given the environment namespace (for example, {{environment.env-input}}).
• Inputs or outputs that the Proton service provides are given the service namespace (for example, {{service.svc-input}}).
• Inputs or outputs that the Proton service instances provide are given the service namespace (for example, {{service_instance.svc-input}}).
• Inputs or outputs that the Proton pipelines provide are given the pipeline namespace (for example, {{pipeline.svc-input}}).

In the example shown in the original post above, two values come from the service instance's spec (cpu and memory) and one value comes from the environment (ECSTaskExecutionRoleArnOutput). It seems that some kind of mapping file may need to be included in the template bundle to map between the full selection of Proton-injected values and the CFN parameter that Proton should set with that value.

[clareliguori]

Another area I'd love feedback on is translating potentially complex specifications into the relatively "flat" structure of CloudFormation parameters. We chose to initially support OpenAPI + Jinja for schema definition + injection in Proton so that specifications could be very expressive including complex objects, arrays, and dictionaries, which CFN parameters generally does not support. For example:

proton: ServiceSpec

pipeline:
  simple_pipeline_string_input: "hello"
  pipeline_array_of_objects_input:
    - name: "staging"
      description: "this is my staging stage"
    - name: "prod"
      description: "this is my prod stage"

instances:
  - name: "my-prod-instance"
    environment: "prod-env"
    spec:
      simple_service_instance_number_input: 256
      service_instance_dictionary_input:
        en: "Hello!"
        fr: "Bonjour!"

What would be preferred here? Maybe a separate mapping file using Jinja that translates from complex specs into flat CFN parameters? Or only supporting the input types supported by CFN parameters in the case of a template that injects values via CFN parameters?

[JohnPreston]

I am not sure of what the best scenario to make both things work together would be but I would imagine, given one has to upload bundles with templates inside etc. that a detection of the Parameters key in the template uploaded would be easy enough to do to understand that there are parameters native to CFN to consider, extract the metadata of these (via CFN validate-template possibly?).

Also speaking of metadata, templates that use CFN Metadata for the parameters to improve the UI experience for users, that should also be taken into account when presenting users in the Proton UI.

But so, can you confirm @clareliguori that we cannot in the current state of AWS Proton use existing CFN templates with parameters and expect it to work for us?

My 2cts as a long time user of CFN: yes CFN is not always as breezy as I would love it to be, but with CFN macros + custom resources + private registry + modules, I need none of the Jinja interpolation stuff going on. I understand, it could be far easier to delegate to Proton the rendering than making sure the Macro is registered + shared to multiple accounts, for example.

But this feels like AWS wants to have something that look "new and shiny" as opposed to improve usability of existing services. I build my own analogy to Proton with CFN templates and at the announcement of that service, I was so very excited about it. So, the lack of CFN Parameters (TBC by @clareliguori ) out of the box is a pretty big miss. And it feels like "Oh I know, people like how helm charts look, so let's put some of the same syntax, that should look cool" is what's happened with this direction.

Probably back to the previous topic (or might a feature request of its own): if in the future people can use CDK or else (ie. tf) can we then allow people to provide a lambda function or define their own rendering engine? That way people come with what they want to render templates from the bundle with whatever they had so long as it generates valid information (CFN templates + CFN Configuration files / parameters files || tf plans).

[clareliguori]

But so, can you confirm @clareliguori that we cannot in the current state of AWS Proton use existing CFN templates with parameters and expect it to work for us?

Correct @JohnPreston, Proton does not currently specify any parameter values on the CloudFormation stacks it manages for you. Any parameters specified in the CloudFormation template must have a default value to be used with Proton.

As you suggested, one potential path for enabling support for CFN parameters is for Proton to effectively use the parameters as a simple schema for the specification of the environment, service instance, or pipeline inputs. This option would enable you to provide "raw" CloudFormation templates, without needing any OpenAPI or Jinja configuration in the Proton template bundle. If you have existing CFN templates that already work for your organization, this would be a very easy way to get started with Proton. Using the example above:

** For the following CFN template for a service instance: **

Parameters:
  ContainerCpu:
    Type: Number
  ContainerMemory:
    Type: Number
  TaskExecutionRoleArn:
    Type: String

** The specification would look like this: **

proton: ServiceSpec
instances:
  - name: "my-prod-instance"
    environment: "prod-env"
    spec:
      ContainerCpu: 256
      ContainerMemory: 1024
      TaskExecutionRoleArn: "arn:aws:iam::131296546870:role/myEcsExecutionRole"

Based on what I described here, some of the current Proton template generation features (like environment value injection into service instance templates, injection of service instance values into pipeline templates, Jinja language features like loops, complex specification data types like arrays, etc) would not be available or would be supplemented by the CFN-native options (exports/imports, macros, custom resources, etc). I'd love to hear from folks if this path would meet your needs in Proton.

[clareliguori]

Another path (which I think you hinted at @JohnPreston) would be to support rendering the stack's parameter values separately from the CFN template. For example, if Proton supported something like a separate stack-parameters.jinja file in the template bundle to map from the various values Proton can inject to the parameters that should be set on the stack.

Using the example above:

** For the following CFN template for a service instance: **
Parameters:
  ContainerCpu:
    Type: Number
  ContainerMemory:
    Type: Number
  TaskExecutionRoleArn:
    Type: String

** A stack-parameters.jinja file maps from Proton namespaced values to the stack's parameters: **
(just an example format for this file)
  ContainerCpu: {{service_instance.cpu}}
  ContainerMemory: {{service_instance.memory}}
  ExecutionRoleArn: '{{environment.ECSTaskExecutionRoleArnOutput}}'

** And the specification would look like this: **
proton: ServiceSpec
instances:
  - name: "my-prod-instance"
    environment: "prod-env"
    spec:
      cpu: 256
      memory: 1024

Again, I'd love to hear from folks which of these paths would meet your needs in Proton!