Support for YAML templates

[pplu]

CloudFormation accepts templates in JSON and YAML.

YAML templates make use of YAML tags to call functions:

AWSTemplateFormatVersion: '2010-09-09'
Resources:
  ClientSecurityGroup:
    Type: 'AWS::EC2::SecurityGroup'
    Properties:
      GroupDescription: !Ref 'AWS::StackName'
      VpcId: {'Fn::ImportValue': !Sub '${ParentVPCStack}-VPC'}

We should be able to generate the same object model from YAML than from JSON (they represent the same thing)

[pplu]

The only parser that has the potential for doing something with tags is YAML::PP. Although stated that tags are completely ignored (https://metacpan.org/pod/YAML::PP#Tags) there seems to be code for handling tags.

@perlpunk: Can you advise me on how I can wire up YAML::PP so I can run some custom routine when a tag is found? (or maybe point me to a YAML parser that supports tags... You're the absolute YAML queen!)

[perlpunk]

Although stated that tags are completely ignored

Oops, that's a leftover ;-)

Currently standard tags are handled, and you can define your own constructors for scalar tags, but it seems your use case is to dump YAML with tags from a data structure with objects. This is currently in the works and released as a developer release, but the inner API of it might change.

Here's a quick example of how you could use this with v0.10_002:

use YAML::PP;
my $data = {
    Properties => {
        GroupDescription => (bless {}, 'AWS::StackName'),
        VpcId => {
          'Fn::ImportValue' => (bless {}, 'ParentVPCStack' ),
        },
    },
};

#    Properties:
#      GroupDescription: !Ref 'AWS::StackName'
#      VpcId: {'Fn::ImportValue': !Sub '${ParentVPCStack}-VPC'}

my $yp = YAML::PP->new();

$yp->schema->add_representer(
    class_equals => 'AWS::StackName',
    code => sub {
        my ($representer, $node) = @_;
        # $node->{value} contains the object
        $node->{tag} = '!Ref';
        $node->{data} = 'AWS::StackName';
        return 1;
    },
);
$yp->schema->add_representer(
    class_equals => 'ParentVPCStack',
    code => sub {
        my ($representer, $node) = @_;
        $node->{tag} = '!Sub';
        $node->{data} = '${ParentVPCStack}-VPC';
        return 1;
    },
);

my $yaml = $yp->dump_string($data);

[pplu]

Currently standard tags are handled, and you can define your own constructors for scalar tags, but it seems your use case is to dump YAML with tags from a data structure with objects.

I'm actually interested in being able to read the YAML with tags and covert that into the object model. I would like to "hook in" to the parse process when there is a tag, and insert some special value to the resulting hashref that gets returned.

This is currently in the works and released as a developer release, but the inner API of it might change.

I'll adapt if there is breakage :)

Here's a quick example of how you could use this with v0.10_002:

Thanks for the example @perlpunk! This covers the use case of converting the object model to YAML, which I'll play around with a bit to give you feedback. It would be great if you can point me in the right direction for the use case of reading a YAML with tags, hooking into the parse process to get the tags out. Is this possible?

[pplu]

Hi @perlpunk : Following your pointers, I've come up with this: https://github.com/pplu/cfn-perl/blob/feature/yaml_support/test_yaml. It generates the desired output:

---
Resources:
  R1:
    Properties:
      Groups:
      - Group1
      - Group2
      Path: !Ref AWS::StackName
    Type: AWS::IAM::User
  R2:
    Properties:
      Path: !Ref R1
    Type: AWS::IAM::User
  R3:
    Properties:
      Path: !GetAtt R1.Arn
    Type: AWS::IAM::User

Thinking about maintenance: I haven't found out how to control some aspects of the serialization process: There doesn't seem to be a way to do a "class_isa" on the representers. In my case, I can handle a lot of classes with one representer. I even have the case where the values will be from non-preknown subclasses, but with one common ancestor. From what I've navigated through YAML::PP, I think this feature would be implemented here https://metacpan.org/source/TINITA/YAML-PP-0.010_002/lib/YAML/PP/Representer.pm#L181. Are you interested in such a feature?

[pplu]

@perlpunk BTW: I've refactored the test_yaml script to use a class "catchall" (https://github.com/pplu/cfn-perl/blob/feature/yaml_support/test_yaml#L70), and I do the "isa" tests there, so what I was proposing is really not needed.

I'm liking a lot the way YAML::PP is written, so kudos to you :+1:

I'm very interested in getting from a YAML string with tags to a datastructure that has the information of the tags in some way or another. Can YAML::PP be wired up to do that?

[perlpunk]

Oh I see! :) Yeah I wasn't completely sure if you wanted to read or write YAML. Here is an example for creating objects when finding certain tags. Here I'm also still not sure if the API is good or if I might want to change it. Suggestions welcome!

use YAML::PP;

my $yaml = <<'EOM';
Properties:
      GroupDescription: !Ref 'AWS::StackName'
      VpcId: {'Fn::ImportValue': !Sub '${ParentVPCStack}-VPC'}
EOM

my $yp = YAML::PP->new();

$yp->schema->add_resolver(
    tag => '!Ref',
    # Implicit 0 means, this is only called if there is a tag
    implicit => 0,
    # so we can just use .* in the regex
    match => [ regex => qr{^(.*)$} => sub {
        my ($value) = @_;
        return bless { name => $value }, 'MyRef';
    } ],
);
$yp->schema->add_resolver(
    tag => '!Sub',
    match => [ regex => qr{^(.*)$} => sub {
        my ($value) = @_;
        return bless { name => $value }, 'MySub';
    } ],
    implicit => 0,
);

my $data = $yp->load_string($yaml);

[perlpunk]

The class_isa could be worth implementing, yeah. As you found out yourself (without documentation!) you can always use a catchall match. If you return 0 at the end, it will just try the next in the list.

[perlpunk]

I will probably change the resolver callback. Currently it is called with the string value, or, if a regex match is used, with the captures as a plain list. In the future I will pass a hashref with the complete node information (quoting style etc.)

[pplu]

Thanks @perlpunk for your help! I've already obtained significant advances, and have some more feedback:

1. I've created a Schema for all the serialization stuff: https://github.com/pplu/cfn-perl/blob/feature/yaml_support/test_yaml. I'm a bit worried that the Schema has to be in the YAML::PP::Schema namespace, so my CPAN dist would introduce YAML::PP::Schema::Cfn. Is that OK with you?

2. I've also started serializing the Object Model to YAML here: https://github.com/pplu/cfn-perl/blob/feature/yaml_support/t/301_yaml_parse.t. I'm finding that I'd like to match a tag on a "wildcard". I see I'll have to add each tag I want to support to as a resolver, and that's OK with me. The reason I want a match on a "wildcard" tag, is to take action on an unknown (unsupported) tag like #!Foo, since this is not part of the CloudFormation YAML spec, and I want to handle this situation (basically die on it). I've studied a bit the resolver code and there doesn't seem anything to support that. Are you up for a patch for that?

3. For the class_isa representer, the patch seems quite trivial to write, but I'd also like to contribute a test case. Any pointers as to where to put / write it?

4. Are you up for a bit of documentation on the resolver / representer side? Or do you consider it too internal still?

[perlpunk]

I'm a bit worried that the Schema has to be in the YAML::PP::Schema namespace

Currently it has, but I want to change that. I'm still thinking about to differentiate between schemas in the namespace and those which are not. I thought about the convention to add a single : at the beginning, e.g. :Cfn for modules that are in the schema namespace, and allow full package names for the others. Of course you can use that Cfn namespace, that doesn't sound like I will be using it, but maybe you can add a little patch to YAML::PP for now to support your own namespace.

I'm finding that I'd like to match a tag on a "wildcard".

That's on my TODO list ;-)

But that part is really subject to big changes. I hope I can continue working on that soon.

For the class_isa representer, ...

Yeah, I think this test would deserve a new testfile.

Documentation

I really need to write documentation, but I think it's still too early...

[pplu]

Hi @perlpunk ,

I've been working on the YAML support for this module. I'm able to set up resolvers (https://github.com/pplu/cfn-perl/blob/feature/yaml_support/lib/YAML/PP/Schema/Cfn.pm#L30) for catching !Base64: valueToEncode YAML constructs (thanks for the help :))

I haven't been able to get a resolver working for this type of construct: !Cidr [ "192.168.0.0/24", 6, 5 ] the resolver is never called. I've tried with regex => qr{^(.*)$} and with equal => '', but the resolver never fires. Can you help me get a resolver that fires on this type of construct? Note that in https://github.com/pplu/cfn-perl/blob/feature/yaml_support/t/300_yaml.t#L111 you have more examples of YAML constructs I'm trying to parse.

[perlpunk]

For sequences and mappings there are different methods add_sequence_resolver and add_mapping_resolver.

They are slightly more complicated. add_mapping_resolver is already working, but add_sequence_resolver is not working correctly yet. Thanks for your use case.

I pushed a fix to the branch sequence-resolver, but I would like to add a test. One important reason for being that complicated is that it is necessary if one wants to support cyclic references. The drawback is that it is also more complicated even if you don't need cyclic references.

The data is constructed in two steps. When the parser sees the start of a sequence or mapping, it calls the on_create handler which sets the initial data structure. At the end of the sequence or mapping, it calls the on_data handler, which fills the variable with the collected items (which both come in the form of an array reference, so no information gets lost).

If you checkout this branch, then your code could look like this:

$schema->add_sequence_resolver(
    tag => "!Cidr",
    on_create => sub {
        my ($constructor, $event) = @_;
        return { "!Cidr" => [] };
    },
    on_data => sub {
        my ($constructor, $ref, $items) = @_;
        push @{ $ref->{"!Cidr"} }, @$items;
    },
);

Then the result would be:

{
          '!Cidr' => [
                       '192.168.0.0/24',
                       6,
                       5
                     ]
        };

Feedback welcome =)

edit: this is (as a lot of this code) experimental. I might need to change the arguments of the callbacks, because certain things are not possible yet, for example creating strings instead of hashes or arrays).

[perlpunk]

I released YAML::PP 0.013_001

The example will now work like this:

$schema->add_sequence_resolver(
    tag => "!Cidr",
    on_create => sub {
        my ($constructor, $event) = @_;
        return { "!Cidr" => [] };
    },
    on_data => sub {
        my ($constructor, $ref, $items) = @_;
        push @{ $$ref->{"!Cidr"} }, @$items; # $ref is a reference to the actual data
    },
);

Because of CPAN infrastructure problems you can't see thew new release on MetaCPAN so far.

[pplu]

@perlpunk I've updated all the code in the YAML support branch of this project to use YAML::PP 0.015 and extended the range of support for all the tags that I want to support. We're down to solely one test case failing, and I think it has something to do with YAML::PP.

The test case failing is: https://github.com/pplu/cfn-perl/blob/feature/yaml_support/t/300_yaml.t#L354 Basically it looks like the !Condition resolver is not triggered, making the datastructure invalid. The curious thing is that the !Condition resolver is triggering in https://github.com/pplu/cfn-perl/blob/feature/yaml_support/t/300_yaml.t#L295.

Is this a bug in YAML::PP, or do I have to cope with some special condition / resolver?

[pplu]

@perlpunk is there a way to make YAML::PP die on unrecognized tags? I would like to be able to configure a resolver to act as a catchall, so Cfn can die when a tag that hasn't been handled yet is parsed (note that if it's not supported, but you are interested in the feature, with a couple of pointers, I can contribute it).

[pplu]

note to self: there are some in-the-wild YAML templates for Glue here: https://docs.aws.amazon.com/glue/latest/dg/populate-with-cloudformation-templates.html

[perlpunk]

@pplu I think the problem there is the colon:

!Or [!Equals [sg-mysggroup, !Ref ASecurityGroup], !Condition: SomeOtherCondition]

You probably want this instead:

!Or [!Equals [sg-mysggroup, !Ref ASecurityGroup], !Condition SomeOtherCondition]

Note that there have been some changes in the newest version for add_resolver. You can probably figure it out by looking at its usage in YAML::PP::Schema::Core, for example.

[pplu]

Hi @perlpunk,

Thanks for spotting the error! Basically the YAML bit was lifted off AWS documentation, so I was working on an invalid example :disappointed:. I now consider YAML support in Cfn good enough to release to CPAN.

Since I'm relying on still unstable YAML::PP API, there can be some breakage, but I take it as my sole responsibility to keep up with YAML::PP undocumented API changes. Please don't feel obliged to be backwards-compatible.

BTW: when you feel that the resolvers and representers API is stable enough, please drop a message, since I can chip in some documentation.

[pplu]

Hi @perlpunk just wanted to give you a heads up: YAML support has been published to CPAN in v0.06 :dancer: https://metacpan.org/pod/release/JLMARTIN/Cfn-0.06/lib/Cfn.pm#Contributions