Add aws ec2 command parameter resolution

[cornelf]

• [x] Ready for review.
• [x] I found it useful and necessary at the same time to add a parameters value resolution with boto3 ec2 call, specifically to retrieve the list of availability zones in a region. Why this help? us-east-1 has 4 availability zones, but AZs letters can be any 4 from (a, b, c, d, e), depending on AWS account. This is only one use of the new parameter value resolution feature, and I can envisage further usage and extension to such a feature.

[cornelf]

@acaire any chance this PR would be considered, or this kind of feature does not follow the direction in which you see cumulus growing?

[acaire]

Hi @cornelf, thanks for the PR - Just discussing this internally.

[cornelf]

:+1:

[acaire]

Hi @cornelf That's an interesting use case - We're doing something similar with s3 lookups that we haven't had a chance to merge to master yet.

We're hoping in the immediate future to do a cleanup and rewrite for boto3, improve caching and error handling/retry logic etc. We can certainly see the benefits of this functionality. The implementation however in it's current state it's perhaps a bit clunky for the user.

Rather than:

AZ1:
    index: 0
    var_name: AvailabilityZones
    command: describe-availability-zones

I'd sooner see something closer to:

AZ1:
    az_list: 0

It seems that a simple plugin architecture would be fitting here to keep this logic away from the core - Are you happy to keep this open for now and we can revisit this after the cleanup?

[Raniz85]

You can do this in your templates with Fn::GetAZs

Here's a snippet from our VPC template showing how to use it:

"PrivateSubnet1" : {
  "Type" : "AWS::EC2::Subnet",
  "Metadata" : {
    "Comment" : "Subnet for internal infrastructure in the first availability zone, shouldn't have a public IP address and will be routed through the NAT instance"
  },
  "Properties" : {
    "AvailabilityZone" : {"Fn::Select" : ["0", {"Fn::GetAZs" : {"Ref" : "AWS::Region"}}]},
    "VpcId" : { "Ref" : "SpiideoVpc"},
    "CidrBlock" : "10.0.128.0/19",
    "Tags" : [ {"Key" : "subnet:type", "Value" : "private" } ]
  }
},

"PrivateSubnet2" : {
  "Type" : "AWS::EC2::Subnet",
  "Metadata" : {
    "Comment" : "Subnet for internal infrastructure in the second availability zone, shouldn't have a public IP address and will be routed through the NAT instance"
  },
  "Properties" : {
    "AvailabilityZone" : {"Fn::Select" : ["1", {"Fn::GetAZs" : {"Ref" : "AWS::Region"}}]},
    "VpcId" : { "Ref" : "SpiideoVpc"},
    "CidrBlock" : "10.0.160.0/19",
    "Tags" : [ {"Key" : "subnet:type", "Value" : "private" } ]
  }
},

[cornelf]

@Raniz85 Javisst! I got the the hint from @acaire - I just left this one open by mistake, should have closed it at the time. However the feature extension idea may be useful.