Call Lambda via SNS

[spa-87]

Hi, I've faced with a problem of sending lambda calls via SNS to https://github.com/SungardAS/spotinst-lambda function. But as it's based on current framework, here is the correct place for that issue I think.

Lambda isn't available in all AWS regions and CloudFormation doesn't allow to assign resource from a different regions for ServiceToken parameter at custom resource. In that case SNS looks like a sensible solution as it available at all regions and has native integration with Lambda. But when message is sent via SNS, there's added SNS-relative metadata and the source message is located deeper in JSON.

This is how message looks like when it comes directly from CloudFormation to Lambda

{
  "StackId": "arn:aws:cloudformation:eu-west-1:******:stack/spotinst-test1/d******0-33b3-11e6-9d89-5**********e",
  "ResponseURL": "https://cloudformation-custom-resource-response-euwest1.s3-eu-west-1.amazonaws.com/arn%3Aaws%3Acloudformation%3Aeu-west-1%3A********%3Astack/spotinst-test1/d******0-33b3-11e6-9d89-5**********e%7CCreateSpotInstElasticGroup%7C*******a-0277-462f-8882-e51********eb?AWSAccessKeyId=*************&Expires=1466083032&Signature=G**********94%3D",
  "ResourceProperties": {
    "group": {
      "capacity": {
        "minimum": "0",
        "maximum": "2",
        "target": "0"
      },
      "compute": {
        "product": "Linux/UNIX (Amazon VPC)",
        "availabilityZones": [
          {
            "subnetId": "subnet-*****",
            "name": "eu-west-1a"
          },
          {
            "subnetId": "subnet-*****",
            "name": "eu-west-1b"
          },
          {
            "subnetId": "subnet-*****",
            "name": "eu-west-1c"
          }
        ],
        "instanceTypes": {
          "spot": [
            "m3.medium"
          ],
          "ondemand": "t2.nano"
        },
        "launchSpecification": {
          "monitoring": "false",
          "tags": [
            {
              "tagKey": "Name",
              "tagValue": "spotinst-test1-default"
            }
          ],
          "imageId": "ami-******",
          "securityGroupIds": [
            "sg-******"
          ],
          "keyPair": "*******",
          "blockDeviceMappings": [
            {
              "deviceName": "/dev/sda1",
              "ebs": {
                "volumeSize": "15",
                "deleteOnTermination": "true",
                "volumeType": "gp2"
              }
            }
          ],
          "loadBalancerNames": [
            "dev-spotinst-test"
          ],
          "healthCheckGracePeriod": "900",
          "healthCheckType": "ELB"
        }
      },
      "name": "spotinst-test1",
      "strategy": {
        "availabilityVsCost": "balanced",
        "drainingTimeout": "300",
        "onDemandCount": "1"
      }
    },
    "ServiceToken": "arn:aws:lambda:eu-west-1:*******:function:Bootstrap_SpotInst_Node",
    "accessToken": "111111"
  },
  "RequestType": "Create",
  "ServiceToken": "arn:aws:lambda:eu-west-1:*********:function:Bootstrap_SpotInst_Node",
  "ResourceType": "Custom::elasticgroup",
  "RequestId": "5********a-0277-462f-8882-e********b",
  "LogicalResourceId": "CreateSpotInstElasticGroup"
}

And this is the message delivered via SNS:

{
  "Records": [
    {
      "EventVersion": "1.0",
      "EventSubscriptionArn": "arn:aws:sns:eu-west-1:*********:SpotInst_CFN_Lambda:7***1-2b86-43cc-ab61-a****5",
      "EventSource": "aws:sns",
      "Sns": {
        "SignatureVersion": "1",
        "Timestamp": "2016-06-16T11:21:57.033Z",
        "Signature": "L*****w==",
        "SigningCertUrl": "https://sns.eu-west-1.amazonaws.com/SimpleNotificationService-bb75*****ee.pem",
        "MessageId": "b*****4-5605-5183-92c2-d********1",
        "Message": "{\"RequestType\":\"Create\",\"ServiceToken\":\"arn:aws:sns:eu-west-1:**********:SpotInst_CFN_Lambda\",\"ResponseURL\":\"https://cloudformation-custom-resource-response-euwest1.s3-eu-west-1.amazonaws.com/arn%3Aaws%3Acloudformation%3Aeu-west-1%3A*******%3Astack/spotinst-test2/8******0-33b4-11e6-b2fe-5*****2%7CCreateSpotInstElasticGroup%7C******-9ab0-4578-bfb6-5*******5?AWSAccessKeyId=A*********A&Expires=1466083316&Signature=2*******Q%3D\",\"StackId\":\"arn:aws:cloudformation:eu-west-1:*******:stack/spotinst-test2/8******0-33b4-11e6-b2fe-5******2\",\"RequestId\":\"7********5-9ab0-4578-bfb6-5********5\",\"LogicalResourceId\":\"CreateSpotInstElasticGroup\",\"ResourceType\":\"Custom::elasticgroup\",\"ResourceProperties\":{\"ServiceToken\":\"arn:aws:sns:eu-west-1:*********:SpotInst_CFN_Lambda\",\"accessToken\":\"1111\",\"group\":{\"compute\":{\"product\":\"Linux/UNIX (Amazon VPC)\",\"instanceTypes\":{\"spot\":[\"m3.medium\"],\"ondemand\":\"t2.nano\"},\"launchSpecification\":{\"healthCheckType\":\"ELB\",\"imageId\":\"ami-*******\",\"loadBalancerNames\":[\"testtesttest\"],\"blockDeviceMappings\":[{\"ebs\":{\"volumeType\":\"gp2\",\"deleteOnTermination\":\"true\",\"volumeSize\":\"15\"},\"deviceName\":\"/dev/sda1\"}],\"healthCheckGracePeriod\":\"900\",\"securityGroupIds\":[\"sg-*****\"],\"keyPair\":\"****\",\"monitoring\":\"false\",\"tags\":[{\"tagValue\":\"spotinst-test2-default\",\"tagKey\":\"Name\"}]},\"availabilityZones\":[{\"name\":\"eu-west-1a\",\"subnetId\":\"subnet-*******\"},{\"name\":\"eu-west-1b\",\"subnetId\":\"subnet-*******\"},{\"name\":\"eu-west-1c\",\"subnetId\":\"subnet-*******\"}]},\"name\":\"spotinst-test2\",\"strategy\":{\"onDemandCount\":\"1\",\"availabilityVsCost\":\"balanced\",\"drainingTimeout\":\"300\"},\"capacity\":{\"maximum\":\"2\",\"minimum\":\"0\",\"target\":\"0\"}}}}",
        "MessageAttributes": {},
        "Type": "Notification",
        "UnsubscribeUrl": "https://sns.eu-west-1.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:eu-west-1:*******:SpotInst_CFN_Lambda:7*******1-2b86-43cc-ab61-a*******5",
        "TopicArn": "arn:aws:sns:eu-west-1:*******:SpotInst_CFN_Lambda",
        "Subject": "AWS CloudFormation custom resource request"
      }
    }
  ]
}

So we can't use in all AWS regions CloudFormation templates which call Lambda functions based on current framework.

[kmcgrath]

Thank you for creating the ticket. This is the correct place for this issue.

Just to restate the request:

To make lambda formation compatible with regions that do not support Lambda, lambda-formation should be able to accept messages from SNS.

lambda-formation should:

1. Accept messages from SNS
2. Run the correct ResourceType and Action
3. Respond to the SNS callback accordingly

This would add SNS to all downstream projects, such as spotinst-lambda.

[kmcgrath]

Added to release 0.1.6

[spa-87]

@kmcgrath thanks a lot for a really fast fix! When do you plan to release spotinst-lambda based on the new version of the framework? Should I create an issue for that in the corresponding repo?

[kmcgrath]

How are you running spotinst-lambda? This is a patch update so if you have checked out the project from GitHub then you should be able to do a npm update and you will be ready to go. Just use the SNS topic ARN as the ServiceToken.

If you are using the links from particles-spotinst-lambda then I do need to push a new version of that out to s3.

[spa-87]

@kmcgrath thank you! Sorry for a stupid questions, I though lambda-formation is embedded into spotinst-lambda :) I've successfully rebuild spotinst-lambda with new version of framework and it works excellent. CFN stack with spotinst elastic group was successfully created, updated an removed in Sao Paulo where Lambda is unavailable.

[kmcgrath]

@spa-87 no stupid questions. This project, lambda-formation, is a building block to make anything an AWS CustomResource. Spotinst was the first. We have a few projects in the works to add functionality to AWS KMS, MySQL, Rancher and other tools that don't have full CloudFormation support. All will use lambda-formation as a dependency.

Thank you for submitting the very useful feature request.