Adding outputs from nested forwarder stack

lmpardey commented 3 years ago

What does this PR do?

This PR adds an output to the CloudFormation template for the Datadog integration. The output contains the ARN of the SecretsManager secret (if created) which holds the Datadog API key and the ARN of the forwarder Lambda function. Both of these resources are created by the nested Forwarder stack, therefore this PR depends on the release of https://github.com/DataDog/datadog-serverless-functions/pull/427.

Motivation

The SecretsManager and Lambda resources created by CloudFormation have randomized suffixes in their names. These random characters make it hard to integrate with other automation tools which require the exact ARN of the resources.

Exporting the ARN allows integrations with other tools, such as Terraform, which require the secret ARN but only have access to the outputs of a stack. The nested stacks export the ARNs of these outputs, so this PR is required to propagate those outputs to the parent stack. The nested stacks have randomized names themselves therefore it is not sufficient to take the outputs from there using other tools.

Testing Guidelines

Note: testing was carried out inside the aws directory.

Render the template with a value for the S3 bucket containing the nested templates

cp main.yaml main.yaml.bak
sed -i "s/<BUCKET_PLACEHOLDER>/datadog-cloudformation-template/g" main.yaml

Validate the syntax of the new template using the AWS command-line tool
```
aws cloudformation validate-template --template-body file://main.yaml
```
Run the testing scenarios in the sections below
Restore the bucket placeholders to the template file
```
mv main.yaml.bak main.yaml
```

Update Existing Stack

Locate the name of the existing Datadog integration stack

$ aws cloudformation describe-stacks --query "Stacks[*].StackName"
[
    "datadog"
    "datadog-ForwarderStack-AAABBBCCCDDD",
    "datadog-DatadogIntegrationRoleStack-AAABBBCCCDDD",
    "datadog-DatadogPolicyMacroStack-AAABBBCCCDDD",
]

Update the main stack using the new template file and keeping the original values of the required variables. This action depends on the deployment of https://github.com/DataDog/datadog-serverless-functions/pull/427 since the Forwarder stack must export certain outputs.

$ aws cloudformation update-stack --stack-name datadog --template-body file://main.yaml --parameters ParameterKey=DdApiKey,UsePreviousValue=true ParameterKey=ExternalId,UsePreviousValue=true
{
    "StackId": "arn:aws:cloudformation:eu-west-1:XXXXXXXXXXXX:stack/datadog/aaaabbbb-1111-2222-3333-0000ffff9999"
}

When the stack is updated, validate that the outputs contain the Lambda function ARN and the Api Secret ARN. Validate that the outputs

$ aws cloudformation describe-stacks --stack-name datadog --query "Stacks[*].Outputs[]"

[
    {
        "OutputKey": "DatadogForwarderArn",
        "OutputValue": "arn:aws:lambda:eu-west-1:XXXXXXXXXXXX:function:datadog-ForwarderStack-AAABBBCCCDDD-Forwarder-111222333444",
        "Description": "Datadog Forwarder Lambda Function ARN",
        "ExportName": "datadog-DatadogForwarderArn"
    },
    {
        "OutputKey": "DdApiKeySecretArn",
        "OutputValue": "arn:aws:secretsmanager:eu-west-1:XXXXXXXXXXXX:secret:DdApiKeySecret-AAABBBCCCDDD-YYYYYY",
        "Description": "ARN of SecretsManager Secret with Datadog API Key",
        "ExportName": "datadog-ApiKeySecretArn"
    }
]

Create New Stack

Create a new stack with the new (rendered) template, using appropriate values for the required variables and capabilities:

$ aws cloudformation create-stack --stack-name datadog --template-body file://main.yaml --parameters ParameterKey=DdApiKey,ParameterValue=DummyApiKey ParameterKey=ExternalId,ParameterValue=DummyExternalId --capabilities "CAPABILITY_AUTO_EXPAND" "CAPABILITY_NAMED_IAM"
{
    "StackId": "arn:aws:cloudformation:eu-west-1:XXXXXXXXXXXX:stack/datadog/aaaabbbb-1111-2222-3333-0000ffff9999"
}

When the stack is created, validate that the outputs contain the Lambda function ARN and the Api Secret ARN.

$ aws cloudformation describe-stacks --stack-name datadog --query "Stacks[*].Outputs[]"
[
    {
        "OutputKey": "DatadogForwarderArn",
        "OutputValue": "arn:aws:lambda:eu-west-1:XXXXXXXXXXXX:function:datadog-ForwarderStack-AAABBBCCCDDD-Forwarder-111222333444",
        "Description": "Datadog Forwarder Lambda Function ARN",
        "ExportName": "datadog-DatadogForwarderArn"
    },
    {
        "OutputKey": "DdApiKeySecretArn",
        "OutputValue": "arn:aws:secretsmanager:eu-west-1:XXXXXXXXXXXX:secret:DdApiKeySecret-AAABBBCCCDDD-YYYYYY",
        "Description": "ARN of SecretsManager Secret with Datadog API Key",
        "ExportName": "datadog-ApiKeySecretArn"
    }
]

Additional Notes

Depends on https://github.com/DataDog/datadog-serverless-functions/pull/427

winnieoursbrun commented 3 years ago

Love this ! I would use the main stack if this PR is accepted ! I will have a clearer terraform code with that !

tianchu commented 3 years ago

@lmpardey thanks for your contribution, again! Really sorry for the delay, we need to improve tracking of PRs.

DataDog / cloudformation-template