lauren-themis
commented
2 years ago I ran into the same thing - and even the "quick start" integration template's InstallLambdaLogForwarder: true fails for me.
I hesitate to call this a workaround, but I did get it working. I pulled the latest.yaml, and removed the following lines:
ReservedConcurrentExecutions:
Ref: ReservedConcurrencyAnd then I set up the stack with the edited template body instead of the url.
So my terraform for the forwarder ended up as:
resource "aws_secretsmanager_secret" "dd_api_key" {
name = "datadog_api_key"
description = "Encrypted Datadog API Key"
}
resource "aws_secretsmanager_secret_version" "dd_api_key" {
secret_id = aws_secretsmanager_secret.dd_api_key.id
secret_string = var.DATADOG_API_KEY
}
resource "aws_cloudformation_stack" "datadog_forwarder" {
name = "datadog-forwarder"
capabilities = ["CAPABILITY_IAM", "CAPABILITY_NAMED_IAM", "CAPABILITY_AUTO_EXPAND"]
parameters = {
DdApiKeySecretArn = aws_secretsmanager_secret.dd_api_key.arn,
DdSite = "datadoghq.com",
FunctionName = "datadog-forwarder"
}
template_body = <<WHAT
AWSTemplateFormatVersion: "2010-09-09"
Description: Pushes logs, metrics and traces from AWS to Datadog.
Mappings:
Constants:
DdForwarder:
Version: 3.44.0
LayerVersion: 13
Parameters:
DdApiKey:
Type: String
NoEcho: true
Default: ""
Description: The Datadog API key, which can be found from the APIs page (/account/settings#api). It will be stored in AWS Secrets Manager securely. If DdApiKeySecretArn is also set, this value is ignored.
DdApiKeySecretArn:
Type: String
AllowedPattern: "arn:.*:secretsmanager:.*"
Default: "arn:aws:secretsmanager:DEFAULT"
Description: The ARN of the secret storing the Datadog API key, if you already have it stored in Secrets Manager. You must store the secret as a plaintext, rather than a key-value pair.
DdSite:
Type: String
Default: datadoghq.com
Description: Define your Datadog Site to send data to. Possible values are `datadoghq.com`, `datadoghq.eu`, `us3.datadoghq.com`, `us5.datadoghq.com` and `ddog-gov.com`.
AllowedPattern: .+
ConstraintDescription: DdSite is required
FunctionName:
Type: String
Default: DatadogForwarder
Description: The Datadog Forwarder Lambda function name. DO NOT change when updating an existing CloudFormation stack, otherwise the current forwarder function will be replaced and all the triggers will be lost.
MemorySize:
Type: Number
Default: 1024
MinValue: 128
MaxValue: 3008
Description: Memory size for the Datadog Forwarder Lambda function
Timeout:
Type: Number
Default: 120
Description: Timeout for the Datadog Forwarder Lambda function
TagsCacheTTLSeconds:
Type: Number
Default: 300
Description: TTL (in seconds) for the Datadog tags cache
ReservedConcurrency:
Type: Number
Default: 100
Description: Reserved concurrency for the Datadog Forwarder Lambda function
LogRetentionInDays:
Type: Number
Default: 90
Description: CloudWatch log retention for logs generated by the Datadog Forwarder Lambda function
SourceZipUrl:
Type: String
Default: ""
Description: DO NOT CHANGE unless you know what you are doing. Override the default location of the function source code.
InstallAsLayer:
Type: String
Default: true
Description: Whether to use the layer-based installation flow. Set to false to use our legacy installation flow, which installs a second function that copies the forwarder code from Github to an S3 bucket. Defaults to true.
AllowedValues:
- true
- false
LayerARN:
Type: String
Default: ""
Description: ARN for the layer containing the forwarder code. If empty, the script will use the version of the layer the forwarder was published with.
DdTags:
Type: String
Default: ""
Description: Add custom tags to forwarded logs, comma-delimited string, no trailing comma, e.g., env:prod,stack:classic
DdFetchLambdaTags:
Type: String
Default: true
AllowedValues:
- true
- false
Description: Let the forwarder fetch Lambda tags using GetResources API calls and apply them to logs, metrics and traces. If set to true, permission tag:GetResources will be automatically added to the Lambda execution IAM role. The tags are cached in memory so that they'll only be fetched when the function cold starts or when the TTL (1 hour) expires. The forwarder increments the aws.lambda.enhanced.get_resources_api_calls metric for each API call made.
DdFetchLogGroupTags:
Type: String
Default: true
AllowedValues:
- true
- false
Description: Let the forwarder fetch Log Group tags using ListTagsLogGroup and apply them to logs, metrics and traces. If set to true, permission logs:ListTagsLogGroup will be automatically added to the Lambda execution IAM role. The tags are cached in memory so that they'll only be fetched when the function cold starts or when the TTL (1 hour) expires. The forwarder increments the aws.lambda.enhanced.list_tags_log_group_api_call metric for each API call made.
DdUseTcp:
Type: String
Default: false
AllowedValues:
- true
- false
Description: By default, the forwarder sends logs using HTTPS through the port 443. To send logs over an SSL encrypted TCP connection, set this parameter to true.
DdNoSsl:
Type: String
Default: false
AllowedValues:
- true
- false
Description: Disable SSL when forwarding logs, set to true when forwarding logs through a proxy.
DdUrl:
Type: String
Default: ""
Description: The endpoint URL to forward the logs to, useful for forwarding logs through a proxy
DdPort:
Type: String
Default: ""
Description: The endpoint port to forward the logs to, useful for forwarding logs through a proxy
DdSkipSslValidation:
Type: String
Default: false
AllowedValues:
- true
- false
Description: Send logs over HTTPS, while NOT validating the certificate provided by the endpoint. This will still encrypt the traffic between the forwarder and the log intake endpoint, but will not verify if the destination SSL certificate is valid.
RedactIp:
Type: String
Default: false
AllowedValues:
- true
- false
Description: Replace text matching \d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3} with xxx.xxx.xxx.xxx
RedactEmail:
Type: String
Default: false
AllowedValues:
- true
- false
Description: Replace text matching [a-zA-Z0-9_.+-]+@[a-zA-Z0-9-]+\.[a-zA-Z0-9-.]+ with xxxxx@xxxxx.com
DdScrubbingRule:
Type: String
Default: ""
Description: Replace text matching the supplied regular expression with xxxxx (default) or DdScrubbingRuleReplacement (if supplied). Log scrubbing rule is applied to the full JSON-formatted log, including any metadata that is automatically added by the Lambda function. Each instance of a pattern match is replaced until no more matches are found in each log. Note, using inefficient regular expression, such as `.*`, may slow down the Lambda function.
DdScrubbingRuleReplacement:
Type: String
Default: ""
Description: Replace text matching DdScrubbingRule with the supplied text
ExcludeAtMatch:
Type: String
Default: ""
Description: DO NOT send logs matching the supplied regular expression. If a log matches both the ExcludeAtMatch and IncludeAtMatch, it is excluded. Filtering rules are applied to the full JSON-formatted log, including any metadata that is automatically added by the function. Note, using inefficient regular expression, such as `.*`, may slow down the Lambda function.
IncludeAtMatch:
Type: String
Default: ""
Description: Only send logs matching the supplied regular expression and not excluded by ExcludeAtMatch. Note, using inefficient regular expression, such as `.*`, may slow down the Lambda function.
DdMultilineLogRegexPattern:
Type: String
Default: ""
Description: Use the supplied regular expression to detect for a new log line for multiline logs from S3, e.g., use expression "\d{2}\/\d{2}\/\d{4}" for multiline logs beginning with pattern "11/10/2014".
DdForwardLog:
Type: String
Default: true
AllowedValues:
- true
- false
Description: Set to false to disable log forwarding, while continuing to forward other observability data, such as metrics and traces from Lambda functions.
DdUseCompression:
Type: String
Default: true
AllowedValues:
- true
- false
Description: Set to false to disable log compression. Only valid when sending logs over HTTP.
DdUsePrivateLink:
Type: String
Default: false
AllowedValues:
- true
- false
Description: DEPRECATED, DO NOT CHANGE. See README.md for details. Set to true to deploy the Forwarder to a VPC and send logs, metrics, and traces via AWS PrivateLink. When set to true, must also set VPCSecurityGroupIds and VPCSubnetIds.
DdUseVPC:
Type: String
Default: false
AllowedValues:
- true
- false
Description: Set to true to deploy the Forwarder to a VPC and send logs, metrics, and traces via a proxy. When set to true, must also set VPCSecurityGroupIds and VPCSubnetIds.
DdHttpProxyURL:
Type: String
Default: ""
Description: "Sets the standard web proxy environment variables HTTP_PROXY and HTTPS_PROXY. These are the url endpoints your proxy server exposes. Don't use this in combination with AWS Private Link. Make sure to also set DdSkipSslValidation to true."
DdNoProxy:
Type: String
Default: ""
Description: "Sets the standard web proxy environment variable NO_PROXY. It is a comma-separated list of domain names that should be excluded from the web proxy."
VPCSecurityGroupIds:
Type: CommaDelimitedList
Default: ""
Description: Comma separated list of VPC Security Group Ids. Used when DdUsePrivateLink or DdUseVPC is enabled.
VPCSubnetIds:
Type: CommaDelimitedList
Default: ""
Description: Comma separated list of VPC Subnet Ids. Used when DdUsePrivateLink or DdUseVPC is enabled.
DdCompressionLevel:
Type: Number
Default: 6
AllowedValues: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
Description: Set the compression level from 0 (no compression) to 9 (best compression) when sending logs.
DdMaxWorkers:
Type: Number
Default: 20
Description: Set the max number of workers sending logs concurrently.
PermissionsBoundaryArn:
Type: String
Default: ""
Description: ARN for the Permissions Boundary Policy
AdditionalTargetLambdaArns:
Type: CommaDelimitedList
Default: ""
Description: Comma separated list of lambda ARNs that get invoked asynchronously with the same input event
DdApiUrl:
Type: String
Default: ""
Description: The endpoint URL to forward the metrics to, useful for forwarding metrics through a proxy
DdTraceIntakeUrl:
Type: String
Default: ""
Description: The endpoint URL to forward the traces to, useful for forwarding traces through a proxy
DdForwarderBucketName:
Type: String
Default: ""
Description: The name of the forwarder bucket to create. If not provided, AWS will generate a unique name.
Conditions:
IsAWSChina:
Fn::Equals:
- Ref: AWS::Partition
- "aws-cn"
IsGovCloud:
Fn::Equals:
- Ref: AWS::Partition
- "aws-us-gov"
UseZipCopier:
Fn::Or:
- Condition: IsAWSChina
- Fn::And:
- Fn::Equals: [!Ref InstallAsLayer, "false"]
- Fn::Not:
- Condition: SetLayerARN
CreateDdApiKeySecret:
Fn::Equals:
- Ref: DdApiKeySecretArn
- "arn:aws:secretsmanager:DEFAULT"
SetFunctionName:
Fn::Not:
- Fn::Equals:
- Ref: FunctionName
- "DatadogForwarder"
SetSourceZipUrl:
Fn::Not:
- Fn::Equals:
- Ref: SourceZipUrl
- ""
SetS3SourceZip:
Fn::Equals:
- !Select [0, !Split ["/", !Ref SourceZipUrl]]
- "s3:"
SetDdTags:
Fn::Not:
- Fn::Equals:
- Ref: DdTags
- ""
SetDdUseTcp:
Fn::Equals:
- Ref: DdUseTcp
- true
SetDdNoSsl:
Fn::Equals:
- Ref: DdNoSsl
- true
SetDdUrl:
Fn::Not:
- Fn::Equals:
- Ref: DdUrl
- ""
SetDdPort:
Fn::Not:
- Fn::Equals:
- Ref: DdPort
- ""
SetRedactIp:
Fn::Equals:
- Ref: RedactIp
- true
SetRedactEmail:
Fn::Equals:
- Ref: RedactEmail
- true
SetDdScrubbingRule:
Fn::Not:
- Fn::Equals:
- Ref: DdScrubbingRule
- ""
SetDdScrubbingRuleReplacement:
Fn::Not:
- Fn::Equals:
- Ref: DdScrubbingRuleReplacement
- ""
SetExcludeAtMatch:
Fn::Not:
- Fn::Equals:
- Ref: ExcludeAtMatch
- ""
SetIncludeAtMatch:
Fn::Not:
- Fn::Equals:
- Ref: IncludeAtMatch
- ""
SetDdMultilineLogRegexPattern:
Fn::Not:
- Fn::Equals:
- Ref: DdMultilineLogRegexPattern
- ""
SetDdSkipSslValidation:
Fn::Equals:
- Ref: DdSkipSslValidation
- true
SetDdFetchLambdaTags:
Fn::Equals:
- Ref: DdFetchLambdaTags
- true
SetDdFetchLogGroupTags:
Fn::Equals:
- Ref: DdFetchLogGroupTags
- true
CreateS3BucketForTags:
Fn::Or:
- Fn::Equals:
- Ref: DdFetchLogGroupTags
- true
- Fn::Equals:
- Ref: DdFetchLambdaTags
- true
SetDdUsePrivateLink:
Fn::Equals:
- Ref: DdUsePrivateLink
- true
SetDdUseVPC:
Fn::Equals:
- Ref: DdUseVPC
- true
SetDdHttpProxyURL:
Fn::Not:
- Fn::Equals:
- Ref: DdHttpProxyURL
- ""
SetDdNoProxy:
Fn::Not:
- Fn::Equals:
- Ref: DdNoProxy
- ""
SetLayerARN:
Fn::Not:
- Fn::Equals:
- Ref: LayerARN
- ""
UseVPC:
Fn::Or:
- Condition: SetDdUsePrivateLink
- Condition: SetDdUseVPC
SetDdForwardLog:
Fn::Equals:
- Ref: DdForwardLog
- false
SetDdUseCompression:
Fn::Equals:
- Ref: DdUseCompression
- false
SetDdCompressionLevel:
Fn::Not:
- Fn::Equals:
- Ref: DdCompressionLevel
- 6
SetDdMaxWorkers:
Fn::Not:
- Fn::Equals:
- Ref: DdMaxWorkers
- 20
SetPermissionsBoundary:
Fn::Not:
- Fn::Equals:
- Ref: PermissionsBoundaryArn
- ""
SetAdditionalTargetLambdas:
Fn::Not:
- Fn::Equals:
- Fn::Join: ["", !Ref AdditionalTargetLambdaArns]
- ""
SetDdApiUrl:
Fn::Not:
- Fn::Equals:
- Ref: DdApiUrl
- ""
SetDdTraceIntakeUrl:
Fn::Not:
- Fn::Equals:
- Ref: DdTraceIntakeUrl
- ""
SetDdForwarderBucketName:
Fn::Not:
- Fn::Equals:
- Ref: DdForwarderBucketName
- ""
Rules:
MustSetDdApiKey:
Assertions:
- Assert:
Fn::Or:
- Fn::Not:
- Fn::Equals:
- Ref: DdApiKey
- ""
- Fn::Not:
- Fn::Equals:
- Ref: DdApiKeySecretArn
- "arn:aws:secretsmanager:DEFAULT"
AssertDescription: DdApiKey or DdApiKeySecretArn must be set
Resources:
Forwarder:
Type: AWS::Lambda::Function
DependsOn: ForwarderZipReady
Properties:
FunctionName:
Fn::If:
- SetFunctionName
- Ref: FunctionName
- Ref: AWS::NoValue
Description: Pushes logs, metrics and traces from AWS to Datadog.
Role: !GetAtt "ForwarderRole.Arn"
Handler: lambda_function.lambda_handler
Layers:
Fn::If:
- UseZipCopier
- []
- - Fn::If:
- SetLayerARN
- !Ref LayerARN
- Fn::Join:
- ":"
- - arn
- !Ref AWS::Partition
- lambda
- !Ref AWS::Region
- Fn::If: [IsGovCloud, "002406178527", "464622532012"]
- layer
- Datadog-Forwarder
- Fn::FindInMap: [Constants, DdForwarder, LayerVersion]
Code:
Fn::If:
- UseZipCopier
- S3Bucket: !Ref ForwarderBucket
S3Key:
Fn::Sub:
- "aws-dd-forwarder-$${DdForwarderVersion}.zip"
- {
DdForwarderVersion:
!FindInMap [Constants, DdForwarder, Version],
}
- ZipFile: " "
MemorySize:
Ref: MemorySize
Runtime: python3.7
Timeout:
Ref: Timeout
Tags:
- Key: "dd_forwarder_version"
Value: !FindInMap [Constants, DdForwarder, Version]
Environment:
Variables:
DD_ENHANCED_METRICS: "false"
DD_API_KEY_SECRET_ARN:
Fn::If:
- CreateDdApiKeySecret
- Ref: DdApiKeySecret
- Ref: DdApiKeySecretArn
DD_S3_BUCKET_NAME:
Fn::If:
- CreateS3BucketForTags
- Ref: ForwarderBucket
- Ref: AWS::NoValue
DD_SITE:
Ref: DdSite
DD_TAGS:
Fn::If:
- SetDdTags
- Ref: DdTags
- Ref: AWS::NoValue
DD_TAGS_CACHE_TTL_SECONDS:
Ref: TagsCacheTTLSeconds
DD_FETCH_LAMBDA_TAGS:
Fn::If:
- SetDdFetchLambdaTags
- Ref: DdFetchLambdaTags
- Ref: AWS::NoValue
DD_FETCH_LOG_GROUP_TAGS:
Fn::If:
- SetDdFetchLogGroupTags
- Ref: DdFetchLogGroupTags
- Ref: AWS::NoValue
DD_USE_TCP:
Fn::If:
- SetDdUseTcp
- Ref: DdUseTcp
- Ref: AWS::NoValue
DD_NO_SSL:
Fn::If:
- SetDdNoSsl
- Ref: DdNoSsl
- Ref: AWS::NoValue
DD_URL:
Fn::If:
- SetDdUrl
- Ref: DdUrl
- Ref: AWS::NoValue
DD_PORT:
Fn::If:
- SetDdPort
- Ref: DdPort
- Ref: AWS::NoValue
REDACT_IP:
Fn::If:
- SetRedactIp
- Ref: RedactIp
- Ref: AWS::NoValue
REDACT_EMAIL:
Fn::If:
- SetRedactEmail
- Ref: RedactEmail
- Ref: AWS::NoValue
DD_SCRUBBING_RULE:
Fn::If:
- SetDdScrubbingRule
- Ref: DdScrubbingRule
- Ref: AWS::NoValue
DD_SCRUBBING_RULE_REPLACEMENT:
Fn::If:
- SetDdScrubbingRuleReplacement
- Ref: DdScrubbingRuleReplacement
- Ref: AWS::NoValue
EXCLUDE_AT_MATCH:
Fn::If:
- SetExcludeAtMatch
- Ref: ExcludeAtMatch
- Ref: AWS::NoValue
INCLUDE_AT_MATCH:
Fn::If:
- SetIncludeAtMatch
- Ref: IncludeAtMatch
- Ref: AWS::NoValue
DD_MULTILINE_LOG_REGEX_PATTERN:
Fn::If:
- SetDdMultilineLogRegexPattern
- Ref: DdMultilineLogRegexPattern
- Ref: AWS::NoValue
DD_SKIP_SSL_VALIDATION:
Fn::If:
- SetDdSkipSslValidation
- Ref: DdSkipSslValidation
- Ref: AWS::NoValue
DD_FORWARD_LOG:
Fn::If:
- SetDdForwardLog
- Ref: DdForwardLog
- Ref: AWS::NoValue
DD_USE_COMPRESSION:
Fn::If:
- SetDdUseCompression
- Ref: DdUseCompression
- Ref: AWS::NoValue
DD_COMPRESSION_LEVEL:
Fn::If:
- SetDdCompressionLevel
- Ref: DdCompressionLevel
- Ref: AWS::NoValue
DD_MAX_WORKERS:
Fn::If:
- SetDdMaxWorkers
- Ref: DdMaxWorkers
- Ref: AWS::NoValue
DD_USE_PRIVATE_LINK:
Fn::If:
- SetDdUsePrivateLink
- true
- false
DD_USE_VPC:
Fn::If:
- UseVPC
- true
- false
HTTP_PROXY:
Fn::If:
- SetDdHttpProxyURL
- Ref: DdHttpProxyURL
- Ref: AWS::NoValue
HTTPS_PROXY:
Fn::If:
- SetDdHttpProxyURL
- Ref: DdHttpProxyURL
- Ref: AWS::NoValue
NO_PROXY:
Fn::If:
- SetDdNoProxy
- Ref: DdNoProxy
- Ref: AWS::NoValue
DD_ADDITIONAL_TARGET_LAMBDAS:
Fn::If:
- SetAdditionalTargetLambdas
- !Join
- ","
- !Ref AdditionalTargetLambdaArns
- !Ref AWS::NoValue
DD_API_URL:
Fn::If:
- SetDdApiUrl
- Ref: DdApiUrl
- Ref: AWS::NoValue
DD_TRACE_INTAKE_URL:
Fn::If:
- SetDdTraceIntakeUrl
- Ref: DdTraceIntakeUrl
- Ref: AWS::NoValue
VpcConfig:
Fn::If:
- UseVPC
- SecurityGroupIds: !Ref VPCSecurityGroupIds
SubnetIds: !Ref VPCSubnetIds
- Ref: AWS::NoValue
ForwarderRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
- Action:
- sts:AssumeRole
Effect: Allow
Principal:
Service:
- lambda.amazonaws.com
ManagedPolicyArns:
- arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
- arn:aws:iam::aws:policy/service-role/AWSLambdaVPCAccessExecutionRole
PermissionsBoundary:
Fn::If:
- SetPermissionsBoundary
- Ref: PermissionsBoundaryArn
- Ref: AWS::NoValue
Policies:
- PolicyName: ForwarderRolePolicy0
PolicyDocument:
Version: "2012-10-17"
Statement:
# Access the s3 bucket that is used by the forwarder as a datastore
- Action:
- s3:GetObject
- s3:PutObject
- s3:DeleteObject
- s3:ListBucket
Resource:
- Fn::Join:
- "/"
- - Fn::GetAtt: ForwarderBucket.Arn
- "*"
Effect: Allow
# Get the actual log content from the s3 bucket based on the received s3 event.
# Use PermissionsBoundaryArn to limit (allow/deny) access if needed.
- Action:
- s3:GetObject
Resource: "*"
Effect: Allow
# To get object from encrypted s3 buckets. Use PermissionsBoundaryArn to limit access if needed.
# https://aws.amazon.com/premiumsupport/knowledge-center/s3-troubleshoot-403/#AWS_KMS_encryption
- Action:
- kms:Decrypt
Resource: "*"
Effect: Allow
# Access the Datadog API key from Secrets Manager
- Action:
- secretsmanager:GetSecretValue
Resource:
Fn::If:
- CreateDdApiKeySecret
- Ref: DdApiKeySecret
- Fn::Sub: "$${DdApiKeySecretArn}*"
Effect: Allow
# Fetch Lambda resource tags for data enrichment
- Fn::If:
- SetDdFetchLambdaTags
- Action:
- tag:GetResources
Resource: "*"
Effect: Allow
- Ref: AWS::NoValue
# Get tags for log groups and attach them to the logs sent to Datadog
- Fn::If:
- SetDdFetchLogGroupTags
- Action:
- logs:ListTagsLogGroup
Resource: "*"
Effect: Allow
- Ref: AWS::NoValue
# Required for Lambda deployed in VPC
- Fn::If:
- UseVPC
- Action:
- ec2:CreateNetworkInterface
- ec2:DescribeNetworkInterfaces
- ec2:DeleteNetworkInterface
Resource: "*"
Effect: Allow
- Ref: AWS::NoValue
# To invoke a follower Lambda with the same event received by the forwarder for dual-shipping
- Fn::If:
- SetAdditionalTargetLambdas
- Action:
- lambda:InvokeFunction
Resource:
Ref: AdditionalTargetLambdaArns
Effect: Allow
- Ref: AWS::NoValue
Tags:
- Value:
Fn::FindInMap:
- Constants
- DdForwarder
- Version
Key: dd_forwarder_version
CloudWatchLogsPermission:
Type: AWS::Lambda::Permission
Properties:
FunctionName: !Ref "Forwarder"
Action: lambda:InvokeFunction
Principal:
Fn::If:
- IsAWSChina
- !Sub "logs.$${AWS::Region}.amazonaws.com.cn"
- !Sub "logs.$${AWS::Region}.amazonaws.com"
SourceAccount: !Ref "AWS::AccountId"
S3Permission:
Type: AWS::Lambda::Permission
Properties:
FunctionName: !Ref "Forwarder"
Action: lambda:InvokeFunction
Principal: "s3.amazonaws.com"
SourceAccount: !Ref "AWS::AccountId"
SNSPermission:
Type: AWS::Lambda::Permission
Properties:
FunctionName: !Ref "Forwarder"
Action: lambda:InvokeFunction
Principal: "sns.amazonaws.com"
SourceAccount: !Ref "AWS::AccountId"
CloudWatchEventsPermission:
Type: AWS::Lambda::Permission
Properties:
FunctionName: !Ref "Forwarder"
Action: lambda:InvokeFunction
Principal:
Fn::If:
- IsAWSChina
- "events.amazonaws.com.cn"
- "events.amazonaws.com"
SourceAccount: !Ref "AWS::AccountId"
LogGroup:
Type: AWS::Logs::LogGroup
Properties:
LogGroupName:
Fn::Sub: /aws/lambda/$${Forwarder}
RetentionInDays:
Ref: LogRetentionInDays
DdApiKeySecret:
Type: AWS::SecretsManager::Secret
Condition: CreateDdApiKeySecret
Properties:
Description: Datadog API Key
SecretString:
Ref: DdApiKey
# A s3 bucket used by the Forwarder as a datastore
ForwarderBucket:
Type: AWS::S3::Bucket
Properties:
BucketName:
Fn::If:
- SetDdForwarderBucketName
- Ref: DdForwarderBucketName
- Ref: AWS::NoValue
BucketEncryption:
ServerSideEncryptionConfiguration:
- ServerSideEncryptionByDefault:
SSEAlgorithm: "aws:kms"
PublicAccessBlockConfiguration:
BlockPublicAcls: true
BlockPublicPolicy: true
IgnorePublicAcls: true
RestrictPublicBuckets: true
ForwarderBucketPolicy:
Type: "AWS::S3::BucketPolicy"
Properties:
Bucket: !Ref ForwarderBucket
PolicyDocument:
Version: "2012-10-17"
Statement:
- Sid: AllowSSLRequestsOnly
Action:
- s3:*
Effect: Deny
Resource:
- !Sub "$${ForwarderBucket.Arn}"
- !Sub "$${ForwarderBucket.Arn}/*"
Condition:
Bool:
"aws:SecureTransport": "false"
Principal: "*"
ForwarderZip:
Type: Custom::ForwarderZip
Condition: UseZipCopier
Properties:
ServiceToken: !GetAtt "ForwarderZipCopier.Arn"
DestZipsBucket: !Ref "ForwarderBucket"
SourceZipUrl:
Fn::If:
- SetSourceZipUrl
- !Ref SourceZipUrl
- Fn::Sub:
- "https://github.com/DataDog/datadog-serverless-functions/releases/download/aws-dd-forwarder-$${DdForwarderVersion}/aws-dd-forwarder-$${DdForwarderVersion}.zip"
- {
DdForwarderVersion:
!FindInMap [Constants, DdForwarder, Version],
}
# The Forwarder's source code is too big to fit the inline code size limit for CloudFormation. In most of AWS
# partitions and regions, the Forwarder is able to load its source code from a Lambda layer attached to it.
# In places where Datadog can't/doesn't yet publish Lambda layers, use another Lambda to copy the source code
# from github to a s3 bucket in the partition & region where the forwarder is deployed to.
ForwarderZipCopier:
Type: AWS::Lambda::Function
Condition: UseZipCopier
Properties:
Description: Copies Datadog Forwarder zip to the destination S3 bucket
Handler: index.handler
Runtime: python3.7
Timeout: 300
Code:
ZipFile: |
import json
import logging
import threading
import boto3
import urllib.request
import os
def send_cfn_resp(event, context, response_status):
resp_body = json.dumps({
'Status': response_status,
'Reason': f'See reasons in CloudWatch Logs - group: {context.log_group_name}, stream:{context.log_stream_name}',
'PhysicalResourceId': context.log_stream_name,
'StackId': event['StackId'],
'RequestId': event['RequestId'],
'LogicalResourceId': event['LogicalResourceId'],
'Data': {}
}).encode('utf-8')
req = urllib.request.Request(url=event['ResponseURL'], data=resp_body, method='PUT')
with urllib.request.urlopen(req) as f:
logging.info(f'Sent response to CloudFormation: {f.status}, {f.reason}')
def delete_zips(bucket):
s3 = boto3.resource('s3')
bucket = s3.Bucket(bucket)
bucket.objects.all().delete()
def copy_zip(source_zip_url, dest_zips_bucket):
s3 = boto3.client('s3')
s3_prelude = "s3://"
filename = "aws-dd-forwarder-{}.zip".format(os.environ.get("DD_FORWARDER_VERSION"))
if source_zip_url.startswith(s3_prelude):
parts = source_zip_url[len(s3_prelude):].split('/')
bucket = parts[0]
key = '/'.join(parts[1:])
response = s3.get_object(Bucket=bucket, Key=key)
data = response["Body"]
s3.upload_fileobj(data, dest_zips_bucket, filename)
else:
with urllib.request.urlopen(source_zip_url) as data:
s3.upload_fileobj(data, dest_zips_bucket, filename)
def timeout(event, context):
logging.error('Execution is about to time out, sending failure response to CloudFormation')
send_cfn_resp(event, context, 'FAILED')
def handler(event, context):
# make sure we send a failure to CloudFormation if the function
# is going to timeout
timer = threading.Timer((context.get_remaining_time_in_millis()
/ 1000.00) - 0.5, timeout, args=[event, context])
timer.start()
logging.info(f'Received event: {json.dumps(event)}')
try:
source_zip_url = event['ResourceProperties']['SourceZipUrl']
dest_zips_bucket = event['ResourceProperties']['DestZipsBucket']
if event['RequestType'] == 'Delete':
delete_zips(dest_zips_bucket)
else:
copy_zip(source_zip_url, dest_zips_bucket)
except Exception as e:
logging.exception(f'Exception when copying zip from {source_zip_url} to {dest_zips_bucket}')
send_cfn_resp(event, context, 'FAILED')
else:
send_cfn_resp(event, context, 'SUCCESS')
finally:
timer.cancel()
Environment:
Variables:
DD_FORWARDER_VERSION: !FindInMap [Constants, DdForwarder, Version]
Role: !GetAtt "ForwarderZipCopierRole.Arn"
ForwarderZipReady:
Type: AWS::CloudFormation::WaitConditionHandle
Metadata:
ForwarderZipCopierReady: !If [UseZipCopier, !Ref ForwarderZip, ""]
ForwarderZipCopierRole:
Type: AWS::IAM::Role
Condition: UseZipCopier
Properties:
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
- Action:
- sts:AssumeRole
Effect: Allow
Principal:
Service:
- lambda.amazonaws.com
ManagedPolicyArns:
- arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
- arn:aws:iam::aws:policy/service-role/AWSLambdaVPCAccessExecutionRole
PermissionsBoundary:
Fn::If:
- SetPermissionsBoundary
- Ref: PermissionsBoundaryArn
- Ref: AWS::NoValue
Policies:
- PolicyName: ForwarderZipCopierRolePolicy0
PolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: Allow
Action:
- s3:PutObject
- s3:DeleteObject
Resource:
- Fn::Join:
- "/"
- - Fn::GetAtt: "ForwarderBucket.Arn"
- "*"
- Effect: Allow
Action:
- s3:ListBucket
Resource:
- Fn::GetAtt: "ForwarderBucket.Arn"
- !If
- SetS3SourceZip
- Effect: Allow
Action:
- s3:GetObject
Resource:
- Fn::Join:
- ""
- - "arn:*:s3:::"
- !Select [1, !Split ["s3://", !Ref SourceZipUrl]]
- Ref: AWS::NoValue
Outputs:
DatadogForwarderArn:
Description: Datadog Forwarder Lambda Function ARN
Value:
Fn::GetAtt:
- Forwarder
- Arn
Export:
Name:
Fn::Sub: $${AWS::StackName}-ForwarderArn
DdApiKeySecretArn:
Description: ARN of SecretsManager Secret with Datadog API Key
Value:
Ref: DdApiKeySecret
Export:
Name:
Fn::Sub: $${AWS::StackName}-ApiKeySecretArn
Condition: CreateDdApiKeySecret
ForwarderBucketName:
Description: Name of the S3 bucket used by the Forwarder
Value:
Ref: ForwarderBucket
Export:
Name:
Fn::Sub: $${AWS::StackName}-ForwarderBucketName
Metadata:
AWS::CloudFormation::Interface:
ParameterGroups:
- Label:
default: Required
Parameters:
- DdApiKey
- DdApiKeySecretArn
- DdSite
- Label:
default: Lambda Function (Optional)
Parameters:
- FunctionName
- MemorySize
- Timeout
- ReservedConcurrency
- LogRetentionInDays
- Label:
default: Log Forwarding (Optional)
Parameters:
- DdTags
- DdMultilineLogRegexPattern
- DdUseTcp
- DdNoSsl
- DdUrl
- DdPort
- DdSkipSslValidation
- DdUseCompression
- DdCompressionLevel
- DdMaxWorkers
- DdForwardLog
- Label:
default: Log Scrubbing (Optional)
Parameters:
- RedactIp
- RedactEmail
- DdScrubbingRule
- DdScrubbingRuleReplacement
- Label:
default: Log Filtering (Optional)
Parameters:
- ExcludeAtMatch
- IncludeAtMatch
- Label:
default: Advanced (Optional)
Parameters:
- DdFetchLambdaTags
- DdFetchLogGroupTags
- TagsCacheTTLSeconds
- SourceZipUrl
- InstallAsLayer
- LayerARN
- PermissionsBoundaryArn
- DdUsePrivateLink
- DdUseVPC
- DdHttpProxyURL
- DdNoProxy
- VPCSecurityGroupIds
- VPCSubnetIds
- DdApiUrl
- DdTraceIntakeUrl
- AdditionalTargetLambdaArns
- DdForwarderBucketName
ParameterLabels:
DdApiKey:
default: "DdApiKey *"
DdApiKeySecretArn:
default: "DdApiKeySecretArn *"
DdSite:
default: "DdSite *"
WHAT
}
# must be pulled into dd account manually under the aws integration tile there, 3rd tab
output "dd-forwarder-arn" {
value = aws_cloudformation_stack.datadog_forwarder.outputs.DatadogForwarderArn
}And the integration itself (hacked together from the quick start link:
resource "aws_cloudformation_stack" "DatadogIntegration" {
name = "DatadogIntegration"
capabilities = ["CAPABILITY_IAM", "CAPABILITY_NAMED_IAM", "CAPABILITY_AUTO_EXPAND"]
parameters = {
APIKey = var.DATADOG_API_KEY
APPKey = var.DATADOG_INTEGRATION_APP_KEY
DatadogSite = "datadoghq.com"
IAMRoleName = "DatadogIntegrationRole"
InstallLambdaLogForwarder = false
}
template_url = "https://datadog-cloudformation-template-quickstart.s3.amazonaws.com/aws/main.yaml"
}
So while I'm eager to believe I'm missing something obvious (as I'm new to these resources), it does seem like this whole thing is very broken as-is. But at least this got it off the ground.
miketheman
SamuelMS
rosshettel
ManuelBuri
ramondelafuente
jvanbrie
Describe what happened:
Try to deploy DataDog Forwarder to the AWS using the latest template by link - https://console.aws.amazon.com/cloudformation/home#/stacks/create/review?stackName=datadog-forwarder&templateURL=https://datadog-cloudformation-template.s3.amazonaws.com/aws/forwarder/latest.yaml.
I got the error: "Resource handler returned message: "Specified ReservedConcurrentExecutions for function decreases account's UnreservedConcurrentExecution below its minimum value of [50]. (Service: Lambda, Status Code: 400, Request ID: b26c3792-2b8b-4077-bcc1-262012e3be87)" (RequestToken: 0800020a-8ad8-8f92-0134-5e8c54643977, HandlerErrorCode: InvalidRequest)"
Try to change the params DdMaxWorkers to 1, ReservedConcurrency to 1,10,25,50,100. Always got this error.
Could you please help me? what I did do wrong?
Thanks