Grafana Cloud provided lambda-promtail.yaml missing permissions

[mjpowersjr]

Summary Following the "Logs for Lambda" Grafana Cloud Connection instructions fails.

The provided lambda-promtail.yaml CloudFormation template is missing an additional permission. After doing some research, it turns out that AWS Console set's the missing permission behind-the-scenes when a user manually creates a subscription filter. I suspect the CF Template author was unaware of this, and things worked in their environment after manually creating a subscription filter.

Error Message in CloudFormation

Resource handler returned message: "Could not execute the lambda function. Make sure you have given CloudWatch Logs permission to execute your function. (Service: CloudWatchLogs, Status Code: 400, Request ID: )" (RequestToken: , HandlerErrorCode: InternalFailure)

Fix I downloaded lambda-promtail.yaml manually, and added the following section to resolve the issue:

  LambdaCloudwatchInvokePermission:
    Type: AWS::Lambda::Permission
    DependsOn: LambdaPromtailFunction 
    Properties:
      FunctionName: !Ref LambdaPromtailFunction
      Principal: !Sub
        - "logs.${Region}.amazonaws.com"
        - { Region: !Ref "AWS::Region" }
      Action: "lambda:InvokeFunction"
      SourceAccount:
        Ref: AWS::AccountId
      SourceArn: !Sub
        - "arn:aws:logs:${Region}:${AccountId}:log-group:*:*"
        - { AccountId: !Ref "AWS::AccountId", Region: !Ref "AWS::Region" }

Related Discussions

• https://stackoverflow.com/questions/68075120/terraform-error-creating-cloudwatch-log-subscription-filter-invalidparametere
• https://biercoff.com/how-to-fix-add-permissions-to-aws-lambda/

[cstyan]

@mjpowersjr Yeah I've never used cloudformation myself other than writing that file as a quick example. PRs to improve the template are welcome! This certainly sounds like a valid change to make.