Invalid request provided: AWS::Logs::SubscriptionFilter

[psiservices-ian-kost]

I have been working on setting up the forwarder using the terraform deployment method mentioned in the docs. This looks to be deploying the 1.11 version of the forwarder. I am trying to deploy to both the us-west-2 and eu-west-2 regions. The number of lambda cloudwatch log group arns per region is 42 right now, probably closer to 50 after I get past this testing phase. When the terraform deployment runs for us-west-2 it completes successfully. When it runs for eu-west-2 it fails 9/10 and cloudformation shows the following error:

"Invalid request provided: AWS::Logs::SubscriptionFilter. Could not execute the lambda function. Make sure you have given CloudWatch Logs permission to execute your function."

The only thing I have come up with is that this is a timing issue when the resource-based policy statements are getting created as those provide the lambda:InvokeFunction permission which is what the above error is complaining about. I am not really sure where to go from here as it does technically work. Maybe the deployment needs some additional checks/waits around these permissions, but at the success rate I am having it is going to make using this very hard.

[aspacca]

@psiservices-ian-kost

thanks for reporting the issue, we will double check in the CloudFormation template that bakes terraform and see if any dependencies is missing between subscription filters creation and the lambda creation

[mpozniak]

I can confirm the same error when deploying using AWS CDK TS (eu-west-2): Resource handler returned message: "Invalid request provided: AWS::Logs::SubscriptionFilter. Could not execute the lambda function. Make sure you have given CloudWatch Logs permission to execute your function. (Service: CloudWatchLogs, Status Code: 400, Request ID: XXXX-XXXX-XXXX)" Only after adding permissions 'lambda:InvokeFunction' to the Lambda function manually was it possible to create a subscription filter. It looks like there is a bug in the dependency of the IAM Role 'subscriptionCanInvokeLambda' creation and the creation of the 'AWS::Logs::SubscriptionFilter' function.

[aspacca]

I can confirm the same error when deploying using AWS CDK TS (eu-west-2): Resource handler returned message: "Invalid request provided: AWS::Logs::SubscriptionFilter. Could not execute the lambda function. Make sure you have given CloudWatch Logs permission to execute your function. (Service: CloudWatchLogs, Status Code: 400, Request ID: XXXX-XXXX-XXXX)" Only after adding permissions 'lambda:InvokeFunction' to the Lambda function manually was it possible to create a subscription filter. It looks like there is a bug in the dependency of the IAM Role 'subscriptionCanInvokeLambda' creation and the creation of the 'AWS::Logs::SubscriptionFilter' function.

Hi @mpozniak your problem seems different from @psiservices-ian-kost one.

I guess your deployment solution using AWS CDK TS is custom made.

I can confirm that the lambda:InvokeFunction permission is required for the AWS::Logs::SubscriptionFilter. I don't know the details of the IAM Role 'subscriptionCanInvokeLambda'.

[aspacca]

@psiservices-ian-kost, please check the v1.13.0 release that adds an extra field for every CF Parameters used to define triggers of ESF, if you are hitting the 4096b limit and that's the reason you deploy on two different regions with SAR.

I've double checked also the following, that's was reported also (according to how it does apply) for Kinesis Data Stream:

The only thing I have come up with is that this is a timing issue when the resource-based policy statements are getting created as those provide the lambda:InvokeFunction permission which is what the above error is complaining about

The final CF template on SAR defines two resources:

• ApplicationElasticServerlessForwarder: AWS::Serverless::Function
• ElasticServerlessForwarderPolicy: AWS::IAM::Policy

All the triggers are defined in the first one, as Properties.Events, and the relative permissions and policies are taken care of by AWS (I guess through the Transform: AWS::Serverless-2016-10-31).

The second is a policy for the permissions related to AWS resources different the ESF triggers: things like the SSM Secretes, KMS Keys etc.

There is anyway a dependency on the policy from the application.

So it seems more likely somehow a problem with the deployment order or the final template generated by Transform: AWS::Serverless-2016-10-31 on AWS.

I will close the issue for that.