wafv2_web_acl_logging_configuration throws WAFLogDestinationPermissionIssueException when using cloud watch log group

lorelei-rupp-imprivata commented 2 years ago

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform AWS Provider Version

Terraform 0.14.7 AWS Provider 3.73.0

Affected Resource(s)

wafv2_web_acl_logging_configuration

Terraform Configuration Files

Please include all Terraform configurations required to reproduce the bug. Bug reports without a functional reproduction may be closed without investigation.

resource "aws_cloudwatch_log_group" "waf_log" {
  name  = "aws-waf-logs-regional/${var.env}"
}
resource "aws_wafv2_web_acl_logging_configuration" "example" {
  log_destination_configs = [aws_cloudwatch_log_group.waf_log[0].arn]
  resource_arn            = aws_wafv2_web_acl.rules_web_acl.arn
}

Debug Output

Error: error putting WAFv2 Logging Configuration for resource (arn*****): WAFLogDestinationPermissionIssueException: Unable to deliver logs to the configured destination. You might need to grant log delivery permissions for the destination. If you're using S3 as your log destination, you might have exceeded your bucket limit.

on main.tf line 224, in resource "aws_wafv2_web_acl_logging_configuration" "example": 224: resource "aws_wafv2_web_acl_logging_configuration" "example" {

Expected Behavior

This worked last week, but no longer works. We opened an AWS Support ticket but they are telling me we need to run a CLI command to add a resource policy to the cloud watch log group. We have never had to do this before

Actual Behavior

It fails with the error WAFLogDestinationPermissionIssueException: Unable to deliver logs to the configured destination. You might need to grant log delivery permissions for the destination. If you're using S3 as your log destination, you might have exceeded your bucket limit.

The docs too for https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/wafv2_web_acl_logging_configuration don't say anything about needing to also set up a policy.

Steps to Reproduce

Try to apply a log group and waf logging to a waf This also fails in the AWS Console as well

Important Factoids

We opened an AWS Support ticket but they don't seem to think its an issue and say you have to run manual CLI commands. We have had this terraform code in place for MONTHS and it has worked until last week Even this past ticket https://github.com/hashicorp/terraform-provider-aws/issues/23934 shows an example identical to what we are using. There is no mention of an additional policy?

lorelei-rupp-imprivata commented 2 years ago

Also this appears to work for a Regional WAF, but is failing on a CloudFront Global WAF

lorelei-rupp-imprivata commented 2 years ago

It actually appears this might be related to https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/AWS-logs-and-resource-policy.html and the fact our us-east-1 resource policy doc has exceeded the character limit. 5120 characters It works for regional because thats not us-east-1 for us. I have also updated my aws support ticket, as I can see no way to clean this resource policy up

justinretzolk commented 2 years ago

Hey @lorelei-rupp-imprivata 👋 Thank you for the update -- I was just looking into this when I saw your comment come through 🙂. It looks like this resource hasn't changed since January, so I suspect it's not an issue with a recent change to the provider, but I'll keep an eye out for any additional updates with what you hear back from the AWS support ticket.

lorelei-rupp-imprivata commented 2 years ago

Thanks @justinretzolk so I think this is good info for the community. From AWS Support

CloudWatch Logs resource Policies allows the AWS services to send Logs to Log Groups. 
The Permission can be added automatically when you enabled AWS WAF Logs to CloudWatch if the resource Policy had not been added if you are enabling it via console. 
 An account can only  have:
1. 10 resource policies per AWS Region. This quota can't be changed [1]
2. Each policy Document can have  Maximum length of 5120 [2]

So WAF in particular was automatically adding to this policy, and eventually you will hit the limit

I think we might want to consider updating the Terraform DOCs for the wafv2_web_acl_logging_configuration resource. To state if you are going to go the CloudWatch log group route, you may want to consider setting up the https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/cloudwatch_log_resource_policy yourself. While AWS is going to auto add it for you, you could run into this limit issue we hit. I believe the solution is to create the policy myself and attach it to the log group. I am going to test this out today.

AWS Support also said you can "fix this via the CLI" too by "re putting the policy" and using "regex" or * or combining, grouping. Its not great IMO. The fact they do not clean this up when you delete a log group or disable WAF logging is not great and anyone could get stuck like me

Please note that,  To avoid hit the limit of 5120 characters in each Resource Policy, you have three resolutions:

1. Deleting old Policies and grouping them
2. Combine multiple policies together if this is possible and use it as one policy.
3. Change the resource list to "*" instead of restricting to specific resource resulting to limit issues.

I hope this will help someone else. Can we keep this open and ill post whether the TF policy works? Not sure if you want to update the docs too~

lorelei-rupp-imprivata commented 2 years ago

FYI -- this works. I rolled out a resource policy per region with a regex for what my log groups for waf logging use. This works perfectly and gets around this AWS Bug/Issue. I would highly recommend we update the Note about wafv2_web_acl_logging_configuration with Cloud Watch logs, not only should you name it aws-waf-logs, but you probably should roll your own logging resource policy since you can easily hit the AWS limit with no way to clean up without even realizing it @justinretzolk

TheWallOfDucks commented 2 years ago

Hi @lorelei-rupp-imprivata I am running into this same issue...thank you very much for your comments. Did you need to do anything besides create a resource policy for all of your WAF logs to use? I see you mention attaching the policy to the log group, but I'm not finding any good information on how to do this.

Thanks in advance.

lorelei-rupp-imprivata commented 2 years ago

Hi @lorelei-rupp-imprivata I am running into this same issue...thank you very much for your comments. Did you need to do anything besides create a resource policy for all of your WAF logs to use? I see you mention attaching the policy to the log group, but I'm not finding any good information on how to do this.

Thanks in advance.

We used terraform, there was a resource to create the policy. All we had to do was create this https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/cloudwatch_log_resource_policy

LiamNewtonNHS commented 1 year ago

Hi all just to update you are able to run a CLI Command like the following to add a Resource Policy: (Won't be formatted amazingly on here)

aws logs put-resource-policy --policy-name AWSWAF-LOGS --policy-document '{ "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": { "Service": "delivery.logs.amazonaws.com" }, "Action":[ "logs:CreateLogStream", "logs:PutLogEvents" ], "Resource" : "arn:aws:logs:<region>:<account_number>:log-group:aws-waf-log*:*", "Condition": {"ArnLike": {"aws:SourceArn": "arn:aws:logs:<region>:<account_number>:*"}, "StringEquals": {"aws:SourceAccount": "<account_number>"}} } ] }'

grahamhar commented 1 year ago

Hi,

I have just hit this and I found that adding my own policy using aws_cloudwatch_log_resource_policy just for my webacl/log group meant that I no longer got this issue. It also means the policy is cleaned up when I destroy as it is all in the same code base.

I will look to do a PR to add an example in the docs

github-actions[bot] commented 1 year ago

This functionality has been released in v5.3.0 of the Terraform AWS Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

github-actions[bot] commented 1 year ago

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

hashicorp / terraform-provider-aws