Excessive Log Parser Lambda executions causing rate limit problems

[gavrichards]

Describe the bug I have reported this to the support team and I'm in ongoing conversations with them which you may be aware of, but they mentioned I could post here to receive updates too.

We migrate to WAF v2 recently with the latest version of Security Automations, and since doing so we're having constant problems with rate limit errors, both via the API and in the console. This has now been tracked down to the Log Parser Lambda function being run excessively, meaning it's using up the quota for updates to IP lists, causing us to be unable to make changes ourselves. The knock-on effect of this is that our costs have noticeably increased since migrating.

To Reproduce Migrate to WAF v2 and set up Security Automations by following the instructions. Then try to update an IP set, for example delete an IP. You then get one of two errors: ThrottlingException: Rate exceeded or WAFOptimisticLockException: AWS WAF couldn’t save your changes because someone changed the resource after you started to edit it. Reapply your changes.

Expected behavior No rate limiting errors from regular usage. No increase in costs from WAF Classic with previous version of Security Automations.

Please complete the following information about the solution:

• [ ] Version: 3.0.0

To get the version of the solution, you can look at the description of the created CloudFormation stack. For example, "(SO0021) - Video On Demand workflow with AWS Step Functions, MediaConvert, MediaPackage, S3, CloudFront and DynamoDB. Version v5.0.0". If the description does not contain the version information, you can look at the mappings section of the template:

Mappings:
  SourceCode:
    General:
      S3Bucket: "solutions"
      KeyPrefix: "video-on-demand-on-aws/v5.0.0"

• [ ] Region: us-east-1
• [ ] Was the solution modified from the version published on this repository? No
• [ ] If the answer to the previous question was yes, are the changes available on GitHub?
• [ ] Have you checked your service quotas for the sevices this solution uses?
• [ ] Were there any errors in the CloudWatch Logs?

Screenshots If applicable, add screenshots to help explain your problem (please DO NOT include sensitive information).

Additional context Add any other context about the problem here.

[aijunpeng]

Thanks for reporting the issue. We are aware of this and added it to our backlog for investigation. The WAF service has one request per second limit for Create, Put, or Update action. Any calls beyond that may be throttled. Can you please provide the following information to help our investigation?

1. The input parameter/value for your cloudformation stack
2. Did throttling happen only when you tried to update ip set on your own?

[gavrichards]

1.

Key	Value	Resolved Value
ActivateAWSManagedRulesParam	yes	-
ActivateBadBotProtectionParam	yes	-
ActivateCrossSiteScriptingProtectionParam	yes	-
ActivateHttpFloodProtectionParam	yes - AWS WAF rate based rule	-
ActivateReputationListsProtectionParam	yes	-
ActivateScannersProbesProtectionParam	yes - AWS Lambda log parser	-
ActivateSqlInjectionProtectionParam	yes	-
AppAccessLogBucket	frontend-cf-logs2.aiir.net	-
EndpointType	CloudFront	-
ErrorThreshold	50	-
KeepDataInOriginalS3Location	No	-
RequestThreshold	100	-
WAFBlockPeriod	240	-

2. No - according to CloudTrail, UpdateIPSet events are happening many times per second. This isn't us - we have a script which updates an IP set but it only runs once an hour, and most hours it is blocked. See this sample:

[aijunpeng]

Thank you for the information. The solution shouldn't update ip set that frequently under normal circumstances. I am also communicating with our support engineers to get more context regarding your case. They will keep you updated from here on.

[suthar105]

I am facing the same problem, infect I have noticed that due to "WAFOptimisticLockException: AWS WAF couldn’t save your changes because someone changed the resource after you started to edit it. Reapply your changes." error from updateipset action of logparsar lambda function it is doing same ip update attempts many times. Below is example event of update ip set with error.. {"eventVersion":"1.05","userIdentity":{"type":"AssumedRole","principalId":"xxxx:xxxx-LogParser-xxxx","arn":"arn:aws:sts::xxxx:assumed-role/xxxx-LambdaRoleLogParser-xxxx/xxxx-LogParser-xxxx","accountId":"xxxx","accessKeyId":"xxxx","sessionContext":{"sessionIssuer":{"type":"Role","principalId":"xxxx","arn":"arn:aws:iam::xxxx:role/xxxx-LambdaRoleLogParser-xxxx","accountId":"xxxxx","userName":"xxxx-LambdaRoleLogParser-xxxx"},"webIdFederationData":{},"attributes":{"mfaAuthenticated":"false","creationDate":"2020-09-23T01:29:23Z"}}},"eventTime":"2020-09-23T01:45:22Z","eventSource":"wafv2.amazonaws.com","eventName":"UpdateIPSet","awsRegion":"ap-northeast-1","sourceIPAddress":"x.x.x.x","userAgent":"Boto3/1.14.48 Python/3.8.5 Linux/4.14.177-104.253.amzn2.x86_64 exec-env/AWS_Lambda_python3.8 Botocore/1.17.48","errorCode":"WAFOptimisticLockException","errorMessage":"AWS WAF couldn’t save your changes because someone changed the resource after you started to edit it. Reapply your changes.","requestParameters":{"name":"ScannersProbesSetIPV4","scope":"REGIONAL","id":"eaca0359-0200-4b8f-ac26-383a09fa89af","description":"Block Scanners/Probes IPV4 addresses","addresses":["59.127.234.253/32"],"lockToken":"058953b5-e66a-4da3-ae3f-109bc1087add"},"responseElements":null,"requestID":"6794773d-3290-4a56-9f46-3d7bb7b60ba3","eventID":"9f72ab5b-0aa8-4bc3-a2f7-2287730149f4","eventType":"AwsApiCall","apiVersion":"2019-04-23","recipientAccountId":"xxx"}

[aijunpeng]

Thank you for the information. The solution uses optimistic backoff logic to retry failed api calls. We have added this to our backlog and will address it in future releases as needed.

[dscpinheiro]

Hi!

We just released v3.2.0 of the solution, and this issue has been fixed.