search

awslabs / aws-securityhub-multiaccount-scripts

This script automates the process of running the Security Hub multi-account workflow across a group of accounts that are in your control
MIT No Attribution
271 stars 108 forks source link

Error: The state/task 'UpdateMembers' returned a result with a size exceeding the maximum number of bytes service limit. #63

Open starwolf73 opened 1 year ago

starwolf73 commented 1 year ago

Hi, We have an large Organizations with more than 1300 accounts overs 17+ regions and this solution was a must for our Security Hub installation. Unfortunately, in DEV Org with less accounts and regions it was working fine until we start implementing it in PROD. If failed at the last iteration #1301 for a total of 9124 events. We received the following error in our step-function:

States.DataLimitExceeded The state/task 'UpdateMembers' returned a result with a size exceeding the maximum number of bytes service limit.

Anything we can do to increase this limit? Or it required a change in the code to process it differently?

scottbward commented 1 year ago

Thanks for reporting this. I am looking into this and will get back as soon as I can regarding what can be done to address this error.

scottbward commented 1 year ago

The error that you are reporting is from the AWS Step Functions service. Quotas for Step functions are outlined here: https://docs.aws.amazon.com/step-functions/latest/dg/limits-overview.html, specifically it would be the "Maximum input or output size for a task, state, or execution" quota. From a few searches it looks like the solve is to alter the way that information is being passed around for the tasks in your step function. I suspect that due to the large number of accounts you are accruing a large amount of responses from AWS API calls and that is accounting for the max size exceeded.

Here are a few resources that might help: https://ncoughlin.com/posts/aws-step-functions-notes/#statesdatalimitexceeded. - This seems like it may align well to your situation.

https://stackoverflow.com/questions/55702571/step-function-exceeding-the-maximum-number-of-characters-service-limit https://stackoverflow.com/questions/71833184/aws-step-functions-size-exceeding-the-maximum-number-of-bytes-service-limit?noredirect=1&lq=1

Let me know if this helps or if you have additional questions. I'll do my best to help you along on this.