While running a scalability test in UAT for the fix for #296, the fix itself worked, but revealed another error:
The state/task 'DiscoverGranulesMap' returned a result with a size exceeding the maximum number of bytes service limit.
This occurred because the output from the map state (formed by collecting all outputs from the map iterations into a single array) exceeded the quota for the output size for a state (of any kind).
This may also occur for the 'QueueGranulesMap' state as well.
While running a scalability test in UAT for the fix for #296, the fix itself worked, but revealed another error:
This occurred because the output from the map state (formed by collecting all outputs from the map iterations into a single array) exceeded the quota for the output size for a state (of any kind).
This may also occur for the 'QueueGranulesMap' state as well.
We must make use of a
ResultWriterto write the results to S3 instead, so that we don't hit this limit. See https://docs.aws.amazon.com/step-functions/latest/dg/input-output-resultwriter.html