At the moment the processing relies on retries, which
Puts unnecessary stress on the system with retries that fail often for large number of concurrent Step Function executions
The retries cost state transitions
Retries can reach a limit and fail
The solution is to only allow the number of workflows at any given time close to the concurrenty for the component with the lowest limit (aka bottleneck). This could be a backend system that only allows processing x requests per second or the number of concurrent jobs the asynchronous Textract API for StartDocumentText is set to in the region.
At the moment the processing relies on retries, which
The solution is to only allow the number of workflows at any given time close to the concurrenty for the component with the lowest limit (aka bottleneck). This could be a backend system that only allows processing x requests per second or the number of concurrent jobs the asynchronous Textract API for StartDocumentText is set to in the region.