ThrottlingException

[jjleigh]

Receiving throttling exception in these 2 situations:

• When account assignment is created with a permission set that has 'undefined' as the value for the relay state
• When account assignment is created and the OU or tag has more than 40 accounts. The SQS queues the right number but after 40 the link handler is throttled and none of the unprocessed messages end up in the dead letter queue.

This was tried on an OU with 72 accounts but only 40 were provisioned.

Here is part of the error bellow:

Subject: Error Processing link provisioning operation

"eventSource":"aws:sqs","eventSourceARN":"arn:aws:sqs:us-east-:env-linkManagerQueue.fifo","awsRegion":"us-east-1"}]},"errorDetails":{"name":"ThrottlingException","$fault":"client","$metadata":{"httpStatusCode":400,"requestId":"6cd844bc-a51b-414d-89f9-c9723df46bcb","attempts":2,"totalRetryDelay":175},"__type":"ThrottlingException","message":"There are too many requests processing. Please try again later."}}

[allquixotic]

This is affecting my organization in a PROD deploy right now. It's not the same organization as the one this issue was reported for.

[leelalagudu]

@allquixotic , @jjleigh - ACK on the issue. For context, this is how the solution should behave:

• When an account assignment operation, with root, acount_tag, ou_id scope is provisioned, the solution triggers of a state machine in the ORG main account to discover the list of target AWS accounts that resolve to that scope.
• The state machine is designed such that it only posts a maximum of 5 account assignment payloads (in parallel) at any given time.
• It then posts account assignment creation/deletion payloads to an SNS topic that has cross account lambda subscriber from the target account
• This subscriber then puts the payload into a FIFO queue using combination of target account, permission set and operation(create/delete) as the message group ID
• The FIFO queue subscriber has a batch size of 10 enforced i.e. it would not process any more than 10 account assignments at a given time.
• Additionally, the subscriber would only remove the payload from the queue only after the account assignment is successfully created/deleted/operation timed out/SSO admin API returns an error back
• These limits are set with the expectation that the standard SSO admin API throttle limit is 10.

I believe the missing part here is a mandatory wait enforcement between each pagination operation in the state machine. Because this was missing, the state machine keeps processing the pages and overloads the FIFO queue. We did enforce this pattern on the current config import/ region switch state machines.

We will amend the state machines to fit in with this behaviour and ask @jmejco / @tamara-h to validate on a demo set up that has more than 50 accounts under an account_tag/ou_id/root scope and check if the enforced wait between pages resolves the issue. We're on public holiday here in the UK until 5th June and as such they could validate the behaviour on 6th June.

Hope this helps, Leela

[allquixotic]

I set sleep statements in my Directory Service to SSO migration code that slow down the import and worked around this for my purposes, but it should definitely be resolved in the SSO Extensions code.

[jjleigh]

@leelalagudu Thank you for the update!

[leelalagudu]

Thanks for the udpate @allquixotic , at least this confirms my hypothesis theoretically. As updated, we will go ahead with this design change and load test the solution as part of the PR release.

[leelalagudu]

@jjleigh , @allquixotic , mandatory wait enforcement between each page is now in the solution through #89 . This should handle the throttling exceptions you are seeing.

Please do let us know if this fixed your issue, Leela