Closed Habikki closed 9 months ago
Offending line occurred earlier than thought, in the discovery phase:
Current implementation of limiting TPS can be thought of as:
While this limits calls to a known TPS, in practice it does so by executing calls in batches of the TPS window if the calls are rapid like CloudArtifact Calls are. Some level of smoothing should be applied. Proposed changes:
ThrottlingException
s and replay the action, inforcing a sustancial delay (5 - 10 seconds) to avoid an exponential backoff. (mostly implemented in 8b57079)
The current Exception is thrown when Amazon Rate Limits requests:
Observationally, this only occurred when the process was run off AWS Infra lending to the idea that rate limits are different on-network, or simply that the external system was more performant.
At present no observation is given to rate limiting which should be implemented, but acting on the rate limiting will require rethinking of how discovery and removal occur.
As discovery and removal are already separated, the changes here reside within
ProcessRemovalsAsync()
. The process queue should be processed within bounds of a TPS rate limit respecting an exponential backoff delay if aThrottlingException
is encountered.