NeoAgi / NeoAgi.AWS.CodeArtifact.Pruner

Helper Utility to Mange an Amazon CodeArtifact Domain
https://gallery.ecr.aws/x7q2k3a7/neoagi.aws.codeartifact.pruner
Apache License 2.0
0 stars 0 forks source link

Implement Rate Limiting Observations and Throttling #11

Closed Habikki closed 9 months ago

Habikki commented 10 months ago

The current Exception is thrown when Amazon Rate Limits requests:

---> (Inner Exception #445) System.AggregateException: One or more errors occurred. (Rate exceeded)
 ---> Amazon.CodeArtifact.Model.ThrottlingException: Rate exceeded
 ---> Amazon.Runtime.Internal.HttpErrorResponseException: Exception of type 'Amazon.Runtime.Internal.HttpErrorResponseException' was thrown.

Observationally, this only occurred when the process was run off AWS Infra lending to the idea that rate limits are different on-network, or simply that the external system was more performant.

At present no observation is given to rate limiting which should be implemented, but acting on the rate limiting will require rethinking of how discovery and removal occur.

As discovery and removal are already separated, the changes here reside within ProcessRemovalsAsync(). The process queue should be processed within bounds of a TPS rate limit respecting an exponential backoff delay if a ThrottlingException is encountered.

Habikki commented 9 months ago

Offending line occurred earlier than thought, in the discovery phase:

https://github.com/NeoAgi/NeoAgi.AWS.CodeArtifact.Pruner/blob/main/NeoAgi.AWS.CodeArtifact.Pruner/Worker.cs#L202

Habikki commented 9 months ago

Current implementation of limiting TPS can be thought of as:

  1. Fill up a work action queue (No Limit)
  2. A process counter ticks every second resetting a TPS counter.
  3. When the TPS is below a set threshold, the work scheduler assigns queued items to worker threads (Statically set to 20)

While this limits calls to a known TPS, in practice it does so by executing calls in batches of the TPS window if the calls are rapid like CloudArtifact Calls are. Some level of smoothing should be applied. Proposed changes: