Feature Request: User-Side Rate Limiter for Rate Limit Management in API Requests

[TheMemeticist]

Feature Request: User-Side Rate Limiter for Rate Limit Management in API Requests

Feature Summary: Implement a user-side rate limiter to manage and control the frequency of requests sent to the Anthropic API. This feature would help prevent rate limit errors (e.g., litellm.RateLimitError: AnthropicException - {"type":"rate_limit_error"}) by dynamically adjusting the rate of requests based on the current usage and limit thresholds provided in the API response headers.

Problem Statement: Currently, the application may encounter rate limit errors when the number of request tokens exceeds the daily limit set by Anthropic. This results in unexpected interruptions to agent functionality, causing users to experience downtime and delays. Users are unable to continue their workflows smoothly and are often unaware of their current usage until the error occurs.

Proposed Solution: The solution involves implementing a user-side rate limiter that will:

1. Track the current request token usage by parsing the rate limit information from the response headers.
2. Dynamically throttle or queue requests based on the remaining available tokens, thereby preventing unexpected rate limit errors.
3. Provide feedback to the user (e.g., estimated time to send the next request or current usage vs. limit status) to inform them of their current API usage.

Feature Details:

1. Usage Tracking: Monitor the rate limit status in real time by reading the response headers after each request. Store the current request token count and limit for efficient tracking.
2. Adaptive Throttling: When the token count approaches the limit, reduce the request frequency to avoid hitting the daily threshold. Use an exponential backoff approach when the usage is close to the limit.
3. Queue Management: Allow queued requests when the limit is reached, holding them until more tokens become available.
4. User Feedback: Provide the user with information about the current rate limit status, including the number of remaining tokens and an estimated time for when the next request can be sent.

Benefits:

• Prevents Interruptions: Avoids sudden errors by proactively managing request rates.
• Optimized API Usage: Efficiently manages requests to maximize usage within the rate limit.
• Enhanced User Experience: Informs users about their current usage and gives them control over request timing.

Potential Challenges:

• Complexity in Queue Management: Managing queued requests efficiently may introduce additional logic for handling request timing and sequence.
• Latency Considerations: Throttling may lead to slightly increased response times, but this is offset by the prevention of abrupt errors.

[Kalmuraee]

Thank you for bringing this up!

To address the rate-limiting issue, we could implement a queue handler that retries the last failed request after the rate limit cools down. Using the rate-limit headers provided by the API can help in setting up a structured retry mechanism. Here’s a brief outline of how it would work:

1- Rate-Limit Information: The following headers provide useful details for managing retries:

• anthropic-ratelimit-requests-reset: This header shows when the request rate limit will reset, formatted in RFC 3339.
• anthropic-ratelimit-tokens-reset: This indicates when the token rate limit will reset, also in RFC 3339.
• retry-after: This provides the number of seconds until a request can be retried.

• 2- Queue Handler Logic: The handler would queue any failed requests due to rate limits and check these headers to determine when to retry. Specifically, it could use the retry-after value as a timer to trigger the next attempt, ensuring no additional rate-limit errors occur.

This setup would allow OpenHands to automatically reattempt failed requests without manual intervention, improving stability when handling bursts of requests.

[champ2050]

Rate limits are a problem some workaround should be made for this imo.

[enyst]

Openhands has user configurable retries for rate limits. Please take a look at config.template.toml file, the relevant settings are in the [llm] section:

# Number of retries to attempt when an operation fails with the LLM.
# Increase this value to allow more attempts before giving up
#num_retries = 8

# Maximum wait time (in seconds) between retry attempts
# This caps the exponential backoff to prevent excessively long
#retry_max_wait = 120

# Minimum wait time (in seconds) between retry attempts
# This sets the initial delay before the first retry
#retry_min_wait = 15

# Multiplier for exponential backoff calculation
# The wait time increases by this factor after each failed attempt
# A value of 2.0 means each retry waits twice as long as the previous one
#retry_multiplier = 2.0

You can customize them in the config.toml file, or, if you're running with the docker app, you can add them with -e and the corresponding env var (uppercase, and with the LLM_ prefix). For example, -e LLM_RETRY_MIN_WAIT=20.

What we don't currently do, is read the API headers and adapt to them. We just do what the user configures there. Personally I had to make the values more lenient for Anthropic...

I think you're right we should, and we have a PR on it, but we haven't got it ready yet. 😅