[ByteByteGo Daily] Rate Limiting Fundamentals

[reboottime]

Overview

This is a note extracted from Rate Limiting Fundamentals, By Alex Xu.

Reference:

1. Rate limiting by Alex Xu
2. Wikipedia, priority queue and its implementation

[reboottime]

Rate Limiting Definition and Examples

Rate limiting controls the rate at which users or services can access a resources, like an API, a service, or a network. It plays a critical role in protecting system resources and ensuring fair use among all users.

When the rate of requests exceeds the threshold defined by the rate limiter, the requests are throttled or blocked. Some real world examples can be as following:

(Please notice there is a difference on throttling and blocking)

[reboottime]

Benefits of Rate Limiting

Rate limiting is an integral part of modern large-scale applications.

• Preventing resource starvation: it helps prevent resource starvation caused by Denial of Service (DoS) attacks ( by rejecting the excess calls).
• Reduce cost
  • Reduce outbound cost of third party paid APIs
  • preventing overusing of a resource
• prevent servers from being overloaded: While rate limiting is vital in preventing DoS attacks, it also plays a pivotal role in general load balancing(how load balancing is done under the hood like AWS?) and service quality maintenance

[reboottime]

Applications of Rate Limiting

The applications of rate limiting can be applied at

• application level: limit how many tickets can be sold on a per hour to protect fairness among all users
• user account level: how many tickets an user can purchase for a concert to protect fairness among all users
• API level, for example, open AI API rate limiting

[reboottime]

Core Concepts of Rate Limiting

Most rate limiting implementations share three core concepts. They are the

• the limit: defines the ceiling for allowable requests or actions within a designated time span.
• the window : is the time period where the limit comes into play.
• the identifier: The identifier is a unique attribute that differentiates between individual callers. A user ID or IP address is a common example

After surpassing the rate limit, the action (limiting response) can be three catetgories

• blocking: Blocking takes place when requests exceeding the limit are denied access to the resource. It is commonly expressed as an error message such as HTTP status code 429 (Too Many Requests).
• throttling: by comparison, involves slowing down or delaying the requests that go beyond the limit. An example would be a video streaming service reducing the quality of the stream for users who have gone over their data cap.
• shaping: on the other hand, allows requests that surpass the limit. But those requests are assigned lower priority.

[reboottime]

Common Rate Limiting Algorithms

This article introduced algorithms

• Fixed window counter
• sliding window log
• sliding window counter
• token bucket
• leaky bucket

The algorithms explaination content are unavailable for unpaid user ):