Deliverables, ETAs, Milestones - Rate Limting Middleware | Parent Ticket |

[bajajcodes]

Problem

• The Express.js backend API code is missing a rate-limiting feature, which can lead to server overload, slow response times, and poor user experience.
• The absence of rate-limiting can cause issues such as excessive requests, automated bots abuse, and security threats like brute-force attacks.
• The consequences of missing rate-limiting are numerous, including system vulnerability to DDoS attacks, poor API performance, and reduced user functionality and trust.

Solution

• Our proposed solution is to implement a custom rate-limiting middleware using the "node-rate-limiter-flexible" package, which can be easily integrated with the existing Express.js backend.
• The middleware will enforce rate limits on all incoming requests to the API, preventing abuse and improving overall performance.
• We will use a Redis Hash database to store rate-limiting rules and deploy the application on the "railway.app" platform to ensure scalability and reliability.
• Different rate-limiting rules will be set for users with different roles, and IP address resolution will be implemented to resolve public IP and proxy IP issues. (and more)

Info:

• For More Check the Design Document, referenced below.

Deliverables

Feature Name	Feature ETA	Feature Status	Feature Description	Feature PR/ Discussion	Prod Ready
Rate Limiting Rules: Research, Analysis, and Identifying Acceptable Rates	14/05/2023	IN_PROGRESS	This feature involves researching, identifying, and coming up with rate-limiting rules for common rate limiters and a few routes for testing and debugging while developing. Already build solutions references will be taken for identifying how they came up with the solution.	Proposed: Using newrelic to come up with rate limit parameters and their values	N
Route-Based Rate Limiting: Common Rate Limiter Middleware	18/05/2023	IN_PROGRESS	This feature involves: setting up the node-rate-limiter-flexible package, using in-process memory store(temp), getting rate limiting rules from variables stored in process memory (temp).	PR: Setting Up Rate Limitier	N
Route-Based Rate Limiting: Route Specific Rate Limiter Middleware	24/05/2023	IDLE	This involves implementing the solution for multiple route-based rate limiters, switching between rate limiters based on the request and again using in-process memory store and environment variables temporarily.	--	N
Redis: Database for Storing Rate Limiting Rules	30/05/2023	IDLE	Setting up Redis hash database for storing rate limiting rules, config helper functions/classes for creating rules following the template.	--	N
Redis: Notifying Backend and Rate Limiter Middleware Of Changes	03/06/2023	IDLE	Setting up Redis pub/sub for notifying backend and rate-limiter middleware on create/update/delete of rate limiting rules and redeploying the route-based rate-limiters.	--	N
Redis: Cache Service for Storing Rate Limiter Middleware Store(s)	11/06/2023	IDLE	Setting up Redis cache service as a memory store for rate limiters stores(route-based rate limiting) for maintaining consistency across different instances if in case auto-horizontal-scaled by deployment-service(railway.app) provider.	--	N
Client Unique Identifier and Builder: IP Address Resolution	16/06/2023	IDLE	Helper functions/class to identify and resolve client-unique IP addresses behind a public IP.	--	N
Client Unique Identifier and Builder: IP Address Resolution	22/06/2023	IDLE	Helper functions/class to identify and resolve client-unique IP addresses behind proxy IP. Using headers x-forwarded-for, port, and subnetting.	--	N
Client Unique Identifier and Builder: UserId Resolution	26/06/2023	IDLE	Helper functions/class to identify and resolve client-unique userId.	--	N
Client Unique Identifier and Builder: Miscellaneous	30/06/2023	IDLE	Helper function/class to identify and resolve client-unique-identifier(cuid or client-uid) based on a parameter named type (ipaddress, userId, both) passed.	--	Y
Role-Based Rate Limiting: Identifying Acceptable Rate	05/07/2023	IDLE	Defining and Identifying, different rate-limiting rules for users with different roles, such as app_owner, super_user, member, and user.	--	N
Role-Based Rate Limiting: Implementation	08/07/2023	IDLE	Implementing Different rate-limiting rules for users with different roles, such as app_owner, super_user, member, and user.	--	Y
Dashboard Site: Design For Configuring Rate Limiting Rules	To Be Calculated	IDLE	To Be Added	--	N
Dashboard Site: Design/UI Implementation	To Be Calculated	IDLE	To Be Added	--	N
Dashboard Site: CRUD operations Implementation and Integrating Redis Service, Backend and Rate Limiting Middleware	To Be Calculated	IDLE	To Be Added	--	Y

[Note:]

• Feature ETAs are based on the date to push PR for review.
• Time To Review, Change and Approve is considered 2-3 in the best situation. Hence, the next ETAs are calculated and may be subject to change.

[Info]:

• Prod Ready Meaning: Usable current and previous features in production.

References

• Initial Issue(closed)
• Initial PR(closed)
• Document(with rough plan)
• Design Document

[vikhyat187]

Hey @shmbajaj can we add further breakdown of tasks which are likely to consume more days, like there is a task which estimate is 11days. Also we can divide the tasks and get them built parallel.

[bajajcodes]

Hey @shmbajaj can we add further breakdown of tasks which are likely to consume more days, like there is a task which estimate is 11days. Also we can divide the tasks and get them built parallel.

Hey, @vikhyat187 I have a breakdown of the tasks dashboard feature. FYI:

• I have kept the ideal work duration to be 2-3hrs/day of work.
• For the ETA's, 1 day is reserved as a buffer.
• Few tasks can be done in parallel, like identifying numbers for rate limiting and such(correct if wrong).

[prakashchoudhary07]

Nice breakdown of tasks super cool, well documented 🚀

[bajajcodes]

Rate Limiting Rules: Research, Analysis, and Identifying Acceptable Rates

This feature involves researching, identifying, and coming up with rate-limiting rules for common rate limiters and a few routes for testing and debugging while developing. Already build solutions references will be taken for identifying how they came up with the solution.

Problem Breakdown

• Researching, identifying, and coming up with rate-limiting rules for common rate limiters and a few routes for testing and debugging while developing.
• Referencing already build solutions for identifying how they came up with the solution and rate limiting parameters and values.

Researching, identifying, and coming up with rate-limiting rules for common rate limiters and a few routes for testing and debugging while developing.

To come up with rate-limiting parameters(Rate (Requests / Period), Counting characteristics, Duration) values for global rate limit and per-API rate limit, newrelic APM(Application Performance Monitoring) can be used.

Here's how newrelic APM can help:

• Step 1: Install and Configure New Relic APM (ALREADY__DONE)
• Step 2: Monitor and Analyze Traffic Data
  • Get insights into the usage patterns, request volumes, and response times of API endpoints.
  • Analyzing the traffic data to identify frequent endpoints or high-traffic routes. This analysis will help to understand the overall usage of API and identify areas where rate limiting may be necessary.
• Step 3: API Discovery (will get more on this understanding Cloudfare API Discovery)
• Step 4: Determine Rate Limiting Parameters
  • API-level global rate limit (have to think about this, will come up later on this)
  • Key-level global rate limit
  • Key-level per-API rate limit
• Step 5: Continuously Monitor and Adjust
  • Periodically review and analyze the effectiveness of rate-limiting parameters.
  • Monitor API's performance, response times, and request volumes to ensure that the rate limits are effective in preventing abuse and maintaining a good user experience.
  • Adjust the rate-limiting parameters as needed based on real-world traffic.

INFO:

• Reasoning for choosing newrelic, as it's already used in the backend.

By leveraging New Relic APM and following these steps, can practically analyze traffic data, and determine API rate limiting parameters at different levels.

Referencing already build solutions for identifying how they came up with the solution and rate limiting parameters and values.

• Tried researching and exploring existing practices, but determining rate-limiting parameters depends on factors like API discovery and analyzing past traffic.
• But, I could not find how others(industry solutions) approached rate limiting and used their solutions as references to come up with parameters and values.

References

• Cloudfare: Rate Limiting Rules
• Cloudfare: API Discovery
• Tyk API Management 101: Rate Limiting
• How do you choose the right rate limits parameters, such as time interval, request limit, and scope?
• AWS: How do I apply a rate limit on a specific request parameter or URI in AWS WAF?

[vikhyat187]

LGTM