As the service provider, I want to sustain requests throttling quota - Githubissues

kislerdm / diagramastext

Application to streamline diagram generation using plain English instructions instead of code

https://diagramastext.dev

Apache License 2.0

24 stars 1 forks source link

As the service provider, I want to sustain requests throttling quota #63

Closed kislerdm closed 1 year ago

kislerdm commented 1 year ago

Problem

Throttling is not controlled, hence we risk to hit the OpenAI quotas limits and affect clients.

Proposed Solution

Keep track of the the user's session
Limit the rate:
- 1 request per minute for a non-registered user
- 3 requests per minute for a registered user

A user's fingerprint is defined as a result of hash function applied to

the user-agent string for non-registered users;
the user's email for registered users.

The fingerprint is stored as a custom JWT token's attribute.

Acceptance Criteria

Throttling limit is applied

DoD

Core logic caches the following:
- Call's timestamp
- Caller's fingerprint
- Caller's registration status
The in-memory cache is backed by gcs