stuartleeks / aoai-simulated-api

An exploration into creating a simulated API implementation for Azure OpenAI (AOAI)
MIT License
11 stars 4 forks source link

Add PAYG behaviour for rate-limiting #31

Open stuartleeks opened 3 months ago

stuartleeks commented 3 months ago

When there are a high volume of requests, the PAYG endpoints exhibit some queueing behaviour (i.e. they don't return immediate 429 responses).