KenyonY / openai-forward

🚀 大语言模型高效转发服务 · An efficient forwarding service designed for LLMs. · OpenAI API Reverse Proxy
https://api.openai-forward.com
MIT License
818 stars 282 forks source link

Enable rate limiting #53

Closed michaelfeil closed 1 year ago

michaelfeil commented 1 year ago

Have you considered adding a rate limiter? I have made good experiences with: https://pypi.org/project/fastapi-limiter/

In this case,

from fastapi_limiter import FastAPILimiter
from fastapi_limiter.depends import RateLimiter

@app.on_event("startup")
async def startup():
    redis = redis.from_url("redis://localhost", encoding="utf-8", decode_responses=True)
    await FastAPILimiter.init(redis)

and

app = FastAPI(dependencies=[RateLimiter(times=2, seconds=5))]) 

. If you want, might also be added as optional, or if redis is offline:

# for forward compatability:
if os.environ.get("RATE_LIMIT", False):
    ...
KenyonY commented 1 year ago

It looks great. fastapi-limiter is a nice library, but it requires Redis . Perhaps https://github.com/laurentS/slowapi is also a good alternative?

michaelfeil commented 1 year ago

Also a great choice, seems to have redis support and support for local filesystem.

KenyonY commented 1 year ago

🎉🎉 In version v0.5.0, the rate limit functionality has been implemented. Thank you for your suggestion.