search

comet-ml / opik

Open-source end-to-end LLM Development Platform
Apache License 2.0
2.12k stars 132 forks source link

[OPIK-68] Add rate limit #261

Closed thiagohora closed 1 month ago

thiagohora commented 2 months ago

Details

Add rate limit functionality.

Added @RateLimited annotation to apply rate limit to an endpoint.

The following properties control the limit buckets (For now, only one generalEvents)

rateLimit:
    enabled: ${RATE_LIMIT_ENABLED:-false}
    generalLimit:
      limit: ${RATE_LIMIT_GENERAL_EVENTS_LIMIT:-5000}
      durationInSeconds: ${RATE_LIMIT_GENERAL_EVENTS_DURATION_IN_SEC:-1}

In the case of batch operations, an interface called RateEventContainer allows the interceptor to know the number of items in the batch.

When enabled, the rate limit is communicated via headers and status 429 (TOO_MANY_REQUESTS) when the limit is reached.

    Opik-User-Limit: general_events
    Opik-User-Remaining-Limit: 0
    Opik-User-Remaining-Limit-TTL-Millis: 300

Issues

OPIK-68

Testing

Tested locally via Minikube setup and also via Component Tests in the pipeline.

thiagohora commented 1 month ago

Sorry, I found a typo