search

wso2 / apk

Official Repository for APK distribution
Apache License 2.0
153 stars 90 forks source link

Support AI Backend based ratelimit #2479

Closed Tharsanan1 closed 2 hours ago

Tharsanan1 commented 2 hours ago

Problem

We are introducing AI APIs to apk. In the AI usecases its a common usecase to limit the requests based on the token counts(prompt, completion, total).

Solution

Add a feature to support ratelimiting AI requests based on the token count.

Affected Component

Adapter

Version

1.2.0

Implementation

No response

Related Issues

No response

Suggested Labels

No response

Tharsanan1 commented 2 hours ago

Feature added using this PR: https://github.com/wso2/apk/issues/2479