search

JohnSnowLabs / langtest

Deliver safe & effective language models
http://langtest.org/
Apache License 2.0
488 stars 36 forks source link

Rate Limit feature for Azure/OpenAI results generation #988

Open Jeff-ZYX opened 6 months ago

Jeff-ZYX commented 6 months ago

Is your feature request related to a problem? Please describe. Current architecture doesn't seem to support API instances with inbuilt rate limits, causing errors in generation

Describe the solution you'd like A sleep function that can be configured to change rates of generation of GPT completions.

chakravarthik27 commented 6 months ago

Hi @Jeff-ZYX

We acknowledge and appreciate the information you brought to our attention. Currently, we do not have a built-in rate restriction capability in our architecture for API instances. We understand that controlling the rate of generation is crucial to prevent errors and ensure error-free operation for GPT completions. We thank you for recommending the addition of a programmable sleep function to efficiently regulate the generating rate. We value your input and will prioritize this feature request for future updates, even though we are unable to offer a quick fix.