Stevenic / alphawave

AlphaWave is a very opinionated client for interfacing with Large Language Models.
MIT License
98 stars 9 forks source link

Add retry policy to OpenAI and AzureOpenAI clients #1

Closed Stevenic closed 1 year ago

Stevenic commented 1 year ago

AlphaWave automatically returns any 429's from the client as a rate_limited error. That behavior's correct but we can minimize the effects of rate limiting by letting you configure a simple retry policy for the clients. A retryPolict: number[]; settings which takes an array of delays would let you create a simple exponential backoff policy. retryPolicy: [1000,3000,5000] would retry a request 3 times over the course of 9 seconds.

Stevenic commented 1 year ago

Complete