rate limits

[ghost]

I think the repo seems to be making too many API calls to OpenAI in successive fashion. Any plans to to rate limit this?:

openai.error.RateLimitError: The server is currently overloaded with other requests. Sorry about that! You can retry your request, or contact us through our help center at help.openai.com if the error persists.

[swyxio]

i actually just turned on the concurrency setting in Modal for that when I had an issue with it. @talboren handled it in the application logic which i guess i could also use

[ghost]

Set concurrency to 5, still hitting their rate limits :(

[dhilgarth]

I'm running into this rate limit:

openai.error.RateLimitError: Rate limit reached for default-gpt-4 in organization org-XXX on tokens per min. Limit: 40000 / min. Please try again in 1ms. Contact us through our help center at help.openai.com if you continue to have issues.

[swyxio]

too powerful! let me see if i can get an alternative provider

[JacobFV]

Same. Running w/o modal solves this issue

[saxjonas]

Same here, rate does seem to be one of the key issues that runs up a bill as one has to start over again if something goes wrong. I'm sure some fine tuning of parameters could solve this in the long run:)

Thanks for providing a potential interim solution @JacobFV !

[isuryanarayanan]

RateLimitError: You exceeded your current quota, please check 
your plan and billing details.

I received the above error, I used a fresh openai account to create a token and still, it wont work.

[abhi267266]

after trying different setting this worked for me ''' @stub.function( image=openai_image, secret=modal.Secret.from_dotenv(), retries=modal.Retries( max_retries=5, backoff_coefficient=2.0, initial_delay=20.0, ), concurrency_limit=20, timeout=1200, ) ''' don't ask me why it worked (becouse i don't know ) *i am have not set any pament plans in opne ai api hope it helps

[swyxio]

we'll be moving default off of modal. thanks for the reports!