search

jlewi / foyle

Foyle is a copilot to help developers deploy and operate their applications.
https://foyle.io
Apache License 2.0
106 stars 9 forks source link

Retrying on 429 obscures OpenAI quota limit #82

Open jlewi opened 6 months ago

jlewi commented 6 months ago

It looks like you will get a 429 from OpenAI if you hit the quota limit. This would normally return an openai.APIError with code "insufficent_quota" and value "You exceeded your current quota, please check your plan and billing details. For more information on this error, read the docs: https://platform.openai.com/docs/guides/error-codes/api-errors."

However since we are using hashicorp's retryable client. It keeps retrying until it exhausts the retry. It then returns a Url.Error with message POST https://api.openai.com/v1/embeddings giving up after 5 attempt(s)

So our retryable client is swallowing the openai Error which contains more informative information. We should probably customize the retry function to not retry in that case.

jlewi commented 5 months ago

I think retrying on 429 is the right thing to do because in some cases quota might replenish.

I think what we might want to do is log the 429 in case that contains additional information. Do we have to log each retry or is there someway to get the final error when the client gives up.