Retry strategies for OpenAI RateLimitError

[clstaudt]

For my representation model

bertopic.representation.OpenAI(
        model="gpt-35-turbo", 
        chat=True,
        #delay_in_seconds=1,
        generator_kwargs = {"engine": "gpt-35-turbo", "temperature": 0.1},
        prompt=f"""
                Output a concise, English, lowercase topic label for the following keywords. Output only the label, no punctuation. Prefer single terms. If you are unable to perform the task, output: None. 
                [KEYWORDS]
            """
    )

I am sometimes hitting a RateLimitError without exactly knowing what causes it (seems to happen when training on larger datasets of > 100 000 documents).

Setting a waiting time of even one second between API calls increases the training time several times (not sure why).

• Would a different strategy be possible that catches a RateLimitError when it occurs and then adapts?
• If the RateLimitError is predictable (e.g. depending on dataset size) - is it avoidable?

[MaartenGr]

Would a different strategy be possible that catches a RateLimitError when it occurs and then adapts?

You could use exponential_backoff for that in OpenAI:

    exponential_backoff: Retry requests with a random exponential backoff. 
                         A short sleep is used when a rate limit error is hit, 
                         then the requests is retried. Increase the sleep length
                         if errors are hit until 10 unsuccesfull requests. 
                         If True, overrides `delay_in_seconds`.

If the RateLimitError is predictable (e.g. depending on dataset size) - is it avoidable?

It depends on the number of clusters you created since there will be one call for each cluster to create a label. If you have many clusters and you set it to a couple of seconds delay, then it will be the number of seconds times the number of clusters.