Rate Limit problem while running python ingest.py

[pedrohdemedeiros]

Hi!

When trying to include my own notion page I am facing a ratelimit problem. I know that i can deal with it outside the system, but is there a way to limit the rate inside the ingest.py in order to solve it internally?

[hacksman]

same error

[g-adamante]

I also got this error - a simple fix is using the RateLimiter lib on the OpenAIEmbeddings class, it looked like this:

@RateLimiter(max_calls=20, period=60)
def _embedding_func(self, text: str, *, engine: str) -> List[float]:
        """Call out to OpenAI's embedding endpoint."""
        # replace newlines, which can negatively affect performance.
        text = text.replace("\n", " ")
        return self.client.create(input=[text], engine=engine)["data"][0]["embedding"]

You can find open ai's rate limits here and adjust accordingly. For the free trial, it's 20 calls/minute, which makes things sloooow.

[qcgm1978]

It doesn't work even adding @RateLimiter(max_calls=20, period=60)

The error message

Rate limit reached for default-global-with-image-limits in organization org-EwRRwtrDgN3yDxZ9dBI4r3eT on requests per min. Limit: 60 / min. Please try again in 1s. Contact us through our help center at help.openai.com if you continue to have issues. Please add a payment method to your account to increase your rate limit. Visit https://platform.openai.com/account/billing to add a payment method.