search

chroma-core / chroma

the AI-native open-source embedding database
https://www.trychroma.com/
Apache License 2.0
14.82k stars 1.24k forks source link

[ENH]: OpenAI Rate Limiting #1948

Open johnisanerd opened 6 months ago

johnisanerd commented 6 months ago

What happened?

In Python, I ran a collection.add on a set of documents about ~10,000. Our account is rate limited to 10,000 RPM. We kept getting rate limited when we added a large set of documents.

We solved this by slicing up the list, and adding in a crude sleep. A more elegant solution would be greatly appreciated!

Versions

chromadb==0.4.24 Python 3.11.6

Relevant log output

Don't have any log output.
nicolasgere commented 6 months ago

Hi @johnisanerd, that's a great feature idea, and it could indeed be a part of Chroma. We don't have immediate plans to work on it in the short term, but it's on our radar for future consideration. Thank you

johnisanerd commented 5 months ago

Is there a cookbook or somewhere I could contribute a snippet of code? I made a workaround example that requires a little bit of finesse, I could share.

tazarov commented 5 months ago

@johnisanerd, you can either submit a PR with your proposal or add it here if you want us to discuss it. We should solve this problem for all EFs, not just OpenAI. Many embedding functions rely on external API calls and are susceptible to network issues, rate limiting, etc.