search

groq / groq-python

The official Python Library for the Groq API
Apache License 2.0
385 stars 28 forks source link

Rate limit use case #18

Closed fredzannarbor closed 6 months ago

fredzannarbor commented 7 months ago

Hi,

I got the Python client running with an existing application but am receiving lots of 429 errors. I realize that this is early days but would love to have the rate lifted a bit so I can complete some test jobs. Thought it may help to describe my use case.

I own a book publishing company and have a suite of applications that automatically analyze books on a page by page basis and collate the results into various artifacts that are useful for book publishers; I also have some that create entire new books. Because of the page by page structure these apps can run between 100 and 1000 requests, typically under 500 tokens each. I have considered batcjomg multiple pages into single requests but I am early in the process of scaling so simplicity & reliably parsed results are more important to me than raw efficiency; same with async. I do expect to remedy these issues eventually.

So I run these page by applications quite a bit while I code in test mode (fewer pages/calls), maybe several times an hour, but I only do complete production runs maybe 10x day. This level of activity is quite affordable for gpt-3.5, a dollar or two a day, so I have been assuming groq will wind up priced the same.

I would really like to be able to do complete runs of large books right away because that is what I need to test the speed improvements and to get the "wow" factor for demos. Any help/guidance would be much appreciated.

hozen-groq commented 7 months ago

Hi, Fred! On Friday, we bumped up the free rate limit to 20 queries per minute. 🚀 We're currently working tirelessly to set up a billing page for paid tiers/rates for the API - there has been a huge demand for it and we are hoping to get users up and running soon!

For pricing, please see here. For your use case, since you're trying to incorporate Groq into your demos as soon as possible, I've fielded your question to the core team to see if there's anything we can do to speed up this process.

In the meantime, we greatly appreciate your patience and interest in Groq. I'll update you as soon as I have a better answer!

fredzannarbor commented 7 months ago

Thanks, that all makes sense. As an FYI, if the rate limit issues clear up in the next week or two, I'm planning to hit the "books built with groq" angle pretty hard in all my networks, as this (short) period when adoption is just taking off is a great time to cut thru the clutter.

gradenr commented 6 months ago

Our rate limit has been upped to 30 requests per minutes. https://console.groq.com/docs/rate-limits

We are working on adding a paid tier which will have higher rate limits.