hitting rate limit with an API key after < 50 requests

[alissapajer]

According to the documentation, I should be able to make 240 requests per minute with an API key. But, I hit a rate limit after just a few requests.

How to reproduce:

for i in {1..10}; do curl "https://api.fda.gov/animalandveterinary/event.json?api_key=<REDACTED>&limit=100&skip=0&search=animal.species:Cat+AND+primary_reporter.exact:Veterinarian" >> d.txt; done

Save that as script.sh and then run

$ ./script.sh & ./script.sh & ./script.sh & ./script.sh & ./script.sh

This makes a total of 50 requests in a few seconds. In the output file d.txt I see the following message 7 times:

{
  "error": {
    "code": "OVER_RATE_LIMIT",
    "message": "You have exceeded your rate limit. Try again later or contact us at https://api.fda.gov:443/contact/ for assistance"
  }
}

Additionally, there are 43 successful request responses. I would expect 50 successful responses.

I am not able to reproduce this if I run 50 sequential requests, like so

$ for i in {1..50}; do curl "https://api.fda.gov/animalandveterinary/event.json?api_key=<REDACTED>&limit=100&skip=0&search=animal.species:Cat+AND+primary_reporter.exact:Veterinarian" >> e.txt; done

So, it appears that too many requests within a small time window are causing the rate limit error.

For the bigger picture context, my application is querying this endpoint repeatedly, incrementing the skip variable by 100 until the total is reached. In the case of this query, the total is 13032, so it would take 131 requests. I am seeing a rate limit in my application, and I was able to construct the above reproduction using curl.

[alissapajer]

Also, I wanted to add that the 7 rate limit errors are spread throughout the results, meaning that there are successful requests immediately after rate limited ones.

[rintcius]

FWIW I'm not getting rate limited when doing

./script.sh & ./script.sh & ./script.sh & ./script.sh & ./script.sh

but when I repeat 10 instead of 5 times (i.e. with 100 reqs) I do get rate limited.

[beckyconning]

So is the actual limit per second rather than per minute?

[dkrylovsb]

Fixed. The rate limits are now being enforced exactly as described on the website:

• With no API key: 40 requests per minute, per IP address. 1000 requests per day, per IP address.
• With an API key: 240 requests per minute, per key. 120000 requests per day, per key.

[rintcius]

Thanks for the quick fix @dkrylovsb !