search

abartov / bybeconv

Project Ben-Yehuda's content management system.
https://benyehuda.org/
Other
10 stars 5 forks source link

getting 429 too many request #221

Closed eyaler closed 8 months ago

eyaler commented 8 months ago

hi! my scraper started to get 429 too many requests. is there some new rate limit? i found this. i guess this should be advertised some where (e.g. https://benyehuda.org/api-docs/index.html).

also if i understand correctly:

 # Throttle requests to 5 requests per second per IP
  throttle('req/ip', limit: 300, period: 5.minutes) do |req|

it seems you wanted 5 req per second, but is it now not 300 req per 5 minutes = 60 req per 1 minute = 1 req per second??

this is really painful for my scraper. can we have say... 3000 per 1 minute? pretty please?

btw, i am still getting 429 when doing 1 req/sec so i guess there is more going on?

abartov commented 8 months ago

The comment is out of date. I changed it to 300 per 5 minutes because scraping and web-crawlers were encumbering the site for human readers. So no, I am not going to raise it. You have the public dump at your disposal. The API is not meant for accommodating a full site scrape.