search

ustayready / fireprox

AWS API Gateway management tool for creating on the fly HTTP pass-through proxies for unique IP rotation
GNU General Public License v3.0
1.93k stars 270 forks source link

429 - Too Many Requests #16

Closed alexvasiu closed 2 years ago

alexvasiu commented 4 years ago

We tried to use it for google but when we are doing continuous requests for web crawling we end und up with 429 - Too Many Requests. How often is the IP address rotating?

ustayready commented 4 years ago

Rotates every request. Make sure you are spoofing the X-Forwarded-For header. Not sure context, whatever you are doing is probably being fingerprinting on something else. Setup a simple web server and point FireProx to it then view the request headers.