search

Ge0rg3 / requests-ip-rotator

A Python library to utilize AWS API Gateway's large IP pool as a proxy to generate pseudo-infinite IPs for web scraping and brute forcing.
https://pypi.org/project/requests-ip-rotator/
GNU General Public License v3.0
1.36k stars 140 forks source link

Requests to subdomains #70

Closed nzrv closed 6 months ago

nzrv commented 6 months ago

Hi there! Thanks for a great library! I have a question or maybe a feature request. Is it possible to make requests to a website with different subdomains without registering multiple gateways? https://coolwebsite.com https://subdomain.coolwebsite.com https://anothersubdomain.coolwebsite.com etc…

Ge0rg3 commented 6 months ago

Hi @nsrv, sadly this isn't possible through the AWS Api Gateway method currently, as far as I know. Would love to be corrected if somebody can find a way : )

contrasting commented 2 months ago

Hi @Ge0rg3 , despite you saying it isn't possible, requests made to a subdomain still seem to work when I tried it?

In fact, even if I create a gateway with a completely random domain, e.g. ApiGateway("https://foo.com"), requests to another domain https://bar.com still seem to go through.

Am I doing something wrong? Is the site passed to the constructor just a label?

Edit: nvm, did a bit more digging into the requests library, it seems the session will simply not pick ApiGateway as an adapter if the host doesn't match. So the requests made actually aren't proxied at all.

Ge0rg3 commented 2 months ago

Hey @contrasting, yeah the README needs some tweaks -- the first parameter in ApiGateway only needs to be the protocol (either "http://" or "https://"), the rest is redundant.

It's interesting that subdomains are working though, I'll check that

contrasting commented 2 months ago

Sorry - I meant to say that, while the request to the subdomain will be made, it won't be proxied, because the requests session won't pick the adapter with the correct prefix (aka ApiGateway) to use when sending the request.