search

Ge0rg3 / requests-ip-rotator

A Python library to utilize AWS API Gateway's large IP pool as a proxy to generate pseudo-infinite IPs for web scraping and brute forcing.
https://pypi.org/project/requests-ip-rotator/
GNU General Public License v3.0
1.33k stars 138 forks source link

Reuse gateways without restarting #69

Closed deeeeeelan closed 5 months ago

deeeeeelan commented 5 months ago

Are there any way i can use the ApiGateway object created from my local machine on my Lambda without invoking the gateway.start() method ? reason on Lambda currently its quite slow, i just want to use it without having an error on endpoints, this is my snippet that i tested on my local.

google.com is just a dummy target btw

`

from requests_ip_rotator import ApiGateway adapter = ApiGateway("https://www.google.com", regions=["us-west-1", "us-west-2"],access_key_id="xxxxx", access_key_secret="xxxxx") session = requests.Session() session.mount('https://www.google.com', adapter) res = session.get("https://www.google.com/blabla") Traceback (most recent call last): File "", line 1, in File "/usr/local/lib/python3.11/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/requests_ip_rotator/ip_rotator.py", line 56, in send endpoint = choice(self.endpoints) ^^^^^^^^^^^^^^ AttributeError: 'ApiGateway' object has no attribute 'endpoints' `

Ge0rg3 commented 5 months ago

Hey @deeeeeelan, thanks for the issue.

You can pregenerate your endpoints like:

import json
from requests_ip_rotator import ApiGateway

gateway = ApiGateway("https://site.com")

endpoints = gateway.start()
print(json.dump(endpoints))

Then store these endpoints and do not close the gateway.

In your lambda, you can then do:

from requests_ip_rotator import ApiGateway

gateway = ApiGateway("https://site.com")

gateway.start(endpoints=[...])  #replace with your endpoints

Then there will be no delay.

Hope this helps

deeeeeelan commented 5 months ago

cool @Ge0rg3 , thanks for your prompt reply. its working at the moment, but im still getting kicked with captcha by my target currently haha. still lurking around for a scrape project to bypass this site captcha/protection.

Ge0rg3 commented 5 months ago

Hahah its tough, feel free to message me privately if you want some help, my email is on https://georgeom.net/