search

Ge0rg3 / requests-ip-rotator

A Python library to utilize AWS API Gateway's large IP pool as a proxy to generate pseudo-infinite IPs for web scraping and brute forcing.
https://pypi.org/project/requests-ip-rotator/
GNU General Public License v3.0
1.33k stars 138 forks source link

Add pagination to gateway getting #32

Closed RobertLucey closed 2 years ago

RobertLucey commented 2 years ago

When there are many gateways on an account / region and the relevant ones aren't on the first page (25 per page) they start to accumulate since they may not be found to be deleted.

Ge0rg3 commented 2 years ago

Thank you! Will test this and merge shortly 😄

Ge0rg3 commented 2 years ago

Hi @RobertLucey, do you reckon this change makes more sense? https://github.com/powertohack/requests-ip-rotator/commit/e695445d2ffd4cd241565d959f1dd06fb7eaa1ff

RobertLucey commented 2 years ago

Hey @Ge0rg3,

There's 2 things that may not be right about that change

  1. According to https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/apigateway.html#APIGateway.Client.get_rest_apis the maximum limit value is 500.
  2. Capping at any large number will work for most anyone since the default aws limit on the number of gateways you can have without requesting more is 600 (I think). Iterating through the pages would be the only way to be sure to get them all
Ge0rg3 commented 2 years ago

Great, thanks for the insight here! This will be rolled out in the next release, probably ~1mo ❤️