NorkzYT / Novexity

Open Source SerpAPI Google Search Alternative
MIT License
7 stars 0 forks source link

How does it work? #11

Closed rushout09 closed 1 month ago

rushout09 commented 1 month ago

Hi,

First of all thanks a lot for this repo. sounds like an amazing solution. would love to know more how it works. Like in laymen terms, how it uses aws api endpoint gateway and why.

Thanks

NorkzYT commented 1 month ago

Hello @rushout09, thank you for your interest in this repository. Below is what I believe you are looking for.

How Novexity Uses AWS API Gateway

Novexity's solution for rotating IPs works by creating API Gateway endpoints in multiple AWS regions. Here’s a simplified breakdown:


This solution makes use of AWS’s robust infrastructure, making IP rotation simple and scalable.

How Novexity Works with AWS API Gateway

  1. IP Rotation Process:
    Novexity creates API Gateway instances in different regions across AWS. Each of these instances has its own IP address. When you send requests, they are routed through these gateways, effectively using different IP addresses to avoid being blocked or throttled by services that impose rate limits or IP restrictions.

  2. AWS API Gateway Setup:
    When Novexity starts, it calls AWS to create a REST API in each region. It does this asynchronously, ensuring multiple regions can be set up in parallel. For each API, it sets up the following:

    • Resources: A resource is created that allows any path (using {proxy+}), meaning you can send any request and it will be routed through the API Gateway.
    • Methods: The HTTP methods (GET, POST, etc.) are configured for this resource. This allows the API Gateway to handle any type of request.
    • Integration: The API Gateway forwards requests to the target service, with IP rotation occurring based on the region in which the API was created.
  3. Making Requests:
    Once the API Gateways are set up, Novexity uses them to forward requests. Each request is routed through a randomly chosen API Gateway from the pool of available regions. This ensures that the IP addresses change regularly, enhancing privacy and avoiding rate limits.

  4. Cleaning Up:
    After the requests are made, Novexity can clean up by deleting the API Gateways it created, ensuring there are no lingering resources in AWS.


This system is ideal for users who need to rotate IP addresses while interacting with non-javascript data dependent web services, ensuring requests remain anonymous or avoid rate-limiting mechanisms.

Let me know if you need further clarifications and hope this helps to answer your questions.

rushout09 commented 1 month ago

Thanks.

NorkzYT commented 1 month ago

Of course, no problem.