omkarcloud / botasaurus

The All in One Framework to build Awesome Scrapers.
https://www.omkar.cloud/botasaurus/
MIT License
1.15k stars 103 forks source link

Mobile Proxy Support #18

Closed MaxAndolini closed 7 months ago

MaxAndolini commented 7 months ago

This library needs mobile proxy support which needs to run an IP changer request before the browser opens etc.

Chetan11-dev commented 7 months ago

Thank you for your suggestion. Could you share a Code Example [Like Selenium/Request] of how you would like to use this feature?

MaxAndolini commented 7 months ago
def reset_ip():
    # Reset request

def run_before():
    # Change ip after every 3 browsers for Outlook

    global last_restarted
    last_restarted += 1
    if last_restarted >= 3:
        last_restarted = 0
        reset_ip()

@browser(
    on_before=run_before
}

This might work. Also when you will update Outlook with v3?

Chetan11-dev commented 7 months ago

How will you reset_ip?

MaxAndolini commented 7 months ago

It contains the requests I sent to my own site and the requests I sent to the IP change link. Simply calling a function that runs before the browsers are opened will do the trick.

Chetan11-dev commented 7 months ago

Could you give Code Example for reset_ip?

MaxAndolini commented 7 months ago
def reset_ip():
    while True:
        try:
            response = requests.get('LINK', timeout=5)

            print(response.status_code)
            res = response.json()
            print(res)

            if response.status_code == 200:
                print('Ip changed')
                break

            time.sleep(2)

        except Exception as e:
            print(e)
            time.sleep(10)

It's simply like this, but I also have my own requests, as I said.

Chetan11-dev commented 7 months ago

Does it solve proxy problem

from botasaurus import *

def get_proxy(data):
    if data["username"] == "mahendra-singh-dhoni":
        return "http://proxy1.example.com:8000"
    elif data["username"] == "virender-sehwag":
        return "http://proxy2.example.com:8000"
    return "http://proxy3.example.com:8000"

@browser(
    data=[{"username": "mahendra-singh-dhoni",}, {"username": "virender-sehwag", }],
    proxy=get_proxy,
)
def run_scraper(driver: AntiDetectDriver, data):
    # Navigate to the Omkar Cloud website
    driver.get("https://ipinfo.io/")

if __name__ == "__main__":
    # Initiate the web scraping task
    run_scraper()
MaxAndolini commented 7 months ago

I changed it to this and I guess worked, Also when you will update Outlook with botasaurus v3?

def get_proxy(data):
    global last_restarted
    last_restarted += 1
    if last_restarted >= 3:
        last_restarted = 0
        reset_ip()

    return 'http://' + proxy_url
Chetan11-dev commented 7 months ago

By 27th November

Chetan11-dev commented 7 months ago

Also, interested to know your thoughts on Botasaurus, Could you read readme and tell me your feedback about it?

MaxAndolini commented 7 months ago

I have been using bose-botasaurus for about 2 months already. There are a few shortcomings, for example, there is no proxy support in requests for functions such as ip_details or generate_users, and since there is no GPU in VDS, this should be noticed and an argument that turns off the GPU should be added. (--disable-gpu)

Chetan11-dev commented 7 months ago

Thanks for your feedback

Chetan11-dev commented 7 months ago

So do you think you would use it if you were to create a web scraping project?

MaxAndolini commented 7 months ago

Maybe if the stealth aspects were developed a little more, yes, you should examine and include the tls-client and hrequests libraries for this.

Chetan11-dev commented 7 months ago

Thanks!

Chetan11-dev commented 7 months ago

Hi max i won't be able to deliver outlook account generator by 27th but I am working full time on it and we'll able to delivery it soon.

Chetan11-dev commented 7 months ago

@MaxAndolini Released, Please follow these Steps.

1️⃣ Clone the Magic 🧙‍♀:

   git clone https://github.com/omkarcloud/outlook-account-generator
   cd outlook-account-generator

2️⃣ Install Dependencies 📦:

   python -m pip install -r requirements.txt

3️⃣ Let the Rain of Outlook Accounts Begin 😎:

   python main.py

And Run the following command to update packages:

python -m pip install botasaurus --upgrade