seleniumbase / SeleniumBase

📊 Python's all-in-one framework for web crawling, scraping, testing, and reporting. Supports pytest. UC Mode provides stealth. Includes many tools.
https://seleniumbase.io
MIT License
4.45k stars 908 forks source link

How is it possible to run multiple threads, with diff proxys #2860

Closed Akshay-Shingala closed 2 weeks ago

Akshay-Shingala commented 2 weeks ago

def access_site(cmd): os.system(cmd)

urls = [ 'https://example.com', 'https://example.com', ['https://example.com' ]

processes = [] for url in urls: cmd = f'pytest w.py --proxy=USER_NAME:PASSWORD@pr.oxylabs.io:7777 --gui --uc --no-summary -q' process = thread.Thread(target=run_test, args=(cmd, )) processes.append(process) process.start() for i in processes: i.join()

this is my code

mdmintz commented 2 weeks ago

Duplicate of https://github.com/seleniumbase/SeleniumBase/discussions/2835 / https://github.com/seleniumbase/SeleniumBase/issues/2828#issuecomment-2147511383

See the example that uses ThreadPoolExecutor with UC Mode:

import sys
from concurrent.futures import ThreadPoolExecutor
from seleniumbase import Driver
sys.argv.append("-n")  # Tell SeleniumBase to do thread-locking as needed

def launch_driver(url):
    driver = Driver(uc=True)
    try:
        driver.get(url=url)
        driver.sleep(2)
    finally:
        driver.quit()

urls = ['https://seleniumbase.io/demo_page' for i in range(3)]
with ThreadPoolExecutor(max_workers=len(urls)) as executor:
    for url in urls:
        executor.submit(launch_driver, url)

Or use pytest multithreading with pytest-xdist and the parameterized library: https://github.com/seleniumbase/SeleniumBase/discussions/2394#discussioncomment-7969332

from parameterized import parameterized
from seleniumbase import BaseCase
BaseCase.main(__name__, __file__, "-n3")

class ProxyTests(BaseCase):
    @parameterized.expand(
        [
            ["host1:port1"],
            ["host2:port2"],
            ["host3:port3"],
        ]
    )
    def test_multiple_proxies(self, proxy_string):
        self.get_new_driver(
            undetectable=True, proxy=proxy_string, multi_proxy=True
        )
        self.driver.get("https://browserleaks.com/webrtc")
        self.sleep(30)

multi_webrtc_checks