seleniumbase / SeleniumBase

📊 Python's all-in-one framework for web crawling, scraping, testing, and reporting. Supports pytest. UC Mode provides stealth. Includes many tools.
https://seleniumbase.io
MIT License
4.5k stars 916 forks source link

Documenting the `selenium-stealth` integration with UC Mode #2404

Open mdmintz opened 6 months ago

mdmintz commented 6 months ago

Documenting the selenium-stealth integration with UC Mode

(Info: selenium-stealth lets you mask your web browser's fingerprint and WebGL information.)

Prerequisites: selenium-stealth must be installed separately: pip install selenium-stealth

Here's an example of the selenium-stealth integration with the Driver() manager and UC Mode:

from seleniumbase import Driver
from selenium_stealth import stealth

driver = Driver(uc=True)
stealth(
    driver,
    languages=["en-US", "en"],
    vendor="Google Inc.",
    platform="Win32",
    webgl_vendor="Intel Inc.",
    renderer="Intel Iris OpenGL Engine",
    fix_hairline=True,
)
driver.get("https://browserleaks.com/webrtc")
driver.sleep(10)
driver.quit()

selenium-stealth also integrates with SeleniumBase BaseCase formats:

Use Syntax Format 2 to override the setUp() method so that selenium-stealth settings are used:

from seleniumbase import BaseCase
from selenium_stealth import stealth

class BaseTestCase(BaseCase):
    def setUp(self):
        super().setUp()
        stealth(self.driver,
            languages=["en-US", "en"],
            vendor="Google Inc.",
            platform="Win32",
            webgl_vendor="Intel Inc.",
            renderer="Intel Iris OpenGL Engine",
            fix_hairline=True,
        )

Then have your test classes inherit BaseTestCase instead of BaseCase. (See SeleniumBase/help_docs/syntax_formats.md#sb_sf_02 for details.)

To activate UC Mode in BaseCase formats, use --uc as a pytest command-line option:

pytest --uc

For general information about selenium-stealth, see:


For general information about UC Mode, see https://github.com/seleniumbase/SeleniumBase/issues/2213

AlexPaiva commented 6 months ago

Stealth hasn't been updated in a while, is it still viable?

mdmintz commented 6 months ago

@AlexPaiva Some people are still using it, so it's probably still valid, even though it hasn't been updated in awhile.

But at this point, selenium-stealth is mainly just for fingerprint-masking. The part with avoiding selenium-detection is fully being done by seleniumbase UC Mode.

xipeng5 commented 1 month ago

@ AlexPaiva有些人仍然在使用它,所以它可能仍然有效,即使它有一段时间没有更新。

但在这一点上,selenium-stealth主要是为了指纹掩蔽。 避免硒检测的部分完全由seleniumbaseUC模式完成。

I think in this case, although the fingerprint has been changed, the fingerprint is fixed every time. If only it were random fingerprints every time

xipeng5 commented 1 month ago

Thank you very much for your reply. Your library has benefited me greatly. Sincerely thank you, but I have encountered some issues. I tried to register an account using your library, but every time it failed. I used a proxy IP and set the browser's fingerprint and other parameters, but none of them worked. I don't know where the problem is. If it doesn't take up your time, I hope you can help me take a look at the code. Below is the code snippet link. Thank you very much and I wish you a happy life。https://github.com/xipeng5/testap/blob/main/test.py

kitsune0n commented 2 weeks ago

how use stealth with SB(from seleniumbase import SB) and uc=True? adding

stealth(sb.driver,
         languages=["en-US", "en"],
         vendor="Google Inc.",
         platform="Win32",
         webgl_vendor="Intel Inc.",
         renderer="Intel Iris OpenGL Engine",
         fix_hairline=True,
        )

don't works

mdmintz commented 2 weeks ago

@kitsune0n If the driver needs to disconnect (eg, a uc_open method is called) or if going to a site with a Cloudflare CAPTCHA (eg. via driver.get(), where the disconnect is automatic) then selenium-stealth settings are reset. Unless the maintainer of https://github.com/diprajpatra/selenium-stealth makes changes, there's nothing I can do from my end to help.

Also, selenium-stealth is not needed to bypass CAPTCHAs: Regular seleniumbase UC Mode can do that alone.

PankajSavaliya commented 1 week ago

I have used selenium-stealth, but in UC mode, there is always a need for connect and reconnect methods. Do I need to apply stealth each time? because as you said it resets if I use uc_open or uc_click or any disconnect method use. Is there any way to always attach again and all the info with any customize?

 with SB(uc=True) as sb:
        stealth(
            sb.driver,
            user_agent=device.get('user_agent'),
            languages=["en-US", "en"],
            vendor=device.get('vendor'),
            platform=device.get('platform'),
            webgl_vendor=device.get('webgl_vendor'),
            renderer=device.get('renderer'),
            fix_hairline=device.get('fix_hairline'),
        )
        sb.set_window_size(width=device.get('width'), height=device.get('height'))
        sb.uc_open('https://amiunique.org/')
        time.sleep(50)

In SB there is agent set method but not for timezone, we can set the timezone external raw script but it reset while disconnecting the driver, please let me know if there is a change to customize SB to override the method of reconnect, as I have used SB context-manager base so I need to customize uc_open, uc_click and reconnect method so I can execute timezone method each time

sb.execute_cdp_cmd('Emulation.setTimezoneOverride', tz_params)
mdmintz commented 1 week ago

https://github.com/seleniumbase/SeleniumBase/discussions/2856#discussioncomment-9789603:

When UC Mode disconnects the driver from Chrome to evade detection, it also resets any settings made by selenium-stealth.

Also, it appears that using selenium-stealth to modify settings will get detected as a possible attempt to mask your fingerprint. (There is little that I can do there, as that is an external library.)

Your best bet to avoid detection is by just using regular SeleniumBase UC Mode:

from seleniumbase import SB

with SB(uc=True, incognito=True) as sb:
    sb.uc_open_with_reconnect("https://pixelscan.net/", 10)
    breakpoint()

If you want timezone changes to be permanent, use a Chrome extension that changes your timezone.

PankajSavaliya commented 1 week ago

#2856 (comment):

When UC Mode disconnects the driver from Chrome to evade detection, it also resets any settings made by selenium-stealth.

Also, it appears that using selenium-stealth to modify settings will get detected as a possible attempt to mask your fingerprint. (There is little that I can do there, as that is an external library.)

Your best bet to avoid detection is by just using regular SeleniumBase UC Mode:

from seleniumbase import SB

with SB(uc=True, incognito=True) as sb:
    sb.uc_open_with_reconnect("https://pixelscan.net/", 10)
    breakpoint()

If you want timezone changes to be permanent, use a Chrome extension that changes your timezone.

Yes, I want to change the timezone permanently, I have already removed that library because it's not updated.

Can you please how to do with extension with SB? just give me a hint as we can install dynamically in driver but how? @mdmintz

mdmintz commented 1 week ago

@PankajSavaliya For changing the timezone, you'll have to find an extension that does it. I hear they exist, but I don't know one off the top of my head directly.

PankajSavaliya commented 1 week ago

How to add the extension in SB if I have an extension so how to give the path of extension to SB? @mdmintz

mdmintz commented 1 week ago

Adding an extension: https://github.com/seleniumbase/SeleniumBase/blob/1c526f1c02a9c7f4ec538d1600829acff44aa05e/seleniumbase/plugins/sb_manager.py#L74

PankajSavaliya commented 1 week ago

@mdmintz I have tried a lot can you please help me find this kind of custom extension? I have been trying for 6 hours but no luck finding for set to the timezone and datetime via extension.

PankajSavaliya commented 1 week ago

@mdmintz

I have created an extension and set the path to extension_dir, and it is working perfectly. However, there is one issue: in my extension, I'm using the below command in the script, and due to this debugger command, this window continues to show up. Is there a way to hide it and make it undetectable?

chrome.debugger.sendCommand({ tabId: tabId }, 'Emulation.setTimezoneOverride'

debugging