seleniumbase / SeleniumBase

📊 Python's all-in-one framework for web crawling, scraping, testing, and reporting. Supports pytest. UC Mode provides stealth. Includes many tools.
https://seleniumbase.io
MIT License
4.95k stars 938 forks source link

Calling enable_cdp_events, using --uc | uc=True #1687

Closed MattWaller closed 1 year ago

MattWaller commented 1 year ago

Firstly, well done with this library. Crazy how much you've coded into this.

How can enable_cdp_events | https://github.com/seleniumbase/SeleniumBase/blob/master/seleniumbase/undetected/__init__.py#L57) be called during constructor?

from seleniumbase import Driver

Driver.code.co_varnames

('browser', 'headless', 'headless2', 'headed', 'locale_code', 'protocol', 'servername', 'port', 'proxy', 'proxy_bypass_list', 'proxy_pac_url', 'agent', 'cap_file', 'cap_string', 'recorder_ext', 'disable_js', 'disable_csp', 'enable_ws', 'disable_ws', 'enable_sync', 'use_auto_ext', 'undetectable', 'uc_subprocess', 'no_sandbox', 'disable_gpu', 'incognito', 'guest_mode', 'devtools', 'remote_debug', 'enable_3d_apis', 'swiftshader', 'ad_block_on', 'block_images', 'do_not_track', 'chromium_arg', 'firefox_arg', 'firefox_pref', 'user_data_dir', 'extension_zip', 'extension_dir', 'page_load_strategy', 'use_wire', 'external_pdf', 'is_mobile', 'mobile', 'd_width', 'd_height', 'd_p_r', 'uc', 'undetected', 'uc_sub', 'wire', 'pls', 'constants', 'sys_argv', 'browser_changes', 'browser_set', 'browser_text', 'browser_list', 'message', 'valid_browsers', 'use_grid', 'test_id', 'proxy_string', 'user_agent', 'recorder_mode', 'browser_name', 'browser_launcher', 'driver')

Am I missing something, or was this potentially missed when porting to SeleniumBase? Probably not many people using cdp events?

Cheers, Matt

mdmintz commented 1 year ago

Hello, thank you, and welcome!

You're right... probably not many people using CDP Events yet, and probably not possible to use that here yet unless I make a slight change. I'm thinking of setting enable_cdp_events=True for all UC Mode tests unless you see an issue with that. I ran my existing tests using that setting without issue. I ported over as much as I could from undetected-chromedriver, with improvements to anti-detection abilities.

I'm also kind of curious if uc_click() is useful at all: https://github.com/seleniumbase/SeleniumBase/blob/94e80e460259e6a406c3685eadb4203fe3abc959/seleniumbase/undetected/webelement.py#L6 It slows down tests with the reconnect, but perhaps not as useful as it seemed. (Ported over and renamed from click_safe from https://github.com/ultrafunkamsterdam/undetected-chromedriver/blob/dd0c7ca75e3627f168a6953d4f50494c5d20e71b/undetected_chromedriver/webelement.py#L7)

I'm open to suggestions if you understand UC Mode well. If switches can be permanently on without messing up existing behavior, I can make that the default setting. If not, I can add a command-line switch or a method arg so that it only goes on when called for.

Let me know, and I can ship a new release in under 24 hours as needed (I ship new releases often). And will probably look into that in the morning (almost 2:30 AM here now, so going to bed).

mdmintz commented 1 year ago

The enable_cdp_events=True slowed things down slightly during the initial performance testing, so I might consider adding a --cdp command-line option for it, and a cdp method arg for the switch. Although if the uc_click is found to be ineffective in preventing detection, I'll switch SeleniumBase clicks in UC Mode to be regular clicks instead of uc_click. That would more than offset the loss of speed caused by having enable_cdp_events to always be enabled.

MattWaller commented 1 year ago

To be honest, couldn't give you any real information about uc_click. I do use enable_cdp_events in certain modules to watch certain websockets etc / monitoring endpoints. Wouldn't suggest making enable_cdp_events as default, not sure if it would increase the likelihood of detection. But I'm not 100% certain.

mdmintz commented 1 year ago

Released in 4.11.6 - https://github.com/seleniumbase/SeleniumBase/releases/tag/v4.11.6 Example usage in examples/uc_cdp_events.py