search

seleniumbase / SeleniumBase

📊 Python's all-in-one framework for web crawling, scraping, testing, and reporting. Supports pytest. UC Mode provides stealth. Includes many tools.
https://seleniumbase.io
MIT License
4.45k stars 908 forks source link

Improve multithreading with UC Mode 4.28.x #2883

Closed mdmintz closed 3 days ago

mdmintz commented 4 days ago

Improve multithreading with UC Mode 4.28.x

In case you missed https://github.com/seleniumbase/SeleniumBase/issues/2865, 4.28.0 added a new UC Mode method: uc_gui_handle_cf(), which uses pyautogui to click Cloudflare checkboxes with the keyboard while the driver is disconnected from Chrome. For those of you who might not be familiar with the basics of that, pyautogui keyboard actions only reach the active window on top (in the case of multiple windows). In order for the pyautogui action to be successful, the window with the CAPTCHA must remain on top for the duration of the pyautogui actions. In the case of uc_gui_handle_cf(), that duration is generally less than 2 seconds per call, even if you have a lot of windows open and being controlled at the same time. The "call" includes: Making the current window the active window on top, finding the iframe, switching into the iframe, making the checkbox the active element, and then clicking the checkbox by pressing the spacebar with pyautogui. To prevent that "call" from being disrupted, we need to use thread-locking to prevent other actions from making another window become the active one (for the entire duration of the "call").

Here are some actions that would make another window the active one:

Thread-locking can be placed around browser launches that occur via SeleniumBase. It can also be placed around SeleniumBase methods that call driver.switch_to.window() indirectly. There isn't much that can be done about human actions while the scripts are running, or if people are calling driver.switch_to.window() directly from their scripts.

With the extra thread-locking added, we should be able to see a noticeable improvement in the success rate of uc_gui_handle_cf() calls when multiple threads are being used to spin up multiple browsers at the same time. As a side-effect, there may be some slowdowns when multiple threads are trying to change the active window at the same time, because only one thread will be able to perform such an action at one time. This special thread-locking will only take place during UC Mode. For regular mode, there won't be any blockers from scripts performing actions.

mdmintz commented 3 days ago

This was resolved in 4.28.2 - https://github.com/seleniumbase/SeleniumBase/releases/tag/v4.28.2

OpsecGuy commented 1 day ago

This was resolved in 4.28.2 - https://github.com/seleniumbase/SeleniumBase/releases/tag/v4.28.2

Hey, in my case the latest update doesn't really click on the captcha checkbox on Windows. Im using SB format. Updates 4.28.0 and 4.28.1 are working properly, however using latest update makes my code stuck on uc_gui_handle_cf method. Even after waiting for 5 and more minutes nothing really happens.

mdmintz commented 1 day ago

I see the issue on Windows. I'll ship a new release soon...

mdmintz commented 1 day ago

Upgrade to 4.28.3 (Fixes https://github.com/seleniumbase/SeleniumBase/issues/2889)