seleniumbase / SeleniumBase

📊 Python's all-in-one framework for web crawling, scraping, testing, and reporting. Supports pytest. UC Mode provides stealth. Includes many tools.
https://seleniumbase.io
MIT License
4.46k stars 909 forks source link

Can not solve turnstile using uc mode, caught turnstile error after scrolling. #2806

Closed SSujitX closed 1 month ago

SSujitX commented 1 month ago

Hello @mdmintz , I tried to scroll csfloat database after login in using steam. After 2 scrolls, the browser gets caught turnstile. I had to use a chrome profile because of signing in through steam.

HTML here

`from seleniumbase import SB import time import sys import os

def main():

root_path = os.path.dirname(os.path.abspath(sys.argv[0]))
os.chdir(root_path)

wait_selector = 'tbody[class="mdc-data-table__content"]'
item_selector_main = 'tbody[class="mdc-data-table__content"] > tr[role="row"]'
copyright_selector = 'div[class="copyright"]'
url = 'https://csfloat.com/db?name=Sport%20Gloves&defIndex=5030&order=2&min=0&max=0.14&maxAge=1'
with SB(uc=True, user_data_dir=r"", maximize=True) as sb:
    sb.driver.uc_open_with_reconnect(url, 3)

    sb.wait_for_element_visible(wait_selector, timeout=60)  # Wait up to 60 seconds for the element to be visible
    sb.highlight(item_selector_main)

    print('Showing elements')
    while True:
        sb.slow_scroll_to(copyright_selector)
        sb.sleep(5)

if name == "main": main() `

mdmintz commented 1 month ago

I would start with the usual CAPTCHA bypass example for Steam and work from there: https://github.com/seleniumbase/SeleniumBase/blob/master/examples/raw_games.py

I would also avoid any kind of window maximization, as that makes Selenium detectable.

You'll have to work with the existing UC Mode examples to find a way, as your example requires logging in to a website with an existing account.

SSujitX commented 1 month ago

The problem is here. I checked all of the seleniumbase documents. I didn't find any solution. Can you please help me?

image

mdmintz commented 1 month ago

I've offer you the same advice as https://github.com/seleniumbase/SeleniumBase/issues/2792#issuecomment-2122725656:

For maximum stealth, combine UC Mode with pyautogui and perform actions while disconnected. You can do driver.disconnect() to disconnect the driver. Then perform the pyautogui actions. Then driver.connect() afterwards so that you can perform Selenium actions again.

You'll be able to scroll the page down with pyautogui by using the down arrow key. A regular Selenium scroll can get you detected.

SSujitX commented 1 month ago

Thanks, @mdmintz , it's working with pyautogui.

I want to learn one more thing which is,

        while True:
            sb.driver.disconnect()
            pyautogui.press('end')
            sb.sleep(3)
            sb.driver.connect()

            elements = sb.find_elements (item_selector_main)
            if elements:
                for element in elements:
                    try:
                        item_name = element.find_element("css selector", "div.prefix").text.replace('★', '').strip()
                    except Exception:
                        item_name = None
                    print(item_name)

using the sb method I used find_elements for a list of items then when I used find_element only got one element from the list. why I need to pass "css selector"? also is there another way to write like in seleniumbase style?