seleniumbase / SeleniumBase

📊 Blazing fast Python framework for web crawling, scraping, testing, and reporting. Supports pytest. Stealth abilities: UC Mode and CDP Mode.
https://seleniumbase.io
MIT License
5.49k stars 993 forks source link

Can not solve turnstile using uc mode, caught turnstile error after scrolling. #2806

Closed SSujitX closed 6 months ago

SSujitX commented 6 months ago

Hello @mdmintz , I tried to scroll csfloat database after login in using steam. After 2 scrolls, the browser gets caught turnstile. I had to use a chrome profile because of signing in through steam.

HTML here

`from seleniumbase import SB import time import sys import os

def main():

root_path = os.path.dirname(os.path.abspath(sys.argv[0]))
os.chdir(root_path)

wait_selector = 'tbody[class="mdc-data-table__content"]'
item_selector_main = 'tbody[class="mdc-data-table__content"] > tr[role="row"]'
copyright_selector = 'div[class="copyright"]'
url = 'https://csfloat.com/db?name=Sport%20Gloves&defIndex=5030&order=2&min=0&max=0.14&maxAge=1'
with SB(uc=True, user_data_dir=r"", maximize=True) as sb:
    sb.driver.uc_open_with_reconnect(url, 3)

    sb.wait_for_element_visible(wait_selector, timeout=60)  # Wait up to 60 seconds for the element to be visible
    sb.highlight(item_selector_main)

    print('Showing elements')
    while True:
        sb.slow_scroll_to(copyright_selector)
        sb.sleep(5)

if name == "main": main() `

mdmintz commented 6 months ago

I would start with the usual CAPTCHA bypass example for Steam and work from there: https://github.com/seleniumbase/SeleniumBase/blob/master/examples/raw_games.py

I would also avoid any kind of window maximization, as that makes Selenium detectable.

You'll have to work with the existing UC Mode examples to find a way, as your example requires logging in to a website with an existing account.

SSujitX commented 6 months ago

The problem is here. I checked all of the seleniumbase documents. I didn't find any solution. Can you please help me?

image

mdmintz commented 6 months ago

I've offer you the same advice as https://github.com/seleniumbase/SeleniumBase/issues/2792#issuecomment-2122725656:

For maximum stealth, combine UC Mode with pyautogui and perform actions while disconnected. You can do driver.disconnect() to disconnect the driver. Then perform the pyautogui actions. Then driver.connect() afterwards so that you can perform Selenium actions again.

You'll be able to scroll the page down with pyautogui by using the down arrow key. A regular Selenium scroll can get you detected.

SSujitX commented 6 months ago

Thanks, @mdmintz , it's working with pyautogui.

I want to learn one more thing which is,

        while True:
            sb.driver.disconnect()
            pyautogui.press('end')
            sb.sleep(3)
            sb.driver.connect()

            elements = sb.find_elements (item_selector_main)
            if elements:
                for element in elements:
                    try:
                        item_name = element.find_element("css selector", "div.prefix").text.replace('★', '').strip()
                    except Exception:
                        item_name = None
                    print(item_name)

using the sb method I used find_elements for a list of items then when I used find_element only got one element from the list. why I need to pass "css selector"? also is there another way to write like in seleniumbase style?