seleniumbase / SeleniumBase

📊 Python's all-in-one framework for web crawling, scraping, testing, and reporting. Supports pytest. UC Mode provides stealth. Includes many tools.
https://seleniumbase.io
MIT License
4.45k stars 908 forks source link

Website detected - uc mode #2815

Closed jkoestner closed 1 month ago

jkoestner commented 1 month ago

I've recently had my bot detected on a website. It was working for a few months, but has recently seemed to be flagged. I'm not sure if this is just a product of tighter security. I've done a few tests. Any direction would be appreciated. Thanks!

packages

First I checked pixelscan.net to see what may be causing the website to flag. This seems ok to me because it states "no automation framework detected"

with SB(
        uc=True,
        headless2=True,
        proxy=proxy,
    ) as sb:
        url = "https://pixelscan.net/"
        sb.driver.uc_open_with_reconnect(url, 20)
        sb.sleep(10)
        sb.driver.save_screenshot('folioflex/configs/pixel.png') 

pixel

I then saved a screenshot of how the website is flagging the bot. The code below does work on a laptop using chrome, but does not work on a linux server running docker using chromium.

with SB(
        uc=True,
        headless2=True,
        proxy=proxy,
    ) as sb:
        url = "https://www.wsj.com/livecoverage/stock-market-today-dow-jones-earnings-05-30-2024?mod=lctimeline_finance_viewall"
        sb.driver.uc_open_with_reconnect(url, 20)
        sb.sleep(15)
        sb.driver.save_screenshot('folioflex/configs/bot.png')

bot

mdmintz commented 1 month ago

Same answer as https://github.com/seleniumbase/SeleniumBase/discussions/2802#discussioncomment-9555175:

For UC Mode, you have to use regular chrome, not chromium. The binary would be called google-chrome or google-chrome-stable on Linux.