seleniumbase / SeleniumBase

📊 Python's all-in-one framework for web crawling, scraping, testing, and reporting. Supports pytest. UC Mode provides stealth. Includes many tools.
https://seleniumbase.io
MIT License
5.05k stars 945 forks source link

UC mode Detected by cloud flare, No more undectected #2465

Closed suredarv06 closed 7 months ago

suredarv06 commented 7 months ago

from seleniumbase import SB

with SB(uc=True, test=True,driver_version='106') as sb: sb.driver.uc_open_with_tab("https://nowsecure.nl/#relax") sb.sleep(1.2) if not sb.is_text_visible("OH YEAH, you passed!", "h1"): sb.get_new_driver(undetectable=True) sb.driver.uc_open_with_reconnect( "https://nowsecure.nl/#relax", reconnect_time=3 ) sb.sleep(1.2) if not sb.is_text_visible("OH YEAH, you passed!", "h1"): if sb.is_element_visible('iframe[src="challenge"]'): with sb.frame_switch('iframe[src="challenge"]'): sb.click("span.mark") sb.sleep(2) sb.activate_demo_mode() sb.assert_text("OH YEAH, you passed!", "h1", timeout=3)

sb.open("about:blank")
sb.sleep(0.55)

sb.driver.default_get("https://nowsecure.nl/#relax")
sb.assert_text("OH YEAH, you passed!", "h1", timeout=3)

This code is no more working It is detected by cloudflare

mdmintz commented 7 months ago

Duplicate of https://github.com/seleniumbase/SeleniumBase/discussions/2392#discussioncomment-7956613

Use sb.driver.default_get(URL) when you want to open a URL without the special bot-detection-evasion code. Useful if going back to the same site where you already bypassed detection before in that same browser session. (Then there's no delay for the driver disconnecting from Chrome, pausing for a bit, and reconnecting to Chrome.)

If you need to bypass bot-detection, then don't use sb.driver.default_get(URL).

suredarv06 commented 7 months ago

I can't understand the above definition ,I am not using sb.driver.default_get(URL) method. can you give the code which bypass the cloudflare with selenium base?

mdmintz commented 7 months ago

Examples can be found here: SeleniumBase/help_docs/uc_mode.md

suredarv06 commented 7 months ago

I have already used that Examples which is not bypassing

from seleniumbase import SB with SB(uc=True, test=True) as sb: sb.driver.uc_open_with_tab("https://nowsecure.nl/#relax") sb.sleep(1.2) if not sb.is_text_visible("OH YEAH, you passed!", "h1"): sb.get_new_driver(undetectable=True) sb.driver.uc_open_with_reconnect( "https://nowsecure.nl/#relax", reconnect_time=3 ) sb.sleep(1.2) if not sb.is_text_visible("OH YEAH, you passed!", "h1"): if sb.is_element_visible('iframe[src="challenge"]'): with sb.frame_switch('iframe[src="challenge"]'): sb.click("span.mark") sb.sleep(2) sb.activate_demo_mode() sb.assert_text("OH YEAH, you passed!", "h1", timeout=3)

I have tried all examples , Can you run the above code which is from your example . It is not working

mdmintz commented 7 months ago

Learn how to format your code with proper indentation using Markdown: https://www.markdownguide.org/basic-syntax/

Also, Chrome 121 currently has a memory leak, so you may need to quit Chrome and reopen it to remove any slowness that triggers detection. It's fixed in Chrome 122.

suredarv06 commented 7 months ago

`from seleniumbase import SB

with SB(uc=True, test=True,driver_version=122) as sb: sb.driver.uc_open_with_tab("https://nowsecure.nl/#relax") sb.sleep(1.2) if not sb.is_text_visible("OH YEAH, you passed!", "h1"): sb.get_new_driver(undetectable=True) sb.driver.uc_open_with_reconnect( "https://nowsecure.nl/#relax", reconnect_time=3 ) sb.sleep(1.2) if not sb.is_text_visible("OH YEAH, you passed!", "h1"): if sb.is_element_visible('iframe[src="challenge"]'): with sb.frame_switch('iframe[src="challenge"]'): sb.click("span.mark") sb.sleep(2) sb.activate_demo_mode() sb.assert_text("OH YEAH, you passed!", "h1", timeout=3)

sb.open("about:blank")
sb.sleep(0.55)

sb.driver.default_get("https://nowsecure.nl/#relax")
sb.assert_text("OH YEAH, you passed!", "h1", timeout=3)`

I am running this code which not by passing updated driver version

mdmintz commented 7 months ago

driver_version only changes the driver version. You need to get Chrome 122 to avoid the memory leak in Chrome 121. It's currently in beta, but will be officially released soon. Get Chrome beta versions here: https://www.google.com/chrome/beta/

suredarv06 commented 7 months ago

I have dowloaded beta version chrome ,But still I am unable to bypass cloudflare. I think the issue should to be reopened

mdmintz commented 7 months ago

Are you running locally, or in the cloud? What OS? What SeleniumBase version? (Note that if you run in the cloud, you would likely need to use a proxy server because Cloudflare blacklists IP Ranges, such as those used by GitHub Actions)

suredarv06 commented 7 months ago

I am running locally,There no proxy involved . I am using residential IP which is trusted only , I am using Mac 14.2.1 (23C71)

mdmintz commented 7 months ago

Which version of SeleniumBase? Type seleniumbase --version on the command-line.

suredarv06 commented 7 months ago

surendarv@SCIAD38LP206 zenrow % seleniumbase --version


/ __/ / / __ () __ / _ __ __ \/ \/ / _ \/ \/ / / / / `_ \ / /) / \/ / \ _/ / / / _/ / / / / // / / / / / // /) / (/ / / / /__/_//\// ///_,// // ////_,_//___/

/Volumes/Space/Projects/zenrow/.venv/lib/python3.9/site-packages/ seleniumbase 4.23.2

suredarv06 commented 7 months ago

My chrome beta version is 122.0.6261.6

mdmintz commented 7 months ago

Does this get you through?

from seleniumbase import SB

with SB(uc=True) as sb:
    sb.driver.uc_open_with_reconnect("https://nowsecure.nl/#relax", "breakpoint")

Type c and press ENTER in the console to continue from the breakpoint.

Swap "breakpoint" with 3 (or more) seconds when you're done experimenting.

suredarv06 commented 7 months ago


with SB(uc=True) as sb:
    sb.driver.uc_open_with_reconnect("https://nowsecure.nl/#relax", "breakpoint")```

**These things not working**
suredarv06 commented 7 months ago

On rare occasions is getting bypassed. But it Blocked 99%

mdmintz commented 7 months ago

It's working for me.

Screenshot 2024-02-02 at 2 22 56 PM

But I'll see if I can adjust timings for the next release in case anyone else runs into it.

suredarv06 commented 7 months ago

Can you try another time

mdmintz commented 7 months ago

What happens when you go to https://nowsecure.nl/#relax with a regular browser as a human?

suredarv06 commented 7 months ago

It getting bypassed ,when normally opening in browser

mdmintz commented 7 months ago

Here's a different site with Cloudflare protection. This script bypasses it:

from seleniumbase import SB

with SB(uc=True) as sb:
    sb.driver.uc_open_with_reconnect("https://top.gg/", "breakpoint")

From the looks of CI and https://nowsecure.nl/#relax, it seems Cloudflare has set new limits on how many times an IP Address can hit any specific Cloudflare-protected site before throwing a CAPTCHA, even if they don't detect Selenium. I switched sites, and the script was able to bypass detection right away. That means that using a rotating proxy may be necessary now if hitting a site with heavy automation activity.

suredarv06 commented 7 months ago

Yes it is working for the above site . I will try to use rotating proxy for the (https://nowsecure.nl/#relax).

I thanks for you immediate replies. Thanks for you great efforts .

mdmintz commented 7 months ago

Details on the memory leak of Chrome 121: https://issues.chromium.org/issues/41494493 (Fixed in Chrome 122)