Suddenly unable to bypass CloudFlare challenge (Ubuntu Server)

Jobine23 commented 3 weeks ago

Hello, overnight my instances of seleniumbase became unable to bypass the CloudFlare challenge ( which uses CloudFlare turnstile ).

I was using an older version of SB so I updated to latest ( 4.27.4 ), and it is still not passing the challenge.

cloudflare_chal

I am using your demo code for clicking on the CloudFlare turnstile captcha:

from seleniumbase import SB

def open_the_turnstile_page(sb):
    url = "https://wildbet.gg/"
    sb.driver.uc_open_with_reconnect(url, reconnect_time=5)

def click_turnstile_and_verify(sb):
    sb.switch_to_frame("iframe")
    sb.driver.uc_click("span")
    sb.assert_element("img#captcha-success", timeout=3)

with SB(uc=True, test=True) as sb:
    open_the_turnstile_page(sb)
    try:
        click_turnstile_and_verify(sb)
    except Exception:
        open_the_turnstile_page(sb)
        click_turnstile_and_verify(sb)
    sb.set_messenger_theme(location="top_left")
    sb.post_message("SeleniumBase wasn't detected", duration=3)

if I instead use: sb.driver.uc_open_with_reconnect(url, reconnect_time=9999)

and click manually, it works. This means they are detecting something ?

I also tried adding reconnect_time=5 on uc_click and it did not help.

I'm a big fan of your project and I've been using it for some time :)

mdmintz commented 3 weeks ago

Just tested on seleniumbase 4.27.5, and the following script is working for me:

from seleniumbase import SB

with SB(uc=True, test=True) as sb:
    url = "https://wildbet.gg/"
    sb.driver.uc_open_with_reconnect(url, 10)

If you're still getting blocked, it might be that you exceeded a rate limit for your IP Address.

Jobine23 commented 3 weeks ago

Thank you for your reply. I am not sure to understand. Why would it be related to any kind of ratelimiting if I can edit the code to disconnect permanently then click myself so that it goes through ? The only difference between error and success is that I click manually.

mdmintz commented 3 weeks ago

You can try that if it helps. When I ran the UC Mode script for that page, the script never had to click anything to bypass the CAPTCHA.

JimKarvo commented 3 weeks ago

We have the same problem at headless systems -ubuntu server-. 2 different servers, the ip is rotated - whitelisted

vmolostvov commented 3 weeks ago

@mdmintz same problem here on linux vds (ubuntu without gpu), seleniumbase became unable to bypass the CloudFlare challenge. Using latest sb version. On local macos and windows keep working without any problem.

vmolostvov commented 3 weeks ago

We have the same problem at headless systems -ubuntu server-. 2 different servers, the ip is rotated - whitelisted

did you find any solution sir?

JimKarvo commented 3 weeks ago

We have the same problem at headless systems -ubuntu server-. 2 different servers, the ip is rotated - whitelisted

did you find any solution sir?

Not yet, waiting if @mdmintz has any suggestions

mdmintz commented 3 weeks ago

OK, sounds like something has changed because there are a few different people showing up here. And this also sounds like this is specific to Linux because everything is working normally on macOS and Windows for me.

I'm still going to need more details to assist:

What version of Linux are you using?
What version of SeleniumBase are you using?
Does it still work on an earlier version of SeleniumBase, and if so, what's the highest version that it still works on?
Which Chrome/Version is installed on your machine? (Note that google-chrome and chromium are different!)
Does it work if you use xvfb=True (virtual display) with headed=True (override default headless mode on Linux)?

This will help me figure out what's going on. Everything is working normally for me on macOS and Windows.

EnmeiRyuuDev commented 3 weeks ago

Hello @mdmintz , I also had the issue yesterday at all my VMs, Not only Linux (Ubuntu 22.04, Debian 11), but also in a Windows Server 2022, It works perfectly only in my personal laptop having Windows 11 Home, Basically: At VMs:

it never passes without displaying the CF checkbox challenge
uc_click seems to be detected, and it never recovers
on the same SB browser, when clicking manually instead of letting SB clicks, it works pretty quick after the manual click
tried with or without proxy, the same results
versions: google-chrome version 125.0.6422.142 under Windows Server, google-chrome version 124.0.6367.118 under Ubuntu 22.04, google-chrome version 122.0.6261.111 under Debian 11. I also tried going in SeleniumBase version as old as v4.23.0, v4.20.8, but it didn't help (same blockage behavior).

At my personal laptop (Windows 11):

most of the time, it passes without having to click anything
when it has to click, it passes as well
tried with or without proxy (same proxy that's used for the VMs) -> always working fine
versions: google-chrome version 125.0.6422.142

It looks like Cloudflare is able to tell if I'm using a VM somehow, but yet, it works when clicking manually.. I can say it can be quickly reproduced on any Google Compute Engine VM.

I hope my description is helpful, And thanks for the great project ^^

EnmeiRyuuDev commented 3 weeks ago

@mdmintz To complete answering your questions:

Used SeleniumBase version, the latest: v4.27.5
On Linux instances, I tried: xvfb=True, with headed=True, but it didn't help.

mdmintz commented 3 weeks ago

Looks like we've narrowed it down to an environment issue. Try this script, and let me know which environments it works on and which it fails on:

from seleniumbase import SB

with SB(uc=True, test=True) as sb:
    url = "https://visa.vfsglobal.com/fra/en/hrv/login"
    sb.driver.uc_open_with_reconnect(url, 6)
    sb.switch_to_frame("iframe")
    sb.driver.uc_click("span")
    breakpoint()

Worked for me on my Mac, but sounds like it won't work on some Linux server environments.

SaberTawfiq commented 3 weeks ago

https://github.com/seleniumbase/SeleniumBase/assets/152691314/a369e2a1-5bc4-418c-9ccf-4d88f7106b54

JimKarvo commented 3 weeks ago

@mdmintz For Ubuntu 20.04.6 LTS - server edtition, the script you provided not even clicks at span for some reason

seleniumbase.common.exceptions.NoSuchElementException: Message: 
 Element {span} was not present after 7 seconds!

mdmintz commented 3 weeks ago

OK, getting somewhere. Seeing differences on Mac vs Windows. Two videos below.

On a Mac, script worked as is. (Adjust reconnect_time based on your Internet connection speed.):

https://github.com/seleniumbase/SeleniumBase/assets/6788579/cb9eebdc-7b1a-4079-86fb-01abca6f4bc7

On Windows, had to add incognito=True in order for the script to work:

https://github.com/seleniumbase/SeleniumBase/assets/6788579/487fdd0e-aa27-49a4-883e-14b4842bcb1d

Did adding incognito=True to your SB() call help any of you?

If so, add it to all your scripts. If not, let me know details, such as "worked locally on Windows, but not on Ubuntu Server", for example. In order for me to fix this properly, I need to know exactly what conditions & environments are bad vs good.

OpsecGuy commented 3 weeks ago

"environment": {
    "implementation_name": "cpython",
    "implementation_version": "3.12.1",
    "os_name": "nt",
    "platform_machine": "AMD64",
    "platform_release": "10",
    "platform_system": "Windows",
    "platform_version": "10.0.19045",
    "python_full_version": "3.12.1",
    "platform_python_implementation": "CPython",
    "python_version": "3.12",
    "sys_platform": "win32"
  }

Name: seleniumbase
Version: 4.27.5

Driver object:

def create(self):
        return Driver(
            browser='chrome',
            incognito=True,
            dark_mode=True,
            headless2=self.summary.headless,
            proxy=self.proxy_addr,
            uc=True,
            uc_subprocess=True,
            # extension_dir=self.extension_dir if self.summary.captcha == True else None
        )

Logic:

target = "https://netguard.io/sduyafgewsdfn.php" # Website with cf captcha
self.session = self.create()
self.session.uc_open(target)
self.session.reconnect(2.55 if not self.summary.emulation_force else 6.0)

# In here getting Cloudflare captcha loop
try:
    self.session.switch_to_frame("iframe")
    self.session.uc_click("span")
    self.session.switch_to.default_content()
    self.session.sleep(3.5)
except Exception: pass

Worth to add I have tested on my local IP address, tested on proxyscrape.com dedicated proxies and rotating ones and also other provider with IPv6 proxies. On all proxy versions same issue.

JimKarvo commented 3 weeks ago

I have this code (tryied also with incognito):

from seleniumbase import SB
import time 

with SB(uc=True, test=True) as sb:
    url = "https://visa.vfsglobal.com/fra/en/hrv/login"
    sb.driver.uc_open_with_reconnect(url, 6)
    sb.switch_to_frame("iframe")
    sb.driver.uc_click("span")
    time.sleep(3)
    sb.save_screenshot("bypass.png")

at windows, I get the CF passed. at Linux, i get the CF not clicked

windows 11: bypass

linux - ubuntu server 20.04: bypass

mdmintz commented 3 weeks ago

@JimKarvo Thank you... That is helpful. I see the same thing: Working on both macOS and Windows, but not for Linux (Ubuntu server 20.04). I will look into improvements specifically on the Ubuntu side (since things appear to be working on macOS and Windows, at least with incognito=True). Possibly Cloudflare found a way to see if automation is running on Ubuntu.

I'll be working on improvements to SeleniumBase Ubuntu configuration. In the meantime, people can keep passing on any information that I might find useful. (If anyone is still having macOS or Windows issues, I'll need an example that reproduces your issue, because so far things appear to be normal on the local desktop front, at least with incognito=True.)

JimKarvo commented 3 weeks ago

@mdmintz I don't know, but changing any user agent, maybe will help the situation?

If I can help with access to any of my VMs (ubuntu server), I can send ssh details

mdmintz commented 3 weeks ago

@JimKarvo Can you tell me the user-agent that appears when you ran on Ubuntu? And the agent you saw when running locally? Maybe there's a clue there.

JimKarvo commented 3 weeks ago

@mdmintz

From Linux I get:

GET / HTTP/1.1
Host: test.requestcatcher.com
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7
Accept-Encoding: gzip, deflate, br, zstd
Accept-Language: en-US,en;q=0.9
Connection: keep-alive
Sec-Ch-Ua: "Chromium";v="125", "Not.A/Brand";v="24"
Sec-Ch-Ua-Mobile: ?0
Sec-Ch-Ua-Platform: "Linux"
Sec-Fetch-Dest: document
Sec-Fetch-Mode: navigate
Sec-Fetch-Site: cross-site
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.6422.141 Safari/537.36

from Windows I get:

GET /test HTTP/1.1
Host: test.requestcatcher.com
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7
Accept-Encoding: gzip, deflate, br, zstd
Accept-Language: el-GR,el;q=0.9
Connection: keep-alive
Sec-Ch-Ua: "Google Chrome";v="125", "Chromium";v="125", "Not.A/Brand";v="24"
Sec-Ch-Ua-Mobile: ?0
Sec-Ch-Ua-Platform: "Windows"
Sec-Fetch-Dest: document
Sec-Fetch-Mode: navigate
Sec-Fetch-Site: cross-site
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36

mdmintz commented 3 weeks ago

If there's a human-controlled web browser with that same user-agent on that system (Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.6422.141 Safari/537.36) I'm wondering if they would experience the same issue. Anyone having any issues if they use a Linux computer with a GUI and a regular web browser? (Possibly Cloudflare is only blocking based on that agent - not because they detected Selenium.)

OpsecGuy commented 3 weeks ago

@JimKarvo Thank you... That is helpful. I see the same thing: Working on both macOS and Windows, but not for Linux (Ubuntu server 20.04). I will look into improvements specifically on the Ubuntu side (since things appear to be working on macOS and Windows, at least with incognito=True). Possibly Cloudflare found a way to see if automation is running on Ubuntu.

I'll be working on improvements to SeleniumBase Ubuntu configuration. In the meantime, people can keep passing on any information that I might find useful. (If anyone is still having macOS or Windows issues, I'll need an example that reproduces your issue, because so far things appear to be normal on the local desktop front, at least with incognito=True.)

Not really. As mentioned by me above. I still struggle with bypassing on Windows. I don't know if there is anything else I could add to it. If you need more info tell me exactly what should I be looking for.

mdmintz commented 3 weeks ago

@OpsecGuy You seem to be the only one having Windows issues at this time. You also introduced a lot of variables into the equation in your example. Eg. 1. using Driver() instead of SB(). 2. Using Dark Mode. 3. Using any headless mode. 4. Changing proxy settings. 5. Adding an extension. You should stick with the specific examples I tried in this thread (or another UC Mode example from the SeleniumBase/examples folder) for debugging purposes.

Jobine23 commented 3 weeks ago

I am using a VPS using Windows server 2022, without GPU. incognito=True does not appear to help

Incognito:

with SB(uc=True, test=True, incognito=True) as sb:
    url = "https://visa.vfsglobal.com/fra/en/hrv/login"
    sb.driver.uc_open_with_reconnect(url, 8)
    sb.switch_to_frame("iframe")
    sb.driver.uc_click("span")
    breakpoint()

sb_fail_incognito

with SB(uc=True, test=True) as sb:
    url = "https://visa.vfsglobal.com/fra/en/hrv/login"
    sb.driver.uc_open_with_reconnect(url, 8)
    sb.switch_to_frame("iframe")
    sb.driver.uc_click("span")
    breakpoint()

sb_fail

It is not related to any ratelimiting because I am able to click complete Turnstile captcha inside the same environment/computer, but in a real chrome ( latest 125.0.6422.142 ) instance:

real_chrome

I found something rather interesting comparing both browser info from https://bot.incolumitas.com/ ( both tests executed on the same machine ).

Regular chrome: https://pastebin.com/raw/fj3uaNBM

SB with UC=True: https://pastebin.com/raw/AiV3y58D

SeleniumBase appears to be missing some navigatorProperties/navigatorPrototype. I have no idea if it could impact but I guess it should be as similar as possible to a fresh chrome instance.

EnmeiRyuuDev commented 3 weeks ago

I think the focus should be on the switch_to_frame/ uc_click action. Because with manual click (after uc_open_with_reconnect) it works smoothly (whether we're under Linux or Windows, Desktop or Server). Still puzzling why uc_click still work in a Desktop environment.

Jobine23 commented 3 weeks ago

Yes, sb.driver.uc_open_with_reconnect(url, 99999) and clicking manually appears to work for me so I guess it's not related to what I sent previously.

mdmintz commented 3 weeks ago

OK, I'm trying to figure out why uc_click(selector) behaves differently in different environments (Linux vs macOS/Windows). I'll hopefully have an update ready soon / early this week.

If anyone figures out the reason for the uc_click(selector) differences before I do, let me know. The good news is that sb.driver.uc_open_with_reconnect(url, reconnect_time) is working correctly, and that's the main component of UC Mode.

Hopefully there's a pyautogui alternative to clicking in a stealthy way. Currently, that is an option. Eg. If you use driver.uc_open_with_disconnect(url), then you can use pyautogui to click in a stealthy way, followed by driver.connect() so that you can perform more Selenium actions after that.

JimKarvo commented 3 weeks ago

Tried to implent pyautogui, with no luck, so in case that anybody have working example gor headless systems, that pyautogui working, please write below!

Thank you!

mdmintz commented 3 weeks ago

@JimKarvo pyautogui won't work for headless browsers, or multithreaded, but if you wanted to see an example script that uses pyautogui, here's one:

import pyautogui
from seleniumbase import SB

with SB(uc=True, test=True, incognito=True) as sb:
    sb.driver.uc_open_with_reconnect("https://seleniumbase.io/apps/turnstile")
    sb.send_keys("html", "\t")
    sb.driver.disconnect()
    pyautogui.press(" ")
    sb.sleep(1)
    sb.driver.connect()
    sb.assert_element("img#captcha-success", timeout=3)
    sb.set_messenger_theme(location="top_left")
    sb.post_message("SeleniumBase wasn't detected", duration=3)

Obviously, due to those pyautogui limitations, I want a better workaround for Ubuntu Servers. At least macOS and Windows are still good for now. (And yes, I'm working on Ubuntu solutions.)

JimKarvo commented 3 weeks ago

Guys, I found that changing the user agent, to a valid one, bypasses the CF. EG: I tryied this:

ua = "Mozilla/5.0 (X11; Linux i686) AppleWebKit/534.2 (KHTML, like Gecko) Chrome/94.0.4096.99 Safari/534.2 EdgA/94.01135.87"

with SB(uc=True, test=True, agent=ua) as sb:

@mdmintz are there any changes that we can modify the Sec-Ch-Ua-Platform: "Linux" in order to tell the CF that we are using windows (for future blocks?)

vmolostvov commented 3 weeks ago

Guys, I found that changing the user agent, to a valid one, bypasses the CF. EG: I tryied this:
ua = "Mozilla/5.0 (X11; Linux i686) AppleWebKit/534.2 (KHTML, like Gecko) Chrome/94.0.4096.99 Safari/534.2 EdgA/94.01135.87"

with SB(uc=True, test=True, agent=ua) as sb:
@mdmintz are there any changes that we can modify the Sec-Ch-Ua-Platform: "Linux" in order to tell the CF that we are using windows (for future blocks?)

Changing the UA did not yield any results on ubuntu vds. Tried more than 5 different values. Are you using Linux with GUI ?

JimKarvo commented 3 weeks ago

You are right, after a lot of tests I found that passing a valid user agent just triggers the anti-bot system of cloudflare less than before.

Also I noticed that the checkbox is clicked and trying to resolve the I am human with no luck.

@mdmintz sounds like the main problem is not just the click of span element

mdmintz commented 3 weeks ago

I don't think it's the UA. People said that using sb.driver.uc_open_with_reconnect(url, reconnect_time) followed by a manual click works on Ubuntu (with a GUI). That means it's a problem with uc_click(selector) somehow.

Or maybe people should recheck that... if anyone uses sb.driver.uc_open_with_reconnect(url, reconnect_time) on Ubuntu and manually clicks before the reconnect_time expires, does it work consistently?

mdmintz commented 3 weeks ago

I found something interesting. If I do the following on a Mac (with a Linux UA), and click manually before the reconnect_time expires, then the CAPTCHA is bypassed successfully. But if I wait for the uc_click(selector) to happen, then it fails:

from seleniumbase import SB

agent=("Mozilla/5.0 (X11; Linux i686) AppleWebKit/534.2 (KHTML, like Gecko) Chrome/94.0.4096.99 Safari/534.2 EdgA/94.01135.87")

with SB(uc=True, test=True, agent=agent) as sb:
    url = "https://visa.vfsglobal.com/fra/en/hrv/login"
    sb.driver.uc_open_with_reconnect(url, 15)
    sb.switch_to_frame("iframe")
    sb.driver.uc_click("span")
    breakpoint()

So as long as Selenium is not detected during the clicking part, it doesn't seem like they cared that there was a Linux UA for a macOS system. (But note that the website will still block you if they detect a bad UA string, such as one that contains HeadlessChrome in it.)

vmolostvov commented 3 weeks ago

I found something interesting. If I do the following on a Mac (with a Linux UA), and click manually before the reconnect_time expires, then the CAPTCHA is bypassed successfully. But if I wait for the uc_click(selector) to happen, then it fails:
from seleniumbase import SB

agent=("Mozilla/5.0 (X11; Linux i686) AppleWebKit/534.2 (KHTML, like Gecko) Chrome/94.0.4096.99 Safari/534.2 EdgA/94.01135.87")

with SB(uc=True, test=True, agent=agent) as sb:
    url = "https://visa.vfsglobal.com/fra/en/hrv/login"
    sb.driver.uc_open_with_reconnect(url, 15)
    sb.switch_to_frame("iframe")
    sb.driver.uc_click("span")
    breakpoint()
So as long as Selenium is not detected during the clicking part, it doesn't seem like they cared that there was a Linux UA for a macOS system. (But note that the website will still block you if they detect a bad UA string, such as one that contains HeadlessChrome in it.)

This confirms the fact that they detect uc_click on Linux.

mdmintz commented 3 weeks ago

OK, here's the full scope:

UC Mode: --> macOS: ✅ (All good) --> Windows: ✅ (All good) --> Linux (Ubuntu / server): ⁉️ (conditional) ------> uc_open_with_reconnect(): ✅ ----------> Manual actions (while disconnected): ✅ ----------> pyautogui actions (while disconnected): ✅ ----------> uc_click(selector): ❌

I'll be working on either fixing uc_click(selector) on Linux, or an alternative way to successfully clicking a CAPTCHA checkbox on Linux.

poplu2525 commented 3 weeks ago

I am using inside the docker file(ubuntu), if i click manually as soon as captcha pops up, it gets resolve, otherwise its not. i used PyautoGui also , Not working

poplu2525 commented 3 weeks ago

got it working guys, implemented the crazy solution, Launched 2 threads, 1 thread will handle seleniumbase and another will handle pyautogui and we are good to go

JimKarvo commented 3 weeks ago

got it working guys, implemented the crazy solution, Launched 2 threads, 1 thread will handle seleniumbase and another will handle pyautogui and we are good to go

This will not work at headless systems although

mdmintz commented 3 weeks ago

So for macOS and Windows, the User Agent is automatically set to a "good" one, and the CAPTCHA gets bypassed successfully, automatically:

from seleniumbase import SB

with SB(uc=True, test=True) as sb:
    url = "https://visa.vfsglobal.com/fra/en/hrv/login"
    sb.driver.uc_open_with_reconnect(url, 12)

If you use Linux or set your User Agent to a "not that good" one, then you will be forced to click on the CAPTCHA to bypass it (and due to Cloudflare's recent updates, you'll need to use pyautogui to make it work):

import pyautogui
from seleniumbase import SB

agent=("Mozilla/5.0 (X11; Linux i686) AppleWebKit/534.2 (KHTML, like Gecko) Chrome/94.0.4096.99 Safari/534.2 EdgA/94.01135.87")

with SB(uc=True, test=True, agent=agent) as sb:
    url = "https://visa.vfsglobal.com/fra/en/hrv/login"
    sb.driver.uc_open_with_reconnect(url, 12)
    sb.switch_to_frame("iframe")
    sb.execute_script('document.querySelector("input").focus()')
    sb.disconnect()
    pyautogui.press(" ")
    sb.driver.reconnect(4)

Rumor has it that if you set the correct User Agent on Linux, then the CAPTCHA gets bypassed automatically, like in the current macOS and Windows versions.

I can simplify the pyautogui stuff to a single line in a near future version of seleniumbase. Hopefully, there's a way to avoid pyautogui entirely when needing to click the CAPTCHA checkbox.

Here's how you can help:

If you can figure out the "magic" User Agent to use on Linux, let me know.
If you can figure out a way to successfully click the CAPTCHA without pyautogui now, let me know.

JimKarvo commented 3 weeks ago

@mdmintz I don't think that changing only the useragent will solve the problem.

At request we can see that the user agent is a good on (Windows etc) but the chrome is running under linux Sec-Ch-Ua-Platform: "Linux" is it possible to change parameters like that?

@mdmintz

From Linux I get:

GET / HTTP/1.1
Host: test.requestcatcher.com
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7
Accept-Encoding: gzip, deflate, br, zstd
Accept-Language: en-US,en;q=0.9
Connection: keep-alive
Sec-Ch-Ua: "Chromium";v="125", "Not.A/Brand";v="24"
Sec-Ch-Ua-Mobile: ?0
Sec-Ch-Ua-Platform: "Linux"
Sec-Fetch-Dest: document
Sec-Fetch-Mode: navigate
Sec-Fetch-Site: cross-site
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.6422.141 Safari/537.36

from Windows I get:

GET /test HTTP/1.1
Host: test.requestcatcher.com
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7
Accept-Encoding: gzip, deflate, br, zstd
Accept-Language: el-GR,el;q=0.9
Connection: keep-alive
Sec-Ch-Ua: "Google Chrome";v="125", "Chromium";v="125", "Not.A/Brand";v="24"
Sec-Ch-Ua-Mobile: ?0
Sec-Ch-Ua-Platform: "Windows"
Sec-Fetch-Dest: document
Sec-Fetch-Mode: navigate
Sec-Fetch-Site: cross-site
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36

mdmintz commented 3 weeks ago

@JimKarvo Try this on Linux:

from seleniumbase import SB

with SB(uc=True, test=True, disable_features="UserAgentClientHint") as sb:
    url = "https://visa.vfsglobal.com/fra/en/hrv/login"
    sb.driver.uc_open_with_reconnect(url, 15)
    breakpoint()

The "UserAgentClientHint" should remove the Sec-Ch-Ua-Platform. If that works, I'll make it permanent for UC Mode on Linux. Let me know!

sakarimov commented 3 weeks ago

@JimKarvo Try this on Linux:
from seleniumbase import SB

with SB(uc=True, test=True, disable_features="UserAgentClientHint") as sb:
    url = "https://visa.vfsglobal.com/fra/en/hrv/login"
    sb.driver.uc_open_with_reconnect(url, 15)
    breakpoint()
The "UserAgentClientHint" should remove the Sec-Ch-Ua-Platform. If that works, I'll make it permanent for UC Mode on Linux. Let me know!

tried this, and still get the error

https://github.com/seleniumbase/SeleniumBase/assets/5676876/eb208958-90ad-46c0-abfd-3a0f6089b322

mdmintz commented 3 weeks ago

@sakarimov You may also need to change your User agent to one that doesn't have Linux in it. If you change that too, is there any difference?

jens4626 commented 3 weeks ago

@Jobine23 did you get it working on VPS running Windows Server 2022? @mdmintz I ran this as you have suggest on Windows in incognito, but it gets detected and doesn't get through the verification.

with SB(uc=True, test=True, incognito=True) as sb:

    url = "https://www.virtualmanager.com/da/login?ref_url="
    sb.driver.uc_open_with_reconnect(url, 20)
    sb.switch_to_frame("iframe")
    sb.driver.uc_click("span")

https://github.com/seleniumbase/SeleniumBase/assets/45258332/908d7013-fa8a-4670-9986-83cef1b6374c

mdmintz commented 3 weeks ago

@jens4626 What's the default User Agent that appears when running on your Windows Server? You can use print(sb.get_user_agent()) to get the string.

jens4626 commented 3 weeks ago

@jens4626 What's the default User Agent that appears when running on your Windows Server? You can use print(sb.get_user_agent()) to get the string.

@mdmintz Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36

sakarimov commented 3 weeks ago

@sakarimov You may also need to change your User agent to one that doesn't have Linux in it. If you change that too, is there any difference?

@mdmintz it works for me

from seleniumbase import SB

agent = ("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36")

with SB(uc=True, test=True, disable_features="UserAgentClientHint",
        agent=agent) as sb:
    url = "https://visa.vfsglobal.com/fra/en/hrv/login"
    sb.driver.uc_open_with_reconnect(url, 20)
    sb.switch_to_frame("iframe")
    sb.driver.uc_click("span")
    breakpoint()

i removed all comments to make it clearer

https://github.com/seleniumbase/SeleniumBase/assets/5676876/c4bd1b29-0101-4a66-ba8d-40a1b9d0f8f9

JimKarvo commented 3 weeks ago

@mdmintz I tryied this one, after failing to resolve the captcha

look like the disable_features="UserAgentClientHint" not working at all

from seleniumbase import SB

ua = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36"
with SB(uc=True, test=True, disable_features="UserAgentClientHint", agent=ua) as sb:
    print("getting req catcher") 
    url = "https://jimkarvo.requestcatcher.com/test"
    sb.driver.uc_open_with_reconnect(url, 1)
    breakpoint()

the data i received:

GET /test HTTP/1.1
Host: jimkarvo.requestcatcher.com
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7
Accept-Encoding: gzip, deflate, br, zstd
Accept-Language: en-US,en;q=0.9
Connection: keep-alive
Sec-Ch-Ua: "Chromium";v="125", "Not.A/Brand";v="24"
Sec-Ch-Ua-Mobile: ?0
Sec-Ch-Ua-Platform: "Linux"
Sec-Fetch-Dest: document
Sec-Fetch-Mode: navigate
Sec-Fetch-Site: none
Sec-Fetch-User: ?1
Upgrade-Insecure-Requests: 1
**User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0** Safari/537.36

sakarimov commented 3 weeks ago

from seleniumbase import SB

ua = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36"
with SB(uc=True, test=True, disable_features="UserAgentClientHint", agent=ua) as sb:
    print("getting req catcher") 
    url = "https://jimkarvo.requestcatcher.com/test"
    sb.driver.uc_open_with_reconnect(url, 1)
    breakpoint()

@JimKarvo your reconnect duration seems too small, make it bigger like 7 / 8, in my case i use 20 and it works just fine

seleniumbase / SeleniumBase

Suddenly unable to bypass CloudFlare challenge (Ubuntu Server) #2842