Some websites using Cloudflare CAPTCHA require the challenge again after restarting a script (even with custom profile)

[baptx]

Some websites using Cloudflare CAPTCHA require the CAPTCHA challenge again after restarting a script, even with a custom profile. The issue happened both with old version 3.5.0 and new version 3.5.2 of undetected-chromedriver. A website like nowsecure.nl will not ask the CAPTCHA again when restarting the script with a custom profile where the CAPTCHA was completed, however a website like doctolib.fr will always ask it (which is not always the case without WebDriver). Any idea why? You can find a test script below.

import undetected_chromedriver as uc
import time

driver = uc.Chrome(user_data_dir="/tmp/undetected-chromedriver")

# need to to open in new tab to complete CAPTCHA (https://github.com/ultrafunkamsterdam/undetected-chromedriver/issues/1388#issuecomment-1651886887)
driver.execute_script('window.open("http://nowsecure.nl/#relax");')
#driver.execute_script('window.open("https://www.doctolib.fr");')
time.sleep(5) # wait until page has loaded
#driver.switch_to.window(window_name=driver.window_handles[0]) # switch to first tab (should not be necessary)
driver.close() # close first tab
driver.switch_to.window(window_name=driver.window_handles[0])

time.sleep(30)

[baptx]

I noticed with a normal web browser without WebDriver, the CAPTCHA will be asked again on a website like doctolib.fr if we refresh the web page using Ctrl+Shift+R instead of Ctrl+R or F5. The difference with Ctrl+Shift+R is that it sends the "Pragma: no-cache" and "Cache-Control: no-cache" HTTP request headers. Is there a way to avoid that (without manually editing request headers using an addon) when restarting a WebDriver script, like it is working with nowsecure.nl?

Update: I tried removing request headers like Pragma and Cache-Control with the addon Header Editor on Firefox and Chromium but they are still sent (empty value on Chromium). The interesting thing is that when pressing Ctrl+Shift+R on Chromium without WebDriver, the CAPTCHA is not asked again unlike Firefox. And I can see the 2 headers mentioned previously are sent.

[baptx]

Workaround: we can automate the CAPTCHA validation with the following code (you can change the sleep time if necessary but the best would be to avoid the sleep by knowing when the CAPTCHA finished loading, maybe with MutationObserver).

time.sleep(5)
driver.find_element(By.ID, "challenge-stage").click()

The advantage is that we don't need a custom profile now. But it would be good to have a solution for the CAPTCHA issue that is asked again with a custom profile.

Update: in fact you may not need the sleep with the following code.

el = WebDriverWait(driver, timeout=10).until(lambda d: d.find_element(By.ID,"challenge-stage"))
el.click()

But in some cases, only the first workaround with the sleep will work, which would mean that waiting to find the ID challenge-stage does not mean the CAPTCHA is loaded (waiting for the element to be clickable did not fix the issue: https://stackoverflow.com/questions/26943847/check-whether-element-is-clickable-in-selenium/26943922#26943922).

[NCLnclNCL]

Use chrome 102 can bypass easy

[Krylanc3lo]

@NCLnclNCL I tried 102 but it didn't work. you are using version_number = 102 and download and downgrade chrome browser to 102?

[AntidetectBrowser]

@NCLnclNCL I tried 102 but it didn't work. you are using version_number = 102 and download and downgrade chrome browser to 102?

use this https://github.com/ultrafunkamsterdam/undetected-chromedriver/issues/1456#issuecomment-1672650070

[baptx]

@NCLnclNCL I prefer to avoid old versions of web browsers but I will try it if no other workaround is working in the future. It would be good to know why it works on version 102, so we can try to have the fix on the latest version of Chromium.

[baptx]

@AntidetectBrowser Do you have the source code of the chromedriver.exe file from your repository available somewhere? This would be good so we can see the patch you used and avoid viruses by compiling ourselves. Also the exe file is for Windows only, not Linux.

[teamothyz]

do you have the solution for this? I'm still looking for