HCaptcha/Cloudflare test cannot be passed

[kekkc]

Hi,

I discovered a similar issue than Cloudflare with sites that are protected by HCaptcha (the open source recaptcha alternative): Example Site: https://codebeautify.org/base64-to-image-converter

Even with "newassets.hcaptcha.com" & "assets.hcaptcha.com" (thosre are used to embed HCaptcha on the codebeautify site) on the whitelist it's not possible to pass the hcaptcha test.

Expected Behavior

If the Captcha is correctly solved, the browser redirects to the actual website

Current Behavior

If the Captcha is correclty solved, the browser simply shows again the Captcha

Would be awesome if someone has a solution or workaround already ;)

Greetz

[sereneblue]

Hi @kekkc,

Thanks for bringing this to my attention. I can't reproduce the captcha with the link you provided but I'll try with some other IPs. I wonder if it's HCaptcha or Cloudflare that's causing the issue (as seen here #477) .

[kekkc]

Hi @sereneblue,

thx, was not able to reproduce in my other test browser initially as well.

What worked temporarily for me: I saved the settings, deleted Chameleon completely (not deactivated), reinstalled & reimported my settings and I passed the HCaptcha and was properly redirected, without any addition to the whitelist.

However, after I closed the site and wanted to reconfirm this, I ended up in the redirection loop again without chance to recover (in my usual & test browser). Also I can't change my IP with my ISP. Will also try to test at another place, very hard to nail down :(

Greetz

[kekkc]

Tested at another house with a dedicated IP (instead of mine which is shared with 60 other people by my ISP), but with the same result.

However, I narrowed it down: the redirection error only occurs if the UA is changed. I set it to "WIN10 Firefox 91". Let me know if you were able to reproduce ;)

(BTW: the whitelist works if codebeautify.org & newassets.hcaptcha.com are both added) (BTW2: some tests were inconsistent for me, although I always cleared the browser cache. In addition I sometimes had to also restart the browser)

[sereneblue]

@kekkc I was able to reproduce by changing my IP and using the profile you mentioned. This seems like a cloudflare issue. Outside of whitelisting the site you're trying to access, I don't think this can be fixed.

The initial user agent that's used when first visiting a cloudflare protected website is your profile UA. Cloudflare has enough data to know that you're spoofing your user agent when you solve the random challenge it sends you. If hCaptcha is whitelisted, it'll use your real profile UA. Since that probably doesn't match your spoofed profile, it puts you in a loop. Changing your user agent after using your real profile also kicks you back to the challenge page.

[kekkc]

Thanks, first time I was able to test this properly:

If I use https://addons.mozilla.org/de/firefox/addon/user-agent-string-switcher and set it to the following UA (you have to select "Apply container on window" & Test UA): Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:90.0) Gecko/20100101 Firefox/90.0 I can access https://codebeautify.org/base64-to-image-converter perfectly.

Cloudflare generally accepts spoofed UA, or has to if FF outputs the spoofed values everywhere and in all javascript. Seems there is still some non-standard behavior in Chameleon, although it's using the same UA values (guess the order of the header values is not relevant?).

[sereneblue]

Thanks, first time I was able to test this properly:

If I use https://addons.mozilla.org/de/firefox/addon/user-agent-string-switcher and set it to the following UA (you have to select "Apply container on window" & Test UA): Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:90.0) Gecko/20100101 Firefox/90.0 I can access https://codebeautify.org/base64-to-image-converter perfectly.

Cloudflare generally accepts spoofed UA, or has to if FF outputs the spoofed values everywhere and in all javascript. Seems there is still some non-standard behavior in Chameleon, although it's using the same UA values (guess the order of the header values is not relevant?).

What OS do you use Firefox with? Chameleon's Win 10/FF 91 user agent is almost identical to the one you mentioned.

Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101 Firefox/91.0

[kekkc]

Yes, I'm using Win10 20H2 v19042.1165 & FF91 Portable for testing.

My original UA (works): Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101 Firefox/91.0

UA Swticher UA (works): Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:90.0) Gecko/20100101 Firefox/90.0

Chameleon UA (doesn't work): Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101 Firefox/91.0

Apart from the UA FF version, all use exactly the same request headers (and apart from some random IDs they all get the same reply headers from Cloudflare). The only difference I found is that Chameleon produces 2 errors, which might be related: uncaught exception: Object SecurityError: Permission denied to access property "host" on cross-origin object 2 inject.js:278

[kekkc]

OT: I just read your comment in inject.js "// try injecting again to bypass cors": I'm using userscripts within Firemonkey (the only userscript manager that injects userscripts programmatically via the new contentscripts & userscripts API and not tabs.exectute). This was implemented 2 years ago ( https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/userScripts/Working_with_userScripts ).

However, FF never got a solution for userscripts to bypass CORS. Master bug for this is the again de-prioritized: https://bugzilla.mozilla.org/show_bug.cgi?id=1587494

Also I never found any workaround to disable iframes completely, but to make them clickable (so that they can be activated in case of ReCaptcha / hCaptcha). Either communication was prevented by CORS, or the captcha iframes registered that I had tempered with the iframe src / srcdoc.

On Topic again: maybe a workaround here might be to not use "window.top.location.host" in Chameleon if no "protect window" or any other JS option is used. If Chameleon is just used for HTML headers & running inside an iframe, there might not be the need to access any properties of the parent window.

[sereneblue]

Yes, I'm using Win10 20H2 v19042.1165 & FF91 Portable for testing.

My original UA (works): Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101 Firefox/91.0

UA Swticher UA (works): Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:90.0) Gecko/20100101 Firefox/90.0

Chameleon UA (doesn't work): Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101 Firefox/91.0

Apart from the UA FF version, all use exactly the same request headers (and apart from some random IDs they all get the same reply headers from Cloudflare). The only difference I found is that Chameleon produces 2 errors, which might be related: uncaught exception: Object SecurityError: Permission denied to access property "host" on cross-origin object 2 inject.js:278

Thanks for testing. Were you able to reliably test using the user agent switcher addon and a different IP? Also, have you tried using a Chrome user agent to see if you can trigger the captcha?

OT: I just read your comment in inject.js "// try injecting again to bypass cors": I'm using userscripts within Firemonkey (the only userscript manager that injects userscripts programmatically via the new contentscripts & userscripts API and not tabs.exectute). This was implemented 2 years ago ( https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/userScripts/Working_with_userScripts ).

However, FF never got a solution for userscripts to bypass CORS. Master bug for this is the again de-prioritized: https://bugzilla.mozilla.org/show_bug.cgi?id=1587494

Also I never found any workaround to disable iframes completely, but to make them clickable (so that they can be activated in case of ReCaptcha / hCaptcha). Either communication was prevented by CORS, or the captcha iframes registered that I had tempered with the iframe src / srcdoc.

On Topic again: maybe a workaround here might be to not use "window.top.location.host" in Chameleon if no "protect window" or any other JS option is used. If Chameleon is just used for HTML headers & running inside an iframe, there might not be the need to access any properties of the parent window.

Chameleon uses the contentScript API to inject JS into the page. The issue could be coming from overwriting some browser APIs. I'll check to see if only changing the headers triggers a Cloudflare loop.

[kekkc]

Were you able to reliably test using the user agent switcher addon and a different IP? Also, have you tried using a Chrome user agent to see if you can trigger the captcha? Sry, took me a while to check it at another house. Tested with UA Switcher:

• Win10 FF90 (works, I'm able to pass the Captcha)
• Win10 Opera 76 (works, I'm able to pass the Captcha)
• Win10 Chrome 99 (doesn't work, not able to pass)

Tested several times, browser restarted, but strangely Chrome's UA doesn't seem to work, while the others do with UA.

[sereneblue]

Were you able to reliably test using the user agent switcher addon and a different IP? Also, have you tried using a Chrome user agent to see if you can trigger the captcha? Sry, took me a while to check it at another house. Tested with UA Switcher:

* Win10 FF90 (works, I'm able to pass the Captcha)

* Win10 Opera 76 (works, I'm able to pass the Captcha)

* Win10 Chrome 99 (doesn't work, not able to pass)

Tested several times, browser restarted, but strangely Chrome's UA doesn't seem to work, while the others do with UA.

Thanks for testing. I haven't gotten a chance to look into this yet, but I'm going try soon using both FF and Chrome with different user agents. Is that Chrome version 99 or 93?

[kekkc]

Cool, it was Chrome 99: "Mozilla/5.0 (Windows NT 10.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.7113.93 Safari/537.36"

[sereneblue]

99.0.7113.93

I don't think that's a valid version. Maybe that's why you ran into an issue with hCaptcha? https://chromereleases.googleblog.com/

[sereneblue]

@kekkc I've done some additional testing with a proxy and a temporary container so this should be somewhat accurate. Cell values with an x were able to load the page after completing a captcha. It seems Chameleon's spoofing is causing some issues. Unfortunately, I don't think there's much that I can do to resolve this.

Profile	Spoofed user agent	Full profile spoof
Real Profile	x	x
Edge 94 + Win 7	x
Firefox 78 + Win 7
Firefox 68 + Win 7
Firefox 92 + Win 7	x
Chrome 94 + Win 7
Firefox 92 + Win 10	x
Edge 94 + OS X 10.14	x
Firefox 78 + OS X 10.14
Firefox 68 + OS X 10.14
Firefox 92 + OS X 10.14	x
Chrome 94 + OS X 10.14
Firefox 78 + Linux
Firefox 68 + Linux
Firefox 92 + Linux	x
Chrome 94 + Linux
iOS 13 + Chrome (Phone)	x
iOS 13 + Chrome (Tablet)
iOS 13 + Safari (Phone)	x
iOS 13 + Safari (Tablet)	x
Android 8 + Edge 94	x
Android 8 + Firefox 92 (Phone)	x
Android 8 + Firefox 92 (Tablet)	x
Android 8 + Chrome 94 (Phone)	x
Android 8 + Chrome 94 (Tablet)	x

[kekkc]

Many thanks for this awesome debugging.

I have to test your list more, maybe I find some similarities to UA Switcher & my setup. Propably there's still some general solution. Still don't get why Firefox 92 + Win 10 is not working for me. Also UA Switcher seems to use the same API & code:

chrome.webRequest.onBeforeSendHeaders.addListener(onBeforeSendHeaders, {
   'urls': ['*://*/*', 'ws://*/*', 'wss://*/*']
}, ['blocking', 'requestHeaders']);
chrome.webNavigation.onCommitted.addListener(onCommitted);

https://github.com/ray-lothian/UserAgent-Switcher/blob/master/extension/firefox/common.js

BTW: had some trouble with FF bugs falsely activating 1st party isolation when deactivating extensions and enabling certificates to have the default compatibility on https://clienttest.ssllabs.com:8443/ssltest/viewMyClient.html . These bugs also impacted me so that I could no longer use google and was caught in a captcha loop. However, all were unrelated to Chameleon, since I tested initially with a portable installation. Also the usage of http/3 was unrelated (this is arbitrarily used in FF).

[sereneblue]

Many thanks for this awesome debugging.

I have to test your list more, maybe I find some similarities to UA Switcher & my setup. Propably there's still some general solution. Still don't get why Firefox 92 + Win 10 is not working for me. Also UA Switcher seems to use the same API & code:

chrome.webRequest.onBeforeSendHeaders.addListener(onBeforeSendHeaders, {
   'urls': ['*://*/*', 'ws://*/*', 'wss://*/*']
}, ['blocking', 'requestHeaders']);
chrome.webNavigation.onCommitted.addListener(onCommitted);

https://github.com/ray-lothian/UserAgent-Switcher/blob/master/extension/firefox/common.js

BTW: had some trouble with FF bugs falsely activating 1st party isolation when deactivating extensions and enabling certificates to have the default compatibility on https://clienttest.ssllabs.com:8443/ssltest/viewMyClient.html . These bugs also impacted me so that I could no longer use google and was caught in a captcha loop. However, all were unrelated to Chameleon, since I tested initially with a portable installation. Also the usage of http/3 was unrelated (this is arbitrarily used in FF).

I don't think there's a solution besides whitelist with your real profile.. Cloudflare's test is probably a mix of feature detection and using specific browser APIs to leak your true browser. If your values don't match the "expected" values, you'll be stuck in a loop.

[slrslr]

Infinitely repeated captcha on a CF protected page happens to me in like 1/3 of the cases when visiting a CF protected site. It is a biggest problem with using Chameleon for me. Whitelisting every newly visited site is not practical.

the workaround to this Cloudfllare issue is:

to click Chameleon and click to "change" (randomize) its profile and let it do one more captcha attempt. If failed, change profile again in Chameleon. It usually works on at least 3rd attempt. @sereneblue Should I build a list of a non working profile combinations for later removing these? By a profile combination I mean what is displayed in Chameleon, for example "Win 8.1 / Firefox 128 ESR": Current Profile Win 8.1 / Firefox 128 ESR Default Screen GMT Default Language

btw. this issue may be renamed to contain Cloudflare in it...

[kekkc]

Got no issues recently with only the newest profiles:

But that shouldn't mean sth., anyhow regarding CAPTCHA tests I also found this good test site: https://nopecha.com/captcha

[sereneblue]

Infinitely repeated captcha on a CF protected page happens to me in like 1/3 of the cases when visiting a CF protected site. It is a biggest problem with using Chameleon for me. Whitelisting every newly visited site is not practical.

the workaround to this Cloudfllare issue is:

to click Chameleon and click to "change" (randomize) its profile and let it do one more captcha attempt. If failed, change profile again in Chameleon. It usually works on at least 3rd attempt. @sereneblue Should I build a list of a non working profile combinations for later removing these? By a profile combination I mean what is displayed in Chameleon, for example "Win 8.1 / Firefox 128 ESR": Current Profile Win 8.1 / Firefox 128 ESR Default Screen GMT Default Language

btw. this issue may be renamed to contain Cloudflare in it...

The best solution for Cloudflare is whitelisting and using Firefox profile. Using a VPN does make it more likely that you'll get caught in the CF loop. I could potentially add a feature that automatically detects a CF protected site and adds it to a special CF whitelist profile.