Getting detected by Impreva

D4Vinci commented 2 days ago

Website detecting Camoufox:

What website or WAF is flagging Camoufox? Provide as much detail and additional context as possible.

According to Wappalyzer, it's Impreva.

Screenshots:

If applicable, add screenshots to help explain your problem.

Most of the time it just gives a blank white page but sometimes if you are patient enough, you will see this page:

Playing around and disabling the JS give this too (if helpful):

To Reproduce:

Send a testing site that reproduces the issue, and a snippet of your script. Provide your Camoufox() or AsyncCamoufox() initialization.

Just browse this page https://www.smythstoys.com/uk/en-gb/toys/lego-and-bricks/c/SM060107 The example doesn't matter because I always get blocked locally and with different configs.

Version:

Run python -m camoufox version in your terminal and paste the output here.

Pip package:    v0.3.10
Camoufox:   v132.0-beta.15 (Up to date!)

Also, congratulations on finally getting some of the recognition Camoufox deserves. I made sure everyone knows I'm using your work in Scrapling even in my Reddit post and then happily watched the stars go up haha.

D4Vinci commented 2 days ago

It only worked when I used the allow_webgl parameter locally, I will test it on the server ASAP.

daijro commented 2 days ago

Hello, thanks for integrating my work into Scrapling! I appreciate the recognition!

Thank you for the valuable test site.

From my debugging, it seems that other similar open source anti-detect libraries like Patchright that also use the PW scoping strategy aren't able to pass this site as well.

I haven't been able to produce a success case by enabling WebGL or reverting all the FF preferences set in Camoufox.

Here are a couple notes from my research this morning:

The website doesn't flag when launching the Camoufox binary directly and navigating to the site, which leads me to believe that this is an automation/Juggler detection.
Running a fresh build of Camoufox without Playwright's patches applied still seems to fail. This might have something to do with the Firefox remote automation API causing a leak.

I'll keep you updated — I'm between classes right now, but I'll continue investigating this ASAP.

D4Vinci commented 2 days ago

Nice notes! I have a friend who was able to make camoufox work by enabling WebGL as well on his machine. He was getting blocked before on his personal browser (brave) but the website worked with him on Opera which was weird because both browsers are based on Chromium as you know.

daijro commented 1 day ago

Hello, here's an update. I found 3 things Imperva has started detecting:

Your friend is right — WebGL does seem to be required now (which is weird, a couple hours ago I was able to load the page without it). In the next update, WebGL will be enabled in Camoufox by default, although it might hurt fingerprintability. Developing WebGL spoofing is now very high priority.
It seems I was right about FF's native debugging API now being detected. On a fresh build, FF is flagged when a client connects to the debug server. In Camoufox, the python library uses the FF rdp API to upload addons like uBlock Origin. I will work on a new way to load addons.
The third thing that Imperva detects is Camoufox's build config listed here. I've narrowed it down to the patched locales that Camoufox uses to rebrand from Firefox. One of these locales is likely leaking to the page.

I'm also starting to believe that this repo is being monitored. I may have to start closing the source to certain patches in the future.

After a bit of research, I've successfully fixed all the leaks in a local build:

D4Vinci commented 1 day ago

Impressive work @daijro ! Yeah definitely the repo is monitored it’s their job and with time while the project becoming more famous more protection companies will see it

D4Vinci commented 1 day ago

Hey @daijro, it just came to my mind: Why won't you make installing UBO optional? It has nothing to do with fingerprinting anyway, and now it hurts Camoufox. I see it isn't worth all the time you will waste researching a new way to install add-ons. If the user chooses to install an add-on, give him/her a leak warning.

daijro commented 1 day ago

Hey @daijro, it just came to my mind: Why won't you make installing UBO optional?

UBO helps a lot with memory load on heavier websites, so I see it as pretty valuable to keep, especially if Camoufox is going to be run on datacenters.

It shouldn't be too hard to implement addon loading. It's just a quick call to the AddonActor API that I can implement elsewhere (for example browser-init.js) to install the addon when the browser starts:

  async installTemporaryAddon(addonPath, openDevTools) {
    let addonFile;
    let addon;
    try {
      addonFile = new FileUtils.File(addonPath);
      addon = await AddonManager.installTemporaryAddon(addonFile);
    } catch (error) {
      throw new Error(`Could not install add-on at '${addonPath}': ${error}`);
    }

    Services.obs.notifyObservers(null, "devtools-installed-addon", addon.id);

https://searchfox.org/mozilla-central/source/devtools/server/actors/addon/addons.js#28

daijro commented 1 day ago

Addon loading has just been added 👍

Beta.17 is now building. I'll work on an update for the Python library in a bit

daijro commented 1 day ago

Camoufox beta 17 & Python library release 0.4.1 is now out.

This includes all the fixes listed here, and WebGL fingerprint rotation.

daijro commented 19 hours ago

There seems to be an issue on some devices where certain WebGL fingerprints crash the browser when injected. WebGL rotation has been temporarily rolled back. I'm going to reopen this issue until it's fixed.

D4Vinci commented 18 hours ago

Hi mate, impressive work! Unfortunately, some issues came with the update, I'm testing on the same website with WebGL enabled and sometimes disabled. Sometimes the website works but sometimes I get this with WebGL disabled

and sometimes I get a full crash with WebGL enabled

retrying the code in the second error again right away gets no crash

daijro commented 18 hours ago

Hi mate, impressive work! Unfortunately, some issues came with the update, I'm testing on the same website with WebGL enabled and sometimes disabled. Sometimes the website works but sometimes I get this with WebGL disabled

I noticed this too when I was testing it on a different device. On my PC, I didn't notice any issues with it when I was testing the release, but on my laptop it would frequently crash with certain fingerprints from GPU vendors that clashed with my own. My theory is that when an GL enum is overriden thats reserved for an internal binding it causes a panic.

I've rolled back WebGL fingerprint rotation in 0.4.2 to work on a fix. It's probably best to disable WebGL for sites that don't require it as well, as it can be used to tell if you're lying about your OS.

D4Vinci commented 17 hours ago

I understand, I mean these errors happens on the latest version after rolling back

daijro commented 11 hours ago

Got it. WebGL is disabled by default again until beta 18 is out. I need to test on more devices.

D4Vinci commented 10 hours ago

Alright mate, good luck!

daijro / camoufox