Cloudflare TLS fingerprinting

[Barre]

I have the same experience with other websites managed by Cloudflare. I can't open the > pages as long as the proxy is enabled. For example: https://www.cloudflare.com/ https://community.cloudflare.com https://chat.openai.com The request for "Checking if the site connection is secure" is never completed.

Originally posted by @gitze in https://github.com/Barre/privaxy/issues/62#issuecomment-1380841172

[Barre]

@gitze I opened a new issue as what you experience is pretty different. Cloudflare uses TLS fingerprinting as part of their "bot detection" feature.

Privaxy's TLS fingerprint is not matching your browser user agent TLS fingerprint, which breaks cloudflare. They'd have the same issue with some non-transparent corporate proxies and several security products, which in essence are behaving the same way as Privaxy does.

It would be possible to make Privaxy to match common browsers fingerprints. Still, it would add a lot of complexity, be a maintenance burden and force Privaxy to switch to openssl instead of rustls which is an idea I'm not fond of.

The correct way of fixing this would be cloudflare to whitelist Privaxy. I don't think they'd do that as anyone using the same technical stack as Privaxy would then be able to very easily bypass their bot detection.

They have a form for this here https://forms.gle/dT9muX2aYRqFokkc8 I will still try that and post results here if I get any reply.

For now, as a workaround, you can add websites you cannot access to Privaxy's "exclusions". When a website is "excluded", it's transparently tunnelled.

[maxcountryman]

It would be possible to make Privaxy to match common browsers fingerprints. Still, it would add a lot of complexity, be a maintenance burden and force Privaxy to switch to openssl instead of rustls which is an idea I'm not fond of.

Can you go into a little more detail? It sounds like rustls simply doesn't support this at the moment?

I'm also curious, would it be possibly to forward the client's user agent without requiring much maintenance overhead? I realize for the above reason that may be a moot point, but I wonder if that would be easier than trying to maintain a list of such things.

[Barre]

It would be possible to make Privaxy to match common browsers fingerprints. Still, it would add a lot of complexity, be a maintenance burden and force Privaxy to switch to openssl instead of rustls which is an idea I'm not fond of.

Can you go into a little more detail? It sounds like rustls simply doesn't support this at the moment?

It's more that openssl supports more primitives and in general, would be more flexible for this kind of things. Rustls is designed to have fewer to no foot guns.

I'm also curious, would it be possibly to forward the client's user agent without requiring much maintenance overhead? I realize for the above reason that may be a moot point, but I wonder if that would be easier than trying to maintain a list of such things.

Privaxy does not change the user agent of requests. Privaxy's TLS fingerprint is not matching a known "good" TLS fingerprint + user-agent pair on Cloudflare's side.

[GunGunGun]

The correct way of fixing this would be cloudflare to whitelist Privaxy.

But no way Cloudflare would do that, because Python didn't get that treatment and still get bot blocked, so it's better to parrot popular browser fingerprints like this app TLS-Client: https://github.com/FlorianREGAZ/Python-Tls-Client

[GunGunGun]

It would be possible to make Privaxy to match common browsers fingerprints. Still, it would add a lot of complexity, be a maintenance burden and force Privaxy to switch to openssl instead of rustls which is an idea I'm not fond of.

I actually want to say something about OpenSSL, OpenSSL is even harder to make that happen imo I tried to fake TLS Fingerprint with OpenSSL in Python and failed miserably, it can't change a lot of things about extensions and eliptic curve, so it's impossible to mimic real browser fingerprint anyways, Go actually has the best weapon to do that tho, it's: https://github.com/bogdanfinn/tls-client

Just let you know so you don't waste your time with it. Rather stick with rustls.

[brian6932]

^yep, openssl can't do it, know this firsthand. your best bet is to either do it yourself with something like rustls, or use boringssl, which definitely simplifies what you have to do, since that's chromium's tls handling right there. https://github.com/gssvv/rust-boring-ssl-client

[ajayyy]

https://github.com/lwthiker/curl-impersonate has been very useful for this in my experience

[labac-p]

👀 any update here