search

bytedance / g3

Enterprise-oriented Generic Proxy Solutions
Apache License 2.0
473 stars 35 forks source link

g3proxy: Ability to change ClientHello sent to upstream server #138

Open mspublic opened 11 months ago

mspublic commented 11 months ago

When using g3proxy it is sometime detected by websites running on cloudflare and other sites using TLS fingerprinting such as https://github.com/salesforce/ja3. Having the ability to modify the ClientHello will make the proxy less detectable/less fingerprintable and more usable in enterprise environments.

The ideal “solution” would be to be able to set a ja3 fingerprint and have the proxy send it.

I have not found a way to modify it via OpenSSL but it appears rustls may give access to the ClientHello https://docs.rs/reqwest/latest/reqwest/struct.ClientBuilder.html#method.use_preconfigured_tls

Here is a bit more info and code examples:

https://medium.com/cu-cyber/impersonating-ja3-fingerprints-b9f555880e42

https://github.com/refraction-networking/utls

https://github.com/Danny-Dasilva/CycleTLS

https://github.com/LyleMi/ja3proxy

https://github.com/Kolosok86/http-tls-proxy

zh-jq-b commented 11 months ago

Rustls doesn't support the customization of ClientHello Message at the time of now, see https://github.com/rustls/rustls/pull/1564.

To implement this, we have to parse the client hello message sent by the tls providers and then rewrite them at the underlying stream layer, which may have some limitation for adding unknown ciphersuites and extensions.

For OpenSSL, we will add a new SslConnector in g3-openssl crate in the following days, then we can experiment with client hello rewriting there.

mspublic commented 11 months ago

My mistake. I got reqwest and rustls mixed up.

I look forward to the change! Thanks for the fast reply. G3 is great work!

zh-jq-b commented 11 months ago

Change the client message in the underlying layer is not working, as OpenSSL will check record MAC. I have added extension reordering ability in branch https://github.com/bytedance/g3/tree/openssl-ja3, and the test failed with OpenSSL send BadRecordMAC alert message to target site.

mspublic commented 10 months ago

Sorry for delay - I did not receive a notification.

Thanks for the update! I will review the code. I think this will become a big hinderance to enterprise usage - many sites using fingerprinting/cloudflare etc will block requests that appear to be plain OpenSSL as they think it's a bot. This is one of the major issues with Squid - detected as a bot. Sites such as https://chat.openai.com/ don't work. It makes deployment into most enterprises a problem.

I see you have been working on adding BoringSSL support - this may be the way to go. It seems to be leveraged by other projects to impersonate Chrome - https://dev.to/gssvv/making-tls-client-with-chrome-like-ssl-handshake-rust-boring-ssl-h2-n63

It seems to be a good path to go down as openssl seems a bit strict on the ability to modify.

zh-jq-b commented 10 months ago

I see you have been working on adding BoringSSL support - this may be the way to go. It seems to be leveraged by other projects to impersonate Chrome - https://dev.to/gssvv/making-tls-client-with-chrome-like-ssl-handshake-rust-boring-ssl-h2-n63

The missing seems to be

set_grease_enabled
enable_ocsp_stapling
enable_signed_cert_timestamps

Can you try adding this to the rel/boringssl branch and test it? I won't available for this feature this week.

zh-jq-b commented 10 months ago

For OCSP stapling:

zh-jq-b commented 10 months ago

For GREASE:

zh-jq-b commented 10 months ago

For SCT:

Millione commented 10 months ago

now supported by https://github.com/rustls/rustls/pull/1730

zh-jq-b commented 10 months ago

OCSP Stapling, SCT, GREASE have been added as config options, and you can test them when build with AWS-LC or BoringSSL. And the extension order is random.

The still missing extensions when comparing with chromium are:

zh-jq-b commented 10 months ago

ALPS:

ECH:

mspublic commented 9 months ago

This is great progress! Thank you @zh-jq-b! I finally have some time to do some more testing of this.

mspublic commented 9 months ago

I have been testing with BoringSSL enabled and it has been working great! I find that during regular use I run into less issues using the BoringSSL feature than standard OpenSSL (which is typical of OpenSSL usage in general).

I did notice that it isn't doing TLS extension permutation which is a newer feature to prevent fingerprinting https://chromestatus.com/feature/5124606246518784.

It should be a pretty easy addition. The code is enabled the same way GREASE is. I was going to take a stab at it by reusing the grease code and submitting a PR to as well variant-ssl too. Is that ok? Or would you prefer to?

Thoughts?

https://github.com/google/boringssl/blob/db614a5677d90e48cfb2c0f8197f1b5168fceea5/ssl/ssl_lib.cc#L3015

void SSL_CTX_set_permute_extensions(SSL_CTX *ctx, int enabled) {
  ctx->permute_extensions = !!enabled;
}

void SSL_set_permute_extensions(SSL *ssl, int enabled) {
  if (!ssl->config) {
    return;
  }
  ssl->config->permute_extensions = !!enabled;
}
zh-jq-b commented 9 months ago

I did notice that it isn't doing TLS extension permutation which is a newer feature to prevent fingerprinting https://chromestatus.com/feature/5124606246518784.

It should be a pretty easy addition. The code is enabled the same way GREASE is. I was going to take a stab at it by reusing the grease code and submitting a PR to as well variant-ssl too. Is that ok? Or would you prefer to?

An extra config option should be used and the default value should be set to false.

I haven't look into the details but I remember that the tls ext ordering is already randomized when GREASE is enabled. Do you know what's the difference?

zh-jq-b commented 9 months ago

permute_extensions added in commit 27e34e0795a531bea8617b8fd7ee0af6412ee4d9

mspublic commented 9 months ago

Wow you are fast! Thank you!

Just to reply to your question - based on my understanding from the RFC GREASE adds randomness into its own extension but does not randomize the order of all extensions.

Thank you again!

mspublic commented 9 months ago

I have been testing with enable_grease, enable_sct, and permute_extensions enabled in BoringSSL. Everything is working excellent. I have verified that all are working as expected from a protocol side. Fingerprint is now random.

hellais commented 7 months ago

You might be interested in this issue: https://github.com/rustls/rustls/issues/1932