net4people / bbs

Forum for discussing Internet censorship circumvention
3.21k stars 75 forks source link

Chrome 107.0.5304.88 TLS fingerprint is (partly) blocked in Iran, what are the consequences for TLS based proxies? #153

Open arandomgstring opened 1 year ago

arandomgstring commented 1 year ago

I am not quite sure if this information can be useful to anyone, but I think it can give some insights on how censorship currently works in Iran; or perhaps to some extent in China. I would be very pleased to hear your opinions on this matter. For TL;DR please just see conclusion at the end.

I've set up a WordPress website on NGINX with VLESS + TLS in its back end. I used this wulabing script, however I changed a few configurations to make WordPress accessible. I am using a CDN but in "DNS only" configuration (gray cloud). The website was accessible to my client until they told me while proxy (with V2RAYN client) works just fine; the website itself is not accessible anymore.

I find the situation funny, hilarious even, that they have blocked normal access to website but not proxy itself. It was worrisome as well, because I thought that the traffic of my proxy should be exactly same as visiting the website. If there is a difference, they might exploit it to block proxies in future after all. So I checked quite a few things to came to conclusion mentioned in title:

  1. The website was inaccessible with "Mokhaberat" ISP, but it worked just fine with "Irancell" ISP according to my client. I asked a friend of mine who used "Mokhaberat" ISP and they told me that website was accessible. So either something was seriously wrong with my client, or "Mokhaberat" ISP at that particular region was at fault.

  2. My client checked 3 different devices: 2 PCs with Win 10, 11 and one android 8 device and confirmed that website is inaccessible but proxy worked fine in all of them.

  3. I told them to change uTLS setup on V2rayN and V2rayNG, to see if there is something wrong with TLS fingerprints. It was not the case. Moreover, the website was inaccessible in both of Chrome and Firefox.

  4. Next day I was informed that Firefox can open the website but Chrome cannot. Unlike previous day the website was blocked in Irancell too (but proxy worked fine). Now that was very interesting, either my client was lying or there was something going on. I took access of my client system remotely, and installed WireShark on it. Yes, they weren't lying.

200176205-80794836-5cc7-489e-820d-9c98be5e164f - Copy

This image above shows TLS handshaking of Chrome 107.0.5304.88 (Official Build) (64-bit). As you can see, after TCP acknowledgement a Client Hello was sent, however I couldn't receive Server Hello packet. I checked V2rayN packets, they had similar TLS fingerprint but its Client Hello used TLSv3 and worked fine. Then I checked Firefox 106.0.5 (64bit) packets:

200176826-b3377b46-32dd-4448-8dfa-e99f24a85e29 - Copy

As you can see it worked fine too. Look at TLS version. So I installed Chrome Version 108.0.5359.29 (Official Build) beta (64-bit) just for the sake of test, and very interestingly it worked as well, in its Client Hello packet I could clearly see that TLSv3 was being used and I got the same result of picture above.

Conclusion: Note that these are just my assumptions from above information, take it with grain of salt.

  1. Either Iran's ISPs have no idea what they are doing and this comes from pure stupidity and lack of knowledge in TLS protocol (because they are essentially blocking legitimate sites by doing so) or

  2. This is some kind of anti Google policies. We have seen Google Translate, Google Play and other services get blocked in Iran; perhaps they want to discourage people from using Chrome. Moreover, Chrome itself is not downloadable or update-able without VPN according to my client which furthers proves this point. However, if they wanted to mess with Chrome why they have targeted TLS version rather than say, TLS fingerprint? I mean looking at JA3 is not much slower than TLS version, is it? Not to mention Beta version of Chrome works fine, because it uses TLSv3.

  3. Maybe (definitely) I am overthinking this, but perhaps they do this to force apps to use some specifics ciphers. Ciphers that are easier to detect when they are used as proxies. I don't think that we are still there, but I suppose something like this is possible in theory at least.

wkrp commented 1 year ago

Thanks for your post. What I understand from it is that at least one TLS fingerprint used by Google Chrome appears to be blocked on (at least) the Mokhaberat and Irancell ISPs. Using Firefox, or a beta version of Chrome (which presumably has a different TLS fingerprint than non-beta Chrome).

I am having trouble understanding a few points.

Is the V2RAYN proxy located inside Iran or outside? When you say the proxy works, do you mean the TLS connection to the proxy, or the proxy's TLS connection to the website? I am not sure if you are drawing a contrast between Chrome's connection to the website and Chrome's incoming connection to the proxy; or between Chrome's connection to the website and the proxy's outgoing connection to the website. When you say "I thought that the traffic of my proxy should be exactly same as visiting the website", the answer could be different, depending on whether you are talking about the incoming or outgoing traffic of the proxy.

When you say "TLSv3" I think you mean "TLSv1.3". Where are you reading the version number from the Client Hello? There are three places that have a version number, and for backward compatibility reasons they may contain different values:

Prior versions of TLS used the record layer version number (TLSPlaintext.legacy_record_version and TLSCiphertext.legacy_record_version) for various purposes. As of TLS 1.3, this field is deprecated. … Version negotiation is performed using only the handshake versions (ClientHello.legacy_version and ServerHello.legacy_version, as well as the ClientHello, HelloRetryRequest, and ServerHello "supported_versions" extensions).

I doubt that the difference really has to do with TLS 1.0 versus TLS 1.3. Just a guess on my part, but it is possible that WireShark is labeling the Client Hello packet "TLSv1", and not a more specific version, only because there is no Server Hello and therefore no negotiated version to assign to the flow. You will have to look more closely into the Client Hello to see the range of supported versions in the "supported_versions" extension.

The website was accessible to my client until they told me while proxy (with V2RAYN client) works just fine; the website itself is not accessible anymore.

  1. I told them to change uTLS setup on V2rayN and V2rayNG, to see if there is something wrong with TLS fingerprints. It was not the case. Moreover, the website was inaccessible in both of Chrome and Firefox.

I am confused by these two sentences. I understand, from the first sentence, that a V2RAYN proxy works but a browser does not work. But in the second sentence, you check for "something wrong" in the V2RAYN fingerprint, which makes it sound like V2RAYN was not working.

It is entirely possible that you have discovered a particular blocked TLS fingerprint. The packet captures could be consistent with that. But without more details about the proxy setup it's hard to be sure.

arandomgstring commented 1 year ago

Thank you for showing interest in this topic.

Is the V2RAYN proxy located inside Iran or outside?

A VPS server located outside of Iran acts as a proxy server for an Iranian user who uses V2rayN app to connect to the server.

you say the proxy works, do you mean the TLS connection to the proxy, or the proxy's TLS connection to the website? I am not sure if you are drawing a contrast between Chrome's connection to the website and Chrome's incoming connection to the proxy; or between Chrome's connection to the website and the proxy's outgoing connection to the website.

Oh, let me clarify this. If my client turns off V2rayN app (meaning there is no proxy at all), Chrome cannot initiates TLS connection to the website, and they see "this site cannot be reached error". The reason for this is that my client doesn't receive Server Hello from the website. With a VPN or a proxy however, the website is reachable. Note that since my client is able to establish TCP connection with the website (as shown in first image) it means that neither of IP address or domain of website is directly censored, rather my client simply cannot initiate TLSv1 connection. So outgoing connection from website is blocked, but the website receives incoming connection with no problem.

On the other hand, if they do turn on V2rayN (meaning VPS acts as a proxy now); they are able to initiate TLS connection with VPS server and use it to browse internet freely. The reason for this, is that V2rayN sends TLSv3 client hello, not TLSv1 and user receives Server Hello and TLS connection is established.

When you say "I thought that the traffic of my proxy should be exactly same as visiting the website", the answer could be different, depending on whether you are talking about the incoming or outgoing traffic of the proxy.

Can you please so kind to clarify this a little bit? While I do realize that the "flow" of information (I mean the number of packets received by the user as a function of time) between proxy server and website can be very different (I haven't checked this yet); on the packet level itself there shouldn't be any difference, is there? I mean everything is encrypted through TLS connection; so what's the difference?

When you say "TLSv3" I think you mean "TLSv1.3". Where are you reading the version number from the Client Hello? There are three places that have a version number, and for backward compatibility reasons they may contain different values:

Indeed, as I said I am using WireShark to capture packets, and it just abbreviates TLSv1.3 to TLSv3 and TLSv1.2 to TLSv2 and ... .

I am confused by these two sentences. I understand, from the first sentence, that a V2RAYN proxy works but a browser does not work. But in the second sentence, you check for "something wrong" in the V2RAYN fingerprint, which makes it sound like V2RAYN was not working.

Yes, V2RayN proxy works and with it user can see websites without censorship. However without a proxy or VPN the website is inaccessible in Chrome; because Chrome uses TLSv1.1 to establish TLS connection, unlike proxy. I asked my client to change uTLS option from firefox to chrome, to see that if V2RayN fails to connect. It didn't fail; and still worked even with Chrome fingerprint.

It is entirely possible that you have discovered a particular blocked TLS fingerprint. The packet captures could be consistent with that. But without more details about the proxy setup it's hard to be sure.

The proxy server uses VLESS + WS + TLS configuration. TLS 1.3 and 1.2 are allowed, and we try to use http2 but that is not necessary, http1.1 works just fine and we are not using domain fronting. Is there anything else that I should add?

free-the-internet commented 1 year ago

@arandomgstring Could you draw a table and summarize your findings? e.g:

                      Chrome             Firefox           TLSv1.1             TLSv1.2               TLSv1.3
v2ray                  x                   x                  ?                  x                      x
Website                -                   +                  -                  x                      x

You can also add other columns for Chrome/Firefox versions.

For me, it looks like they don't know what they are doing.

arandomgstring commented 1 year ago

Sure thing. ✓ means it is working, x means it is not working: Note that V2ray doesn't use TLSv1.1 at all, because it was configured this way. Moreover Chrome Beta uses TLSv1.2 by default and the website is working fine.

                      Chrome             Firefox           TLSv1.1             TLSv1.2               TLSv1.3
v2ray                  ✓                   ✓                  ?                  ✓                     ✓
Website                ?                   ✓                   x                  ✓                     ✓

I forced all connection to use TLSv1.3 and now website is working fine with Chrome (107.0.5304.88). Although strangely enough; it seems as if Firefox and Chrome sometime randomly pick TLSv1.1. Refreshing or "CTRL+ F5" doesn't solve this issue, however if we close the browsers, and re-open them, we can access the website just fine.

I checked Client Hello Packets and it seems to me that they are perfectly similar to that of the website (with TLSv1.3). How exactly v2ray traffic is different from website? What I can see from Wireshark is that for some reason, we have lots of Client Hellos; which might be a problem. Even Mux enabled doesn't solve this problem.

mehdifirefox commented 1 year ago

Excuse me in the middle of the discussion

Yesterday Iran's Internet was much more restricted than previous days

V2Rayng had a lot of shortcomings I have log if someone wants

Clash worked well.

arandomgstring commented 1 year ago

Regarding TLSv1.1, I was able to replicate findings above with this web site (https://www.appimagehub.com/) which Linux users use to install application on their OS. this website can be opened from Firefox and Chrome Beta, but not current version of Chrome because of handshake.

@mehdifirefox I can confirm this; although it seems that this blocking was on IP level not on application layer (V2ray); and was lifted today. Can you share your findings? it might be related.

mehdifirefox commented 1 year ago

when clash works but v2ray doesn't... with the same server Is it possible to ip

Give me the Telegram ID and send the log

On Wed, 9 Nov 2022, 13:46 arandomgstring, @.***> wrote:

Regarding TLSv1.1, I was able to replicate findings above with this web site (https://www.appimagehub.com/) which Linux users use to install application on their OS. this website can be opened from Firefox and Chrome Beta, but not current version of Chrome because of handshake.

@mehdifirefox https://github.com/mehdifirefox I can confirm this; although it seems that this blocking was on IP level not on application layer (V2ray); and was lifted today. Can you share your findings? it might be related.

— Reply to this email directly, view it on GitHub https://github.com/net4people/bbs/issues/153#issuecomment-1308524347, or unsubscribe https://github.com/notifications/unsubscribe-auth/AF26MY7DBZHJFWIDXZJSPRTWHN2XPANCNFSM6AAAAAARYQ5ODE . You are receiving this because you were mentioned.Message ID: @.***>

arandomgstring commented 1 year ago

@mehdifirefox

If possible please send it to this email address: arandomstring@airmail.nz

But without knowing v2ray and clash configuration, personally I cannot deduce anything. Perhaps you are using Trojan protocol in clash, and vmess in v2ray? From what I have heard, some ISPs (Irancell, Hamrah Aval, Shatel, ...) have blocked OVH IP range which most of VPS providers use in some regions. But the fact that Clash works and V2ray doesn't, means that the blocking you are experiencing is indeed on application layer.

mehdifirefox commented 1 year ago

Sent. Several people have reported this problem since last night. I sent everyone to clash

On Wed, 9 Nov 2022, 20:04 arandomgstring, @.***> wrote:

@mehdifirefox https://github.com/mehdifirefox

If possible please send it to this email address: @.***

But without knowing v2ray and clash configuration, personally I cannot deduce anything. Perhaps you are using Trojan protocol in clash, and vmess in v2ray? From what I have heard, some ISPs (Irancell, Hamrah Aval, Shatel, ...) have blocked OVH IP range which most of VPS providers use in some regions. But the fact that Clash works and V2ray doesn't, means that the blocking you are experiencing is indeed on application layer.

— Reply to this email directly, view it on GitHub https://github.com/net4people/bbs/issues/153#issuecomment-1309027109, or unsubscribe https://github.com/notifications/unsubscribe-auth/AF26MY6GRLMAIKOZZMQXIKDWHPHA3ANCNFSM6AAAAAARYQ5ODE . You are receiving this because you were mentioned.Message ID: @.***>

wkrp commented 1 year ago

@arandomgstring you are confusing things by focusing only on the TLS version. What you have described looks like a straightforward case of TLS fingerprinting. Chrome 107.0.5304.88 has a particular TLS fingerprint that is blocked; the V2rayN client app has a different TLS fingerprint, so it is not blocked. The beta version 108.0.5359.29 of Chrome changes the TLS fingerprint, so it again is not blocked. The TLS version is part of the TLS fingerprint, but only a minor part.

It is remarkable that the TLS fingerprint of a popular program like Chrome is blocked; but TLS fingerprinting itself is nothing very new or remarkable. In other threads (https://github.com/net4people/bbs/issues/139#issuecomment-1280243079, https://github.com/net4people/bbs/issues/131#issuecomment-1280284051) we have talked about another TLS fingerprint (750e3f0f585283bd, which is produced by the Go programming language) is blocked in Iran.

If you can, it would help to get a packet capture (pcap) of the Client Hello in a blocked connection, and upload it to https://tlsfingerprint.io/pcap. That way, we can at least document what exact fingerprint is being blocked.

I mean everything is encrypted through TLS connection; so what's the difference?

But there are many ways of implementing TLS, and those different implementation choices form a "fingerprint" that can distinguish TLS produced by different implementations. A TLS fingerprint includes the TLS version, but also the order of ciphersuites and extensions, and many other factors. Chrome TLS looks different from Firefox TLS, which is different from OpenSSL, which is different from Safari, etc. And as you've seen for yourself, the TLS fingerprint can change in different versions of the same implementation.

Some links with more information about TLS fingerprinting:

I am using WireShark to capture packets, and it just abbreviates TLSv1.3 to TLSv3 and TLSv1.2 to TLSv2 and ... .

No, you are mistaken about this. There is no such thing as "TLSv3". Take another look at the screenshots: they say TLSv1.3, not TLSv3.

I will rephrase what I said earlier, because I think it was not clear. The TLS version is a property of the entire TLS connection, not a property of a single packet or a single record. The TLS version is negotiated during the TLS handshake: it only becomes determined when the server sends a Server Hello and chooses one of the client's offered versions. WireShark is using the label TLSv1 to indicate that the exact version number is not known, because there was no Server Hello to complete the version negotiation. You should understand it to mean something like TLSv1.?.

Open the Chrome Client Hello in the WireShark dissector. You will see a supported_versions extension, where the client declares support for multiple versions of TLS. Only after the server chooses one of the versions can WireShark assign a more specific version label to the connection.

Extension: supported_versions (len=9)
    Type: supported_versions (43)
    Length: 9
    Supported Versions length: 8
    Supported Version: TLS 1.3 (0x0304)
    Supported Version: TLS 1.2 (0x0303)
    Supported Version: TLS 1.1 (0x0302)
    Supported Version: TLS 1.0 (0x0301)

Although strangely enough; it seems as if Firefox and Chrome sometime randomly pick TLSv1.1.

This is because the version is chosen by the server, not the client. If the client supports versions up to TLS 1.3, but the server only supports up to TLS 1.1, then the connection will use TLS 1.1. It is not random, it just means that some TLS servers do not support TLS 1.3.

free-the-internet commented 1 year ago

Sure thing. ✓ means it is working, x means it is not working: Note that V2ray doesn't use TLSv1.1 at all, because it was configured this way. Moreover Chrome Beta uses TLSv1.2 by default and the website is working fine.

                      Chrome             Firefox           TLSv1.1             TLSv1.2               TLSv1.3
v2ray                  ✓                   ✓                  ?                  ✓                     ✓
Website                ?                   ✓                   x                  ✓                     ✓

I forced all connection to use TLSv1.3 and now website is working fine with Chrome (107.0.5304.88). Although strangely enough; it seems as if Firefox and Chrome sometime randomly pick TLSv1.1. Refreshing or "CTRL+ F5" doesn't solve this issue, however if we close the browsers, and re-open them, we can access the website just fine.

I checked Client Hello Packets and it seems to me that they are perfectly similar to that of the website (with TLSv1.3). How exactly v2ray traffic is different from website? What I can see from Wireshark is that for some reason, we have lots of Client Hellos; which might be a problem. Even Mux enabled doesn't solve this problem.

When do you see many client hellos exactly? Might be this issue https://github.com/2dust/v2rayN/issues/2753 if you are using v2rayN and has something is wrong with your v2ray client when its ports are used by e.g. a browser trying to connect to a website?

I agree with @wkrp, this is more like TLS fingerprint blocking that TLS version. In case they would target TLS by version, TLSv1.3 is more interesting than TLSv1.2 or TLSv1.1 because of SNI field encryption extension.

klzgrad commented 1 year ago

If Chrome default ClientHello is unconditionally blocked, what is the officially approved browser in Iran and what ClientHello does it use? There must be a "mainstream" browser that is needed to support mainstream productivity. I often suspect this mainstream browser would be some repackaged old version of Chrome that would not be up to date with the latest TLS fingerprint, or it could even use modified TLS parameters from Chrome upstream.

alirezaac commented 1 year ago

If Chrome default ClientHello is unconditionally blocked, what is the officially approved browser in Iran and what ClientHello does it use? There must be a "mainstream" browser that is needed to support mainstream productivity. I often suspect this mainstream browser would be some repackaged old version of Chrome that would not be up to date with the latest TLS fingerprint, or it could even use modified TLS parameters from Chrome upstream.

well i saw someone said on mokhaberat and things chrome is blocked, and in experience of naive proxy in iran, well i had a group of people testing it, most of them use CF proxy and think its blocked, but CF things are seem to be blocked and profiled in iran(like that universal SSl, just worked for me turning that off IDK why), second thing is they test NativeProxy on nekoray or matsuri stuff which the default dns settings are set to work in china, and not so many DOH works good in iran, so i just test some DNSCRYPT and naive is working, some others are connecting using 1.1.1.1 DOH. so lets sort problems with Naive 1 by 1. another mobile provider is blocking TLS heavily that's another thing im investigating 10 days till now, and the thing is its random i cant produce same problem with same clients and same network one is working one is not.

mehdifirefox commented 1 year ago

If Chrome default ClientHello is unconditionally blocked, what is the officially approved browser in Iran and what ClientHello does it use? There must be a "mainstream" browser that is needed to support mainstream productivity. I often suspect this mainstream browser would be some repackaged old version of Chrome that would not be up to date with the latest TLS fingerprint, or it could even use modified TLS parameters from Chrome upstream.

well i saw someone said on mokhaberat and things chrome is blocked, and in experience of naive proxy in iran, well i had a group of people testing it, most of them use CF proxy and think its blocked, but CF things are seem to be blocked and profiled in iran(like that universal SSl, just worked for me turning that off IDK why), second thing is they test NativeProxy on nekoray or matsuri stuff which the default dns settings are set to work in china, and not so many DOH works good in iran, so i just test some DNSCRYPT and naive is working, some others are connecting using 1.1.1.1 DOH. so lets sort problems with Naive 1 by 1. another mobile provider is blocking TLS heavily that's another thing im investigating 10 days till now, and the thing is its random i cant produce same problem with same clients and same network one is working one is not.

Is there a tutorial to replace the right settings for Iran?

alirezaac commented 1 year ago

If Chrome default ClientHello is unconditionally blocked, what is the officially approved browser in Iran and what ClientHello does it use? There must be a "mainstream" browser that is needed to support mainstream productivity. I often suspect this mainstream browser would be some repackaged old version of Chrome that would not be up to date with the latest TLS fingerprint, or it could even use modified TLS parameters from Chrome upstream.

well i saw someone said on mokhaberat and things chrome is blocked, and in experience of naive proxy in iran, well i had a group of people testing it, most of them use CF proxy and think its blocked, but CF things are seem to be blocked and profiled in iran(like that universal SSl, just worked for me turning that off IDK why), second thing is they test NativeProxy on nekoray or matsuri stuff which the default dns settings are set to work in china, and not so many DOH works good in iran, so i just test some DNSCRYPT and naive is working, some others are connecting using 1.1.1.1 DOH. so lets sort problems with Naive 1 by 1. another mobile provider is blocking TLS heavily that's another thing im investigating 10 days till now, and the thing is its random i cant produce same problem with same clients and same network one is working one is not.

Is there a tutorial to replace the right settings for Iran?

Well each provider is blowing up the very foundation of normal connection in different strategies and they don't know the result they just keep testing (even they change the behavior hourly) so even when i want to choose a dns i have to change it everyday. its a volatile behavior which makes it hard to know what's gonna last for a week.

mehdifirefox commented 1 year ago

So it is not a permanent setting. For example, I don't have a problem with v2ray to test the url...but it has a problem with Nekoray. Some servers are not working

On Sat, 12 Nov 2022, 21:18 alirezaac, @.***> wrote:

If Chrome default ClientHello is unconditionally blocked, what is the officially approved browser in Iran and what ClientHello does it use? There must be a "mainstream" browser that is needed to support mainstream productivity. I often suspect this mainstream browser would be some repackaged old version of Chrome that would not be up to date with the latest TLS fingerprint, or it could even use modified TLS parameters from Chrome upstream.

well i saw someone said on mokhaberat and things chrome is blocked, and in experience of naive proxy in iran, well i had a group of people testing it, most of them use CF proxy and think its blocked, but CF things are seem to be blocked and profiled in iran(like that universal SSl, just worked for me turning that off IDK why), second thing is they test NativeProxy on nekoray or matsuri stuff which the default dns settings are set to work in china, and not so many DOH works good in iran, so i just test some DNSCRYPT and naive is working, some others are connecting using 1.1.1.1 DOH. so lets sort problems with Naive 1 by 1. another mobile provider is blocking TLS heavily that's another thing im investigating 10 days till now, and the thing is its random i cant produce same problem with same clients and same network one is working one is not.

Is there a tutorial to replace the right settings for Iran?

Well each provider is blowing up the very foundation of normal connection in different strategies and they don't know the result they just keep testing (even they change the behavior hourly) so even when i want to choose a dns i have to change it everyday. its a volatile behavior which is make it hard to know what's gonna last for a week.

— Reply to this email directly, view it on GitHub https://github.com/net4people/bbs/issues/153#issuecomment-1312537340, or unsubscribe https://github.com/notifications/unsubscribe-auth/AF26MY4P6SWZ6QNGTZ35SSLWH7KAFANCNFSM6AAAAAARYQ5ODE . You are receiving this because you were mentioned.Message ID: @.***>

alirezaac commented 1 year ago

If Chrome default ClientHello is unconditionally blocked, what is the officially approved browser in Iran and what ClientHello does it use? There must be a "mainstream" browser that is needed to support mainstream productivity. I often suspect this mainstream browser would be some repackaged old version of Chrome that would not be up to date with the latest TLS fingerprint, or it could even use modified TLS parameters from Chrome upstream.

well i saw someone said on mokhaberat and things chrome is blocked, and in experience of naive proxy in iran, well i had a group of people testing it, most of them use CF proxy and think its blocked, but CF things are seem to be blocked and profiled in iran(like that universal SSl, just worked for me turning that off IDK why), second thing is they test NativeProxy on nekoray or matsuri stuff which the default dns settings are set to work in china, and not so many DOH works good in iran, so i just test some DNSCRYPT and naive is working, some others are connecting using 1.1.1.1 DOH. so lets sort problems with Naive 1 by 1. another mobile provider is blocking TLS heavily that's another thing im investigating 10 days till now, and the thing is its random i cant produce same problem with same clients and same network one is working one is not.

the reason why i said the DNS is important is i had the same problem as you see here: image

but after that GOOGLE DOH kicks in with v1.3 and resolves it: image

it takes a little longer to connect multiple of time DNS is getting blocked in seconds it will managed it, its been a week using it, and the only problem facing is them blocking the dns which i had to change it.

arandomgstring commented 1 year ago

@wkrp You are completely right. I used tlsfingerprint.io as you said and the fingerprints below are blocked. I am not quite sure how these fingerprints are different from JA3 value in Wireshark however, because as far as I am aware JA3 considers everything too:

[Irancell ISP]-> Chrome Version 107.0.5304.107 (Official Build) (64-bit) Windows [blocked] https://tlsfingerprint.io/id/9cda3d9918db9efc https://tlsfingerprint.io/id/ec577893be9fb380

On Mokhaberat ISP 9cda3d9918db9efc was blocked 7 days ago; now it works just fine.

At any rate, server does support TLS 1.3, otherwise Firefox couldn't use TLS 1.3 to negotiate with server. It seems that some of Chrome TLS fingerprints are indeed blocked. But why does Chrome use different TLS handshakes for each website to begin with? More importantly, 108.0.5359.40 (Official Build) beta sends random TLS fingerprints each time. To clarify, when I refresh https://tlsfingerprint.io website in Chrome beta, everytime I see a new fingerprint. Interestingly enough these fingerprints are not added to tlsfingerprint.io database. For example https://tlsfingerprint.io/id/12d5ee80151c78ca works fine but we see no information about it

@free-the-internet

When do you see many client hellos exactly? Might be this issue https://github.com/2dust/v2rayN/issues/2753 if you are using v2rayN and has something is wrong with your v2ray client when its ports are used by e.g. a browser trying to connect to a website?

It seems issue stems from instability of Irancell ISP itself. Because I don't see this issue on Mokhaberat.

@alirezaac Can you clarify a little? I have seen DNS poisoning specially for websites behind cloudflare CDN myself, however, if you receive correct IP address for domain of website without doh (you have darkened destination IPs so I assume you got the correct ip) how exactly did DOH help you?

wkrp commented 1 year ago

But why does Chrome use different TLS handshakes for each website to begin with? More importantly, 108.0.5359.40 (Official Build) beta sends random TLS fingerprints each time. To clarify, when I refresh https://tlsfingerprint.io website in Chrome beta, everytime I see a new fingerprint. Interestingly enough these fingerprints are not added to tlsfingerprint.io database. For example https://tlsfingerprint.io/id/12d5ee80151c78ca works fine but we see no information about it

That's a good question. Perhaps something has changed in a recent release that makes Client Hello records more random than they were previously. Try looking at recent release notes to see if there is any documented change.

There is a feature called GREASE that automatically adds some random variation to TLS records, but Chrome already used GREASE before, and tlsfingerprint.io knows about GREASE and knows to ignore it. So it must be something else, in addition.

alirezaac commented 1 year ago

@wkrp You are completely right. I used tlsfingerprint.io as you said and the fingerprints below are blocked. I am not quite sure how these fingerprints are different from JA3 value in Wireshark however, because as far as I am aware JA3 considers everything too:

[Irancell ISP]-> Chrome Version 107.0.5304.107 (Official Build) (64-bit) Windows [blocked] https://tlsfingerprint.io/id/9cda3d9918db9efc https://tlsfingerprint.io/id/ec577893be9fb380

On Mokhaberat ISP 9cda3d9918db9efc was blocked 7 days ago; now it works just fine.

At any rate, server does support TLS 1.3, otherwise Firefox couldn't use TLS 1.3 to negotiate with server. It seems that some of Chrome TLS fingerprints are indeed blocked. But why does Chrome use different TLS handshakes for each website to begin with? More importantly, 108.0.5359.40 (Official Build) beta sends random TLS fingerprints each time. To clarify, when I refresh https://tlsfingerprint.io website in Chrome beta, everytime I see a new fingerprint. Interestingly enough these fingerprints are not added to tlsfingerprint.io database. For example https://tlsfingerprint.io/id/12d5ee80151c78ca works fine but we see no information about it

@free-the-internet

When do you see many client hellos exactly? Might be this issue 2dust/v2rayN#2753 if you are using v2rayN and has something is wrong with your v2ray client when its ports are used by e.g. a browser trying to connect to a website?

It seems issue stems from instability of Irancell ISP itself. Because I don't see this issue on Mokhaberat.

@alirezaac Can you clarify a little? I have seen DNS poisoning specially for websites behind cloudflare CDN myself, however, if you receive correct IP address for domain of website without doh (you have darkened destination IPs so I assume you got the correct ip) how exactly did DOH help you?

well back in days some audience of mine asked me that they had problem i taught tls fingerprinting is started, but i suggest them to use better DOH on android devices,some could manage to find them and their problem resolved, since then there is a topic on twitter about Cloudflare and it is not good to centralize any part even it is not culprit this

Till it came to naiveproxy, and me and friends were using it for 2 weeks, but i saw people sayin that its getting blocked fast, and saw some issues on repo that chrome tls is getting blocked, so i wanted to reproduce the problem, i knew problem is from android devices cause they can attack better because of maybe weaker aes instructs, so one device it was getting blocked, another same device and network wasn't, so i came back to dns part, and it got it resolved(i'm not getting server hello till change of dns) it is the dns part , but it was hard to find one to work again,so came to share it with some dudes who are testing, they couldn't find any dns working on their side, so one them tried to resolve the dns part through proxy, but it can't be done on android the same way.

free-the-internet commented 1 year ago

@wkrp You are completely right. I used tlsfingerprint.io as you said and the fingerprints below are blocked. I am not quite sure how these fingerprints are different from JA3 value in Wireshark however, because as far as I am aware JA3 considers everything too: [Irancell ISP]-> Chrome Version 107.0.5304.107 (Official Build) (64-bit) Windows [blocked] https://tlsfingerprint.io/id/9cda3d9918db9efc https://tlsfingerprint.io/id/ec577893be9fb380 On Mokhaberat ISP 9cda3d9918db9efc was blocked 7 days ago; now it works just fine. At any rate, server does support TLS 1.3, otherwise Firefox couldn't use TLS 1.3 to negotiate with server. It seems that some of Chrome TLS fingerprints are indeed blocked. But why does Chrome use different TLS handshakes for each website to begin with? More importantly, 108.0.5359.40 (Official Build) beta sends random TLS fingerprints each time. To clarify, when I refresh https://tlsfingerprint.io website in Chrome beta, everytime I see a new fingerprint. Interestingly enough these fingerprints are not added to tlsfingerprint.io database. For example https://tlsfingerprint.io/id/12d5ee80151c78ca works fine but we see no information about it @free-the-internet

When do you see many client hellos exactly? Might be this issue 2dust/v2rayN#2753 if you are using v2rayN and has something is wrong with your v2ray client when its ports are used by e.g. a browser trying to connect to a website?

It seems issue stems from instability of Irancell ISP itself. Because I don't see this issue on Mokhaberat. @alirezaac Can you clarify a little? I have seen DNS poisoning specially for websites behind cloudflare CDN myself, however, if you receive correct IP address for domain of website without doh (you have darkened destination IPs so I assume you got the correct ip) how exactly did DOH help you?

well back in days some audience of mine asked me that they had problem i taught tls fingerprinting is started, but i suggest them to use better DOH on android devices,some could manage to find them and their problem resolved, since then there is a topic on twitter about Cloudflare and it is not good to centralize any part even it is not culprit this

Till it came to naiveproxy, and me and friends were using it for 2 weeks, but i saw people sayin that its getting blocked fast, and saw some issues on repo that chrome tls is getting blocked, so i wanted to reproduce the problem, i knew problem is from android devices cause they can attack better because of maybe weaker aes instructs, so one device it was getting blocked, another same device and network wasn't, so i came back to dns part, and it got it resolved(i'm not getting server hello till change of dns) it is the dns part , but it was hard to find one to work again,so came to share it with some dudes who are testing, they couldn't find any dns working on their side, so one them tried to resolve the dns part through proxy, but it can't be done on android the same way.

AFAIK, DNS should be used in the beginning of a any communication, so I didn't get it from your post that how DNS is related to TLS handshake. Unless, here you are talking about Encrypted SNI and DoH, right? Here, we can have Blocking in 2 stages. If Domain is blocked, using normal DNS, you can not even open a connection towards the server (TCP SYN). If you can reach in some level (SYN, SYN ACK, ACK, CLIENT_HELLO, etc), and you see that problem occurs after Client Hello, it is the SNI blocking; and only ECH or ESNI + DoH can help (of course if they are allowed by the censor). So, please write to us which one the stages you are mentioning in your post.

alirezaac commented 1 year ago

Well as I said a lot of people talk with me and test new protocols, so it often came as that they had problems with android(iPhones too but less), so we believed that it was TLS problem, but uTLS were not helping at all. so the problem were TLS version and they said the similar problem we talk here, specially in naiveproxy, so i wanted to check to see what's different on my side that i can connect and they can not. so i told them to change the dns, and mostly could solve their problem. about the stages,

chrome

This image above shows TLS handshaking of Chrome 107.0.5304.88 (Official Build) (64-bit). As you can see, after TCP acknowledgement a Client Hello was sent, however I couldn't receive Server Hello packet. I checked V2rayN packets, they had similar TLS fingerprint but its Client Hello used TLSv3 and worked fine. Then I checked Firefox 106.0.5 (64bit) packets:

now we have same problem with naiveproxy, such weirdness and random situation on each of those test me and other people done and we share, but client had problem with handshake, and me and another guy could connect, and we had very different conf and we used other DOH that not being used by masses, so i tried to use it with google DOH it was not connecting, Handshake were failing, but after 2-3 minute(Google DOH changes relays) it could connect,and tls was not forcing to v1.0 anymore. so we manually resolved the dns through proxy, but the mobile devices have poor dns options, and implementations. domains are not blocked, SNI too, it is weird, so im testing to find the culprit

free-the-internet commented 1 year ago

Well as I said a lot of people talk with me and test new protocols, so it often came as that they had problems with android(iPhones too but less), so we believed that it was TLS problem, but uTLS were not helping at all. so the problem were TLS version and they said the similar problem we talk here, specially in naiveproxy, so i wanted to check to see what's different on my side that i can connect and they can not. so i told them to change the dns, and mostly could solve their problem. about the stages,

chrome This image above shows TLS handshaking of Chrome 107.0.5304.88 (Official Build) (64-bit). As you can see, after TCP acknowledgement a Client Hello was sent, however I couldn't receive Server Hello packet. I checked V2rayN packets, they had similar TLS fingerprint but its Client Hello used TLSv3 and worked fine. Then I checked Firefox 106.0.5 (64bit) packets:

now we have same problem with naiveproxy, such weirdness and random situation on each of those test me and other people done and we share, but client had problem with handshake, and me and another guy could connect, and we had very different conf and we used other DOH that not being used by masses, so i tried to use it with google DOH it was not connecting, Handshake were failing, but after 2-3 minute(Google DOH changes relays) it could connect,and tls was not forcing to v1.0 anymore. so we manually resolved the dns through proxy, but the mobile devices have poor dns options, and implementations. domains are not blocked, SNI too, it is weird, so im testing to find the culprit

It could be very useful if you also do a capture in server side.

But this capture from the client side reminds me the blocking by SNI. Did you test with another set of Domain and SNI? As @wkrp said before, Client Hello can not decide the TLS version. The TLSv1that you see is a result of a broken TLS handshake (there are no more TLS packets after Client Hello); see this https://superuser.com/a/1618420

Nyxil commented 1 year ago

Your test is unreliable, Cause: if you run a web server and proxy server on same ip/domain at same time and try to access them using different clients, its impossible to figure out which one causing the problem, and its possible for the filtering system to affect both of them when detects one abnormal!

According to history of MF DPI's and their behavior, Sampling and processing data for DPI's is cpu insensitive task. So they don't use same algorithms of detection for blocking traffic, it will maybe cause the ip to get flagged by detection algorithm applied on tunnel's traffic and chrome fingerprint to get blocked by another algorithm and prevent chrome browser from accessing the web page, in conclusion the tunnel can affect chrome, that's the weakness of your test.

wkrp commented 1 year ago

In another thread, @arandomgstring says that blocking of Chrome-based fingerprints has stopped: https://github.com/net4people/bbs/issues/172#issuecomment-1351332838.

For example, on Irancell, every time that I used utls = chrome I couldn't connect to my VLESS + TLS + WS + NGINX setup, 2 weeks ago. On ults = firefox everything was OK. However, I think they understood their mistake, since some websites were broken because of this, and now even utls = chrome works fine.

yifeikong commented 1 year ago

With the release of TLS permutation in Chrome 110, this kind of block will be impossible before they updating their fingerprinting algorithm.

arandomgstring commented 1 year ago

@yifeikong

How unfortunate! Now, each time Chrome attempts to open a webpage with a random fingerprint, in the case of blockage, it will automatically switch to another fingerprint to open the webpage. VPN clients don't support this feature (yet), therefore, they try to use the same fingerprint over and over again until either they connect to server or fail to do so, resulting an obvious pattern for proxy servers.