vmess+ws+tls works on windows/mac, but NOT in Android

[ea0fdb]

We have run two servers a while ago. All of the client was working fine until a week ago. Now computer clients are working fine, but android clients don't work in any of the servers. We have tested v2rayNG with both versions 1.7.20 and 1.7.21 (uTLS fingerprint). It doesn't work on Raspberry Pi either. Maybe related to arm processors.

Server/Protocol: vmess+ws+tls with v2fly-core and nginx Working Clients: Windows / Linux / Mac, with v2fly-core. iOS is working too with Shadowrocket. Not Working Clients: Almost any android client, including v2rayNG. Raspberry Pi Linux with v2fly-core. Country: Iran Related Issue & server logs: https://github.com/2dust/v2rayNG/issues/1685

Please share any idea you have about this situation. Maybe this was happened in China before.

Thanks.

[ea0fdb]

@wkrp Based on this issue, It seems that this is happening in China too. So, please add the China label too.

[wkrp]

It doesn't work on Raspberry Pi either. Maybe related to arm processors.

It is possible it has something to do with the processor type. OONI reported something similar once. (However, I don't know how this kind of hardware dependence might interact with uTLS.)

In mid-April 2021, we discovered the likely cause of blocking. V2Ray developers previously documented this issue in v2fly/v2ray-core#557.

The root cause is that the TLS Client Hello generated by Golang on Android devices changes depending on Golang’s understanding of the hardware capabilities. Golang’s TLS library uses a hardware implementation of AES when possible. Otherwise, it falls back to AES code written in pure Go. According to the official docs, such code is not constant over time. Probably, for this reason, when Golang thinks that there is no hardware support for AES, it reorders the preferred cipher list in the Client Hello. This reordering aims to significantly reduce the probability that the server will select AES ciphers by moving such ciphers towards the bottom of the list.

There was also a report in another thread that snowflake-client worked on macOS and Windows desktop, but not in Orbot on Android or iOS.

It will help if you can get a packet capture of the the TLS Client Hello under both conditions, in order to check whether there are any differences. On desktop, you can use tcpdump or Wireshark. The tcpdump command will be something like this (replace $V2RAY_HOST and $V2RAY_PORT):

tcpdump -U -w v2ray-desktop.pcap -f "host $V2RAY_HOST and port $V2RAY_PORT"

On Android, I have had luck using PCAPdroid. You can select a specific app to capture traffic from and save to a pcap file.

You can then upload the pcap files to https://tlsfingerprint.io/pcap and see if they have the same fingerprint.

To see what's inside each Client Hello record, you can open the pcap files in Wireshark, select the "Client Hello" packet, right-click "Secure Sockets Layer" in the tree view, select "Expand Subtrees", then right-click again and "Copy → All Visible Selected Tree Items". Then paste it into a file. Or you can use the tshark command like so:

tshark -V -O ssl -Y ssl.handshake.ciphersuites -r v2ray-desktop.pcap

Don't post the raw pcap files here, because they may contain IP addresses you do not want to reveal.

[ea0fdb]

@wkrp Thanks for your reply.

I have run the experiments. I used PCAPdroid on Android and tcpdump on other platforms. The results are summarized in the table below.

	Server	Device	Client	Is Working?	TLS Fingerprint
1	S1:vmess+ws+tls	mac	v2fly (5.1.0)	yes	844**1ad
2	S1:vmess+ws+tls	android	xray (1.6.0)	no	750e3f0f585283bd
3	S1:vmess+ws+tls	raspberry pi	v2fly (5.1.0)	no	750e3f0f585283bd
----	-----------------	--------------	---------------	-------------	-----------------
4	S2:vless+ws+tls	raspberry pi	v2fly (5.1.0)	no	750e3f0f585283bd
5	S2:vless+ws+tls	raspberry pi	xray (1.6.0)	no	750e3f0f585283bd
6	S2:vless+ws+tls	raspberry pi	xray (1.6.0) + uTLS (any browser)	no	750e3f0f585283bd

Both servers S1 and S2 are working for linux/mac/windows. None of them are working for Raspberry Pi or Android device (with V2rayNG). The servers have different IPs, Hostnames, and Protocols.

It's interesting that:

1. The use of uTLS didn't change the fingerprint for these device and protocols. It's worth mentioning that the uTLS configuration will change the fingerprint for some other protocols, even in the Raspberry Pi.
2. In these experiments the fingerprint of ARM devices are 750e3f0f585283bd for both servers.

I suspect that the 750e3f0f585283bd fingerprint causes the blocking.

[OnceUponATimeInAmerica]

You could try a VLESS+TCP+TLS config (e.g. using ProxySU) and then set the uTLS fingerprint to "Chrome" in V2rayNG (new release). Also Trojan configs seem to work well with custom uTLS fingerprints.

Or you could use a client like Sagetnet/Matsuri and try setting "browser forwarding" in the server's options.

[cafelinux]

It's all about interfering on TLS handshake by government. even I can not login to Gmail today by getting this msg (The server cannot process the request because it is malformed. It should not be retried.) blocking and manipulating on port 443 and TLS caused all of these problems.

[wkrp]

@ea0fdb thank you, that is super helpful. It looks to me like uTLS support is not working—it may be a bug in xray or possibly in uTLS.

1. The fingerprint 750e3f0f585283bd looks like the default Go crypto/tls fingerprint for platforms without AES acceleration hardware since go1.17. (Take the list cipherSuitesPreferenceOrderNoAES, append defaultCipherSuitesTLS13NoAES, and subtract disabledCipherSuites, and you get exactly the ciphersuite list of 750e3f0f585283bd.) So yeah, it looks like uTLS is not taking effect, for some reason.

   @gaukas, you are more experienced. Is there any reason why uTLS might fail to take effect on ARM? Or is it more likely that xray is using uTLS ineffectively?

   @ea0fdb, are you able to test uTLS support in xray on a non-ARM platform? If it changes the fingerprint on non-ARM but does not change the fingerprint on ARM, it would tell us something.

2. The fingerprint 750e3f0f585283bd is very similar to the fingerprint adfe55afa6f23950 which I observed being produced by snowflake-client in Tor Browser for Linux and Orbot for Android at https://github.com/net4people/bbs/issues/131#issuecomment-1272178121. tlsfingerprint.io has a fingerprint comparison feature. The differences are:

   • adfe55afa6f23950 additionally has "h2" in ALPN.
   • adfe55afa6f23950 additionally supports TLS 1.0 and TLS 1.1.

3. I now suspect that the reason I got the fingerprint adfe55afa6f23950 with Tor Browser for Linux is that I was running it in a VM. The VM may not have indicated support for hardware acceleration of AES, which would cause Go crypto/tls to switch to the NoAES ciphersuites, the same as on ARM.

This all gives us a good hypothesis as to the features used in recent TLS blocking, in Iran at least. It includes the fingerprints of go1.17+ crypto/tls without AES acceleration, which are likely the most commonly occurring fingerprints among Go-based tools running on mobile platforms. These fingerprints include at least 750e3f0f585283bd and adfe55afa6f23950.

[gaukas]

Is there any reason why uTLS might fail to take effect on ARM

I am not aware of any problem that would cause the fingerprint parroting silently fail on ARM in uTLS.

I just ran a test on my Raspberry Pi (armv7 running armv6 go), uTLS worked fine and parrots both Chrome and Firefox fingerprints correctly. While being not very familiar with the code base of xray, I couldn't answer the question of what causes uTLS not functioning with xray.

[gfw-report]

I suspect that the 750e3f0f585283bd fingerprint causes the blocking.

Great analysis, @ea0fdb!

We have not been able to trigger the blocking with 750e3f0f585283bd fingerprint yet, however. In particular, we wrote a program that uses uTLS to make TLS connections with that fingerprint. We then did the following three experiments. In all three experiments, we captured the packets from both client and server to observe if there is any packet manipulation :

1. repetitively sending TLS client hello messages to both 443 and non-443 port of a host outside of China. The host is actually a TCP sink server, which only accepts the hello message but does not respond any data;
2. repetitively making TLS handshakes to both 443 and non-443 port of a host outside of China. The host is actually a TLS sink server, which only accepts the TLS handshakes but does not respond any data;
3. repetitively making TLS handshakes, and then follow by sending 517 bytes and then 126 bytes traffic, to mimic TLS-over-TLS's packet length through the TLS tunnel, to both 443 and non-443 port of a host outside of China. The host is still a TLS sink server, which only accepts the TLS handshakes and the following data but does not respond any data.

In all three cases, we cannot trigger the blocking immediately. Note that this does not necessarily mean that the censor does not consider TLS fingerprint when making a blocking decision, rather, it just says the TLS fingerprint is not a sufficient condition to trigger the blocking. There may be other factors the censor considers together with the fingerprint, for example:

1. The censor may only apply the blocking to certain IP addresses, like what we've seen for the dynamic blocking of Shadowsocks, and our server was not within those IP addresses.
2. The censor may consider in addition consider packet length in TLS handshake and the first several packets after the handshake.
3. The censor may require seeing more traffic than we sent to trigger the blocking.

In any case, since 750e3f0f585283bd is a very unique fingerprint, we should consider changing it.

BTW, we have received some reports that support the TLS fingerprint blocking hypothesis. In particular, the person setup two open ports on the same server, for one port, people used the trojan-go with latest uTLS, which is not blocked; for the other port, people used an android client, which was blocked.

//cc @2dust

[ea0fdb]

Hi everyone. Thanks for your replies.

Are you able to test uTLS support in xray on a non-ARM platform? If it changes the fingerprint on non-ARM but does not change the fingerprint on ARM, it would tell us something.

@wkrp For vmess+ws+tls/vless+ws+tls on a non-ARM device, using uTLS with any browser, produces the same TLS Fingerprint as not using uTLS. All of them are still connecting and working fine.

Now my concern is why doesn't uTLS change the TLS Fingerprint for vmess+ws+tls/vless+ws+tls. Actually I have already seen that uTLS is working with xray for vless+tls (without websocket).

It seems that the issue is not necessarily about ARM devices, as I suspect that it's about effectiveness of uTLS with xray when using websocket.

Note that I'm changing this key in the json configuration to use uTLS with xray 1.6.0.

{
  "outbounds": [
    {
      "streamSettings": {
        "tlsSettings": {
          "fingerprint": "chrome",
          ...
        }, ...
      }, ...
    },
  ], ...
}

[gaukas]

as I suspect that it's about effectiveness of uTLS with xray when using websocket

Nice catch. This reminds me that there's no H2 support for websocket libraries in Go(unfortunately). So possibly xray had to override ALPN with only http1.1.

[wkrp]

From https://github.com/net4people/bbs/issues/136#issuecomment-1280791520, I found the recent commit https://github.com/XTLS/Xray-core/commit/93c7ebe38245a44fb8d7f029677a16e9def1187c in xray, with commit message "Added utls to http2 transport". Maybe it helps?

[klzgrad]

uTLS should verify on its own its fingerprint on the wire and report error if not so, instead of relying on user tcpdumping. Though I don't know if this is technically doable.

[gaukas]

uTLS should verify on its own its fingerprint on the wire and report error if not so

Definitely possible, we have UnmarshalClientHello() to convert []byte into ClientHelloMsg. But would you like to elaborate on how could it be useful? Given that uTLS doesn't implement a hash-to-ClientHello converter, just simply verifying if clientHello == UnmarshalClientHello(clientHello.marshal()) looks redundant to me.

(Assuming all Extensions/FakeExtensions are implemented correctly, of course)

[HirbodBehnam]

Hello I actually like to note that I did another pull request which adds utls to websocket connections of xray. You can read about the implementation here: https://github.com/XTLS/Xray-core/pull/1256 Please do leave a comment if you think I did anything wrong!

P.S: As a workaround you can try specifying something like "cipherSuites":"TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256" in the tls settings of xray client. This will change the fingerprint thus the traffic will be allowed.

[gfw-report]

This is a small script that we've been using to get the TLS fingerprint IDs from pcap files, which you may find handy:

https://github.com/gfw-report/get-tls-fingerprints

//cc @ea0fdb @tomac4t

[Salarshafiei]

@0xLem0nade I tried installing your .apk but it fails to install on my phone.

[Salarshafiei]

@0xLem0nade I uninstalled the PS version first and then I tried installing each and every file in the assets. Non of them could be installed unfortunately. It just says App not installed. My phone is a Samsung Note 9 (exynos9810), updated to the latest version. Android V10, aarch64 (from CPU-Z)

[cafelinux]

@0xLem0nade I uninstalled the PS version first and then I tried installing each and every file in the assets. Non of them could be installed unfortunately. It just says App not installed. My phone is a Samsung Note 9 (exynos9810), updated to the latest version. Android V10, aarch64 (from CPU-Z)

Okay I did reset some signing configs and uploaded the new builds here. Get the v2rayNG_1.7.22_arm64-v8a.apk, reboot your device and then install the apk. Let me know if it works!

Hi, The problem still there, same as before, no problem with other Xray clients on Android, but v2rayNG 1.7.22 still not connecting .

[ea0fdb]

Hi,

With the new pre-release of v2rayNG 1.7.23, uTLS is now working fine and the issue is resolved. We can now close the issue if there is not anything else to discuss.

Thanks everyone for your time and supports. @HirbodBehnam Thanks for your contribution (دمت گرم 😁).

[Ka6k]

Still uTLS is not wotking for TCP+XTLS. For VLESS+TCP+XTLS & Trojan+TCP+XTLS the fingerprint remains the same for any browser option as not using uTLS.

[ghost]

Blog post expanding on the @0xLem0nade config file for v2rayNG + HTTP/2 + TLS from the other thread is at https://freejohn123.github.io/2022/10/23/v2rayNG-http2-tls.html

[cafelinux]

Blog post expanding on the @0xLem0nade config file for v2rayNG + HTTP/2 + TLS from the other thread is at https://freejohn123.github.io/2022/10/23/v2rayNG-http2-tls.html

Hi and thank you, but still have problem with this config, even try to change everything and testing ... The time is correct for both server and client.

"TLS handshake error from MY_IP: read tcp SERVER_IP:443->MY_IP: i/o timeout"

[HirbodBehnam]

As a side note, you might want to use firefox instead of chrome when using the fingerprint option. Iran blocks chrome traffic to some IP addresses. For example my own server is from hetzner and I do have the exact problem; So I simply use firefox's fingerprint. Probably related: #118

[free-the-internet]

Blog post expanding on the @0xLem0nade config file for v2rayNG + HTTP/2 + TLS from the other thread is at https://freejohn123.github.io/2022/10/23/v2rayNG-http2-tls.html

Hi and thank you, but still have problem with this config, even try to change everything and testing ... The time is correct for both server and client.

"TLS handshake error from MY_IP: read tcp SERVER_IP:443->MY_IP: i/o timeout"

This is also happened to me. @HirbodBehnam Changing fingerprint to Firefox is not working.

[free-the-internet]

Blog post expanding on the @0xLem0nade config file for v2rayNG + HTTP/2 + TLS from the other thread is at https://freejohn123.github.io/2022/10/23/v2rayNG-http2-tls.html

Hi and thank you, but still have problem with this config, even try to change everything and testing ... The time is correct for both server and client. "TLS handshake error from MY_IP: read tcp SERVER_IP:443->MY_IP: i/o timeout"

This is also happened to me. @HirbodBehnam Changing fingerprint to Firefox is not working.

Hi, apparently, free domain was blocked. I moved to new domain and now it is working.

[cross-hello]

You have done a good job. תודה אתה

---

From: free-the-internet @.> To: net4people/bbs @.> CC: Subscribed @.**> Date: Oct 25, 2022 06:08:31 Subject: *Re: [net4people/bbs] vmess+ws+tls works on windows/mac, but NOT in Android (Issue #139)

Blog post expanding on the @0xLem0nade[https://github.com/0xLem0nade] config file for v2rayNG + HTTP/2 + TLS from the other thread is at https://freejohn123.github.io/2022/10/23/v2rayNG-http2-tls.html

Hi and thank you, but still have problem with this config, even try to change everything and testing ... The time is correct for both server and client. "TLS handshake error from MY_IP: read tcp SERVER_IP:443->MY_IP: i/o timeout"

This is also happened to me. @HirbodBehnam[https://github.com/HirbodBehnam] Changing fingerprint to Firefox is not working.

Hi, apparently, free domain was blocked. I moved to new domain and now it is working.

— Reply to this email directly, view it on GitHub[https://github.com/net4people/bbs/issues/139#issuecomment-1289683448], or unsubscribe[https://github.com/notifications/unsubscribe-auth/AKGBAYFASGTPSWNYIME25BLWE4CF5ANCNFSM6AAAAAARGNNA3M]. You are receiving this because you are subscribed to this thread.[Tracking image][https://github.com/notifications/beacon/AKGBAYEBV7LZ4CLZDDDVPKDWE4CF5A5CNFSM6AAAAAARGNNA3OWGG33NNVSW45C7OR4XAZNMJFZXG5LFINXW23LFNZ2KUY3PNVWWK3TUL5UWJTSM34A7Q.gif]Message ID: @.***>

[poorp]

Hi, I didn't really follow the discussion here but here is some added info: I use vless+ws and it's working well but when I try to add TLS it doesn't work. I tried Cloudflare and Hostinger as my CDN.

[SakuraSakuraSakuraChan]

@wkrp Based on this issue, It seems that this is happening in China too. So, please add the China label too.

I think this issue is about configuration importing affairs, not about blocking