Confirmed block of default Snowflake broker in China, 2023-05-12 to 2023-05-15

[IrradiatedKiwi]

Confirmed block of default Snowflake in China

Tor browser version: Latest Stable

Problem: Around afternoon time on May 12 2023,tor browser's built in snowflake stops working Connection asist doesn't work neither.

Diagnoses: In Wireshark I see full package loss(TCP Spurious Transmission) to the supposely existing bridge front server(i think that what it is called) ip after initial handshake.

partial tor logs: 2023-05-12 19:47:00.789 [WARN] 2 connections have failed: 2023-05-12 19:47:00.790 [WARN] 2 connections died in state handshaking (TLS) with SSL state SSLv3/TLS write client hello in HANDSHAKE

Interestingly I tried to ping the default existing bridge and receives 0 packet loss I Also tried dig the existing bridge(front),the result is normal too.So it is not my DNS problem.

Conclusion: Is it possible just some misconfiguration on the default Snowflake side? But since the connection always die while 'handshaking (TLS) with SSL state SSLv3/TLS',I doubt so.

I think the GFW had a recent upgrade that need to take notice.

I don't know if it is also related to https://github.com/net4people/bbs/issues/248 Since I am not sure if it is related I opened a new issue

ps @wkrp I don't know if you remember me,Sorry i couldn't respond to the email you reply to me about an issue regarding cloudflare dns quite a long time ago.Please understand It is kinda a struggle to me to even login to my email and github.Thank you for your reply and though i am not using cloudflare anymore I will try the sugguestion you mention next time if have smililar problem.

[wkrp]

Thanks for the news. We will keep an eye on it.

These are OONI charts relevant to Snowflake in China:

• https://explorer.ooni.org/chart/mat?probe_cc=CN&since=2023-04-22&until=2023-05-22&time_grain=day&axis_x=measurement_start_day&test_name=web_connectivity&domain=cdn.sstatic.net
• https://explorer.ooni.org/chart/mat?probe_cc=CN&since=2023-04-22&until=2023-05-22&time_grain=day&axis_x=measurement_start_day&test_name=torsf

See #197 for some suggestions for workarounds for when the default Snowflake rendezvous was temporarily blocked in Iran. The AMP cache rendezvous won't work unless you can find an unblocked Google front domain, but you can also find a list of possible alternative front domains there and instructions for activating them.

You may be able to get more information from the snowflake-client log, but you will have to enable it manually. Here are instructions from #131:

We would like to see snowflake-client logs from failed connections. This log provides more information (e.g. "unable to create broker channel") than the Tor log does ("Bootstrapped 10%"), but you need to take special steps to activate it. In Tor Browser desktop, edit the file Browser/TorBrowser/Data/Tor/torrc-defaults. Find the line that starts with

ClientTransportPlugin snowflake

and add this to the end of the line:

-log snowflake-client.log -log-to-state-dir

Then, when you restart Tor Browser, you will find the log at:

• linux: Browser/TorBrowser/Data/Tor/pt_state/snowflake-client.log
• windows: Browser\TorBrowser\Data\Tor\pt_state\snowflake-client.log
• mac: ~/Library/Application Support/TorBrowser-Data/Tor/pt_state/snowflake-client.log (Use Go to Folder... in the Finder menu.)

[IrradiatedKiwi]

Hello Thank you for the reply

[IrradiatedKiwi]

We would like to see snowflake-client logs from failed connections. This log provides more information (e.g. "unable to create broker channel") than the Tor log does ("Bootstrapped 10%"), but you need to take special steps to activate it. In Tor Browser desktop, edit the file Browser/TorBrowser/Data/Tor/torrc-defaults. Find the line that starts with

ClientTransportPlugin snowflake

and add this to the end of the line:

-log snowflake-client.log -log-to-state-dir

Then, when you restart Tor Browser, you will find the log at:

• linux: Browser/TorBrowser/Data/Tor/pt_state/snowflake-client.log
• windows: Browser\TorBrowser\Data\Tor\pt_state\snowflake-client.log
• mac: ~/Library/Application Support/TorBrowser-Data/Tor/pt_state/snowflake-client.log (Use Go to Folder... in the Finder menu.)

A little hint for anyone who wish to help

the '-log snowflake-client.log -log-to-state-dir' should be a space key not return key at the end of the line

for example,it should be like this

ClientTransportPlugin snowflake exec ./TorBrowser/Tor/PluggableTransports/snowflake-client -log snowflake-client.log -log-to-state-dir

NOT This

ClientTransportPlugin snowflake exec ./TorBrowser/Tor/PluggableTransports/snowflake-client
-log snowflake-client.log -log-to-state-dir

[ghost]

I deleted the previous testing results, which is wrong. Seems my local ISP has other restrictions to Fastly IPs.

wrong results

Yes, from my testing: about 3 times of TLS client hellos containing the SNI `cdn.sstatic.net` to a Fastly CDN IP will cause a temporary packet drop to its 443 port. It differs from what GFW has done for `github.com` or `store.steampowered.com`: the packet drop happens to any IPs outside China instead of a specific AS like Fastly.

[UjuiUjuMandan]

Opened a thread on Tor forum: https://forum.torproject.net/t/snowflake-bridge-does-not-work-in-china-since-days-ago/7635 .

[IrradiatedKiwi]

Opened a thread on Tor forum: https://forum.torproject.net/t/snowflake-bridge-does-not-work-in-china-since-days-ago/7635 .

Thank you for relaying this to tor forum

What I feel strange is that it looks like the front domains are not blocked themselves.if you ping them directly the packet loss seems fine.due to the persistence of censor some packet loss is expected.

But When you use snowflake,it somehow triggers blocking.And this didn't happen so intensely before around the afternoon time of 2023-5-12 .

here is the ping result to cdn.sstatic.net for references.

100 packets transmitted, 100 received, 0% packet loss, time 743327ms

So yes it doesn't seems like the problem is the direct blocking of the front. But when trying to connect with the builtin snowflake of tor browser it does trigger the blocking somehow.The Wireshark result shows TCP Retransmission or sometimes SYN,ACK error right after Client Key Exchange.

I didn't test fastly.jsdelivr.net as you do but i guess the result might be similiar

[UjuiUjuMandan]

if you ping them directly the packet loss seems fine.

ping test should always go through because the SNI-based blocking usually only happens on TCP 443 port. You may use curl to perform a HTTPS request or netcat to probe the 443 port periodically during the Snowflake initialization process.

But when trying to connect with the builtin snowflake of tor browser it does trigger the blocking somehow.The Wireshark result shows TCP Retransmission or sometimes SYN,ACK error right after Client Key Exchange.

Does this blocking target on that Fastly CDN IP? I mean the IP of snowflake-broker.torproject.net.global.prod.fastly.net. Seems Tor will use front's IP instead of url's.

UPDATE: And, I figured out that it's the IP of the front domain: cdn.sstatic.net is blocked in my area. I tried changing it to another Fastly domain mirrors.fastly.net and it works fine.

[UjuiUjuMandan]

In my area there're more limitations to Fastly CDN IPs:

1. Almost all of them are blocked including IPs of fastly.com, fastly.net, fastly.jsdelivr.net and of course cdn.sstatic.net.
2. Two repeated HTTPS request with any same SNI (even including www.gov.cn) to that IP will result in a temporary ban of that IP, even the ICMP won’t go through, requests without SNI won't trigger that. Interval between the two requests should below 60 seconds. The ban lasts for 180 seconds. The ban does not target on some of Fastly CDN IPs.

For 2, some may say it's Fastly applied some types of rate limit, but how could it be 180 seconds accidentally, 180 is very typical of GFW.

[wkrp]

There was a paper published in January 2023 about detecting Snowflake connections. One of the authors is professor Cheng Guang (程光), who works on a lot of topics related to encrypted traffic analysis.

F-ACCUMUL: A Protocol Fingerprint and Accumulative Payload Length Sample-Based Tor-Snowflake Traffic-Identifying Framework

The Snowflake detection algorithm has two parts: a coarse-grained prefilter based on DNS queries; and a fine-grained machine learning classifier on DTLS handshake features. The DNS prefilter (Section 4.1) looks for DNS queries to both the default front domain cdn.sstatic.net and some number of default STUN servers. Source IP addresses that are caught in the prefilter then move on to the DTLS fingerprint classification (Section 4.2). This uses features (Table 1) extracted from the DTLS Client Hello and Server Hello:

Feature	Client Hello	Server Hello
DTLS record version	✓	✓
Record length	✓	✓
Fragment offset	✓	✓
Fragment length	✓	✓
Cipher suite length	✓	✓
Cipher suites	✓
Extension Length	✓	✓
Extensions	✓	✓
Cipher suit chosen		✓

Figure 6 shows the relative importance of DTLS fingerprint features in their classifer:

Feature	Importance	Comment
S_cipher_chosen_b'\xc0/'	0.18	0xc02f = TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
S_fragment_len	0.12
C_cipher_suits_5	0.10
C_cipher_suitlen	0.09
S_HP_len	0.08	do not know what this is
S_cipher_chosen_b'\xc0+'	0.06	0xc02b = TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
C_HP_len	0.05	do not know what this is
S_extensions_8	0.05
C_fragment_len	0.04
S_extensions_3	0.04

Just because someone wrote a paper, does not mean that the technique they propose is actually used in practice. But it gives some ideas to think about.

[wkrp]

2. Two repeated HTTPS request with any same SNI (even including www.gov.cn) to that IP will result in a temporary ban of that IP, even the ICMP won’t go through, requests without SNI won't trigger that. Interval between the two requests should below 60 seconds. The ban lasts for 180 seconds.

That may be a very relevant fact, that two repeated HTTPS requests are required. Back in December 2022, Tor Browser 12.0 added a second Snowflake bridge. Unfortunately, due to technical limitations in the integration with Tor, having two bridges means that the Snowflake client sends two broker rendezvous messages (to the configured front domain) when it starts up. See the notes here, which basically describe how it works now:

• then there can be multiple snowflake bridge lines in torrc, each with different bridge fingerprints. load balancing will come from the tor client's local random selection.
  • there is a "thundering herd" concern with the way tor currently uses multiple bridge lines. tor will attempt to connect to all of them at once, and keep using only one of them. This means N broker transactions and N STUN exchanges, and possibly N−1 proxies held idle.
  • maybe it's not such a big problem, if N is not too large
  • an alternative would be a modification to tor where it shuffles its bridge list, and tries only one at a time
  • another alternative is to write the torrc file dynamically: choose a random bridge line, then write torrc containing only that one randomly selected bridge line

If that is the case, that simultaneous broker rendezvous messages is a feature used to detect Snowflake connections, you may be able to work around it by disabling one of the bridges in Tor Browser. In the bridges configuration, if you enabled Snowflake as a built-in bridge, it should show two bridges:

• snowflake 192.0.2.3:80 2B280B23E1107BB62ABFC40DDCC8824814F80A72 ... (✈️☎️👑🏈)
• snowflake 192.0.2.4:80 8838024498816A039FCBBAB14E6F40A0843051FA ... (🍕🏡🐠🚀)

You can copy and paste one of the bridge lines as a manually entered bridge, rather than a built-in bridge. (Make sure you copy the entire bridge line, everything from snowflake ... utls-imitate=hellorandomizedalpn.) The bridge with fingerprint 8838024498816A039FCBBAB14E6F40A0843051FA is currently less loaded, so it's better to use that one.

Current versions of Orbot (before v17.0) also only have one Snowflake bridge configured; you could try Snowflake in Orbot and see if it works better than Tor Browser desktop.

Issue #40578 in Tor ("Let bridge users choose to only reach their first working bridge") is to fix the multiple rendezvous problem. It has a patch, but no action beyond that so far.

[UjuiUjuMandan]

having two bridges means that the Snowflake client sends two broker rendezvous messages

Basically Tor can do this with only one Client Hello by reusing same HTTPS connection. I'm wondering will Tor make two separated TLS Client Hello which will trigger the 180s ban or not.

EDIT: Yes. From the Wireshark records, Tor will indeed send Client Hello twice. Since there are only two Snowflake bridges, this twice is enough, Tor can bootstrap to 100%, but not if there is one more, because the opportunity has been exhausted, and the GFW has already started blocking the IP.

[IrradiatedKiwi]

ping test should always go through because the SNI-based blocking usually only happens on TCP 443 port. You may use curl to perform a HTTPS request or netcat to probe the 443 port periodically during the Snowflake initialization process.

Thank you,As I am not technical proficient I am glad someone more knowledgeable is willing to help.

If that is the case, that simultaneous broker rendezvous messages is a feature used to detect Snowflake connections, you may be able to work around it by disabling one of the bridges in Tor Browser.

I think you hit the jackpot here. ~I did remove one of the default bridge and using only 1.The Snowflake connection is working for now.~ But It seems this workaround doesn't work really well.I did successfully connected with only 1 default snowflake,but only for no more than 10 minutes.I tried again on each bridge,they are not working now. Anyway Does it means that only have 1 bridge is a better option for now?

Just because someone wrote a paper, does not mean that the technique they propose is actually used in practice. But it gives some ideas to think about.

It is indeed a good lead. ~I think they might actually applied this technique. so if the technique does applied to the GFW.I am afraid removing one brdige is still just a temporal solution.~ Since the workaround doesn't work well for me,I have no clue here.I hope more people with better knowledge than me could help.

And currently I am having trouble with the other workround.I can only hope the tor and snowflake team will find some more stable countermeasure to this.

@wkrp @UjuiUjuMandan Thank you for helping me in this matter.

[IrradiatedKiwi]

Sorry I accidentally close the issue when i was trying to reply.It is still open and i hope more meaningful results can be found.

[wkrp]

Opened an issue in the Tor Project tracker:

Blocking of Snowflake in China, 2023-05-12 (#40038)

[IrradiatedKiwi]

Thank you

Opened an issue in the Tor Project tracker:

Blocking of Snowflake in China, 2023-05-12 (#40038)

I will try to make my previous attempt regarding the brdige work around more clearly.

I removed one brdige and only used 8838024498816A039FCBBAB14E6F40A0843051FA.The connection was slow but bootstrap succeeded.I could open websites then,althought it was slow. It worked for 5-10 minutes,then the connection failed,websites became unusable and shows time out. I closed tor browser and then tried to reconnect with the same brdige,this time it doesn't work.The bootstrap failed.

And then I swtiched to 2B280B23E1107BB62ABFC40DDCC8824814F80A72, bootstrap failed and couldn't even establish connection.

Trien I tried the double bridges,and as expected didn't work.

The block seems lifted for now.Even the default built-in double bridge works for now.I can no long reproduce that behavior.