Unexplained drop in Snowflake client polls and bandwidth, testers wanted

[wkrp]

Since the beginning of protests and shutdowns in Iran, we have been applying numerous performance optimizations to the Snowflake bridge. As a result, peak bandwidth has risen from 1 Gbps to 4 Gbps and the estimated number of simultaneous users has grown from 20,000 to 100,000. But about two days ago, at 2022-10-04 17:15, usage dropped suddenly and drastically. Likely related, users reported failed Snowflake connection from Iran starting 2022-10-05. We have been investigating, but the exact cause is not clear. We would like help, especially logs from failed Snowflake connections.

You can see the sudden drop in client polls at the Snowflake broker:

It was accompanied by a simultaneous drop in bandwidth at the bridge:

The most likely explanation for a pattern like this is censorship of the broker, probably in Iran. Clients cannot reach the broker, therefore they do not get proxy service, therefore they do not use bridge bandwidth. But the evidence is somewhat ambiguous, and anyway we need to understand how the broker is being blocked, if indeed that is what's happening.

Some of the evidence:

• Usage decreased about 70% in Iran, but not only in Iran. The US, Tunisia, and Mauritius also had decreases of 40–70%. Some countries' counts remained about the same (Russia, China), and others' increased (Germany, United Kingdom, France).
• OONI measurements of Snowflake connectivity and the default broker domain front do not show any notable increase in blocking in the past couple of days.
• We have investigated possible causes of slowness at the broker and the bridge (e.g. network misconfigurations), and so far have not found anything.

There are various possible explanations. For example, connectivity to the broker domain front may be blocked only for certain TLS fingerprints. Perhaps OONI coverage is lacking in the networks most affected. For this we would like help.

How you can help

• We would like to see snowflake-client logs from failed connections. This log provides more information (e.g. "unable to create broker channel") than the Tor log does ("Bootstrapped 10%"), but you need to take special steps to activate it. In Tor Browser desktop, edit the file Browser/TorBrowser/Data/Tor/torrc-defaults. Find the line that starts with

  ClientTransportPlugin snowflake

  and add this to the end of the line:

  -log snowflake-client.log -log-to-state-dir

  Then, when you restart Tor Browser, you will find the log at:

  • linux: Browser/TorBrowser/Data/Tor/pt_state/snowflake-client.log
  • windows: Browser\TorBrowser\Data\Tor\pt_state\snowflake-client.log
  • mac: ~/Library/Application Support/TorBrowser-Data/Tor/pt_state/snowflake-client.log (Use Go to Folder... in the Finder menu.)
• If you use Orbot or Onion Browser, try choosing the Built-in snowflake (AMP) option, which uses an alternative form of rendezvous.

[iRhonin]

There is now a release available of Orbot that enables uTLS for Snowflake (from #125 (comment)).

You can download APKs here: https://github.com/guardianproject/orbot/releases/tag/16.6.3-BETA-2-tor.0.4.7.10

This release makes it possible to see the snowflake-client log. If there's a failure to connect, it will help us figure out what is going wrong. Enable Settings → Debug Log, then go back to the main screen and Start. Tap on a status message to show the tor log, then tap the snowflake snowflake icon to view the snowflake-client log.

I can confirm this works in Iran.

[free-the-internet]

There is now a release available of Orbot that enables uTLS for Snowflake (from #125 (comment)).

You can download APKs here: https://github.com/guardianproject/orbot/releases/tag/16.6.3-BETA-2-tor.0.4.7.10

This release makes it possible to see the snowflake-client log. If there's a failure to connect, it will help us figure out what is going wrong. Enable Settings → Debug Log, then go back to the main screen and Start. Tap on a status message to show the tor log, then tap the snowflake snowflake icon to view the snowflake-client log.

Tonight, it worked also for a friend using DSL line. Mobile data has not tested yet.

It could be very nice if reporters also include the type of network and the operator.

[free-the-internet]

There is now a release available of Orbot that enables uTLS for Snowflake (from #125 (comment)). You can download APKs here: https://github.com/guardianproject/orbot/releases/tag/16.6.3-BETA-2-tor.0.4.7.10 This release makes it possible to see the snowflake-client log. If there's a failure to connect, it will help us figure out what is going wrong. Enable Settings → Debug Log, then go back to the main screen and Start. Tap on a status message to show the tor log, then tap the snowflake snowflake icon to view the snowflake-client log.

I can confirm this works in Iran.

@wkrp Another friend tested it in MTN network, and he also could connect via new Orbot. However it couldn't connect a voice call; neither in Whatsapp nor Telegram. Probably due to very low BW?

[SaSyda]

@gusgustavo sorry that i deleted my comment, realized the fact and did so. will check on android and will be back with more details and outcomes. cheers.

[wkrp]

However it couldn't connect a voice call; neither in Whatsapp nor Telegram. Probably due to very low BW?

Yes, it might be due to low bandwidth, but I'm not sure.

Tor has been experiencing a DDoS attack for several months, which could make the connection slow, independent of Snowflake or network throttling.

I was trying Tor Browser for Android with Snowflake today and the performance was poorer than I am used to, using Snowflake on desktop. It couldn't get through a 10 MB without the download being interrupted and having to be restarted. Just migrating between two Snowflake proxies should not be enough to make that happen. I don't know what the cause could be.

[SaSyda]

ok just tested the new android version on a Xiaomi phone and it works like a charm on adsl connection, but not so good on MCI connection. So enabled the logging procedure, worked fine but as i pressed the share button above, the app asserted and closed its window and vpn connection. Some kind of a file or memory permission problem i guess? else tell me how to solve it so that i can share my logs with u. Cheers.

[free-the-internet]

ok just tested the new android version on a Xiaomi phone and it works like a charm on adsl connection, but not so good on MCI connection. So enabled the logging procedure, worked fine but as i pressed the share button above, the app asserted and closed its window and vpn connection. Some kind of a file or memory permission problem i guess? else tell me how to solve it so that i can share my logs with u. Cheers.

Thanks for report. What do you mean "not so good" by MCI connection? At the end, was it able to connect but it was too slow (took longer than DSL)?

[SaSyda]

@free-the-internet I'm not under good coverage of mobile networks right now, so i can't say it for sure, but it seemed like yes, the connection was slow, or under pressure! hopefully i'll have better connection tonight and will test with both main carriers and come back with a more detailed report. In the meantime i hope there's a breakthrough on log extraction so that i can report in a more accurate way.

[free-the-internet]

@wkrp Other users are also succeeded to connect via new beta Orbot in Iran. I received 5 6 reports from various areas and networks. I'm very happy at the end the problem have been resolved.

However, it is hard to config for many people. there are 2 main difficulties:

• Many people should aware of Snowflake and know how to set it
• Many people reported to me that even after VPN mode, their apps could not connect. They have to select them from the list provided by "Tor-Enabled Apps" option. but they don't know it. So many of them, after installation in restricted areas could not use. Even after successful connection, they don't know what to do.

I suggest Orbot team could provide some tips at first run, and also after, in the main window by a big "?" button, so that people press it and read. Of course screen shots will help them. As I see the people who hasn't enough knowledge about android and apps, can't really benefit from Tor socks proxy, or Orbot.

@wkrp Do you know when the new version of Orbot would be real released in google play? Could you provide a brief traffic status? Is there any improvements since sudden drop in the traffic? Could you please provide information regarding the number of the users from Iran and their statistics?

Thanks in advance.

[wkrp]

Do you know when the new version of Orbot would be real released in google play?

No, sorry, I don't know that.

Could you provide a brief traffic status? Is there any improvements since sudden drop in the traffic? Could you please provide information regarding the number of the users from Iran and their statistics?

I don't see much of a change in bandwidth:

I had posted some homemade graphs at https://forum.torproject.net/t/graphs-of-user-counts-from-iran-since-the-onset-of-shutdowns/4843. Making special graphs was necessary because at the time, Tor Metrics did not correctly handle Snowflake statistics, because of the way we run multiple tor processes to distribute CPU load. Since then, Tor Metrics has gained the ability to process Snowflake descriptors, so you can easily get graphs from Tor Metrics. (The caveat about meek I stated at the forum still applies. I don't think the number of meek users was as high as the graph shows. The number of meek users is zero since Oct 04 because the bridge is being replaced.)

https://metrics.torproject.org/userstats-bridge-combined.html?start=2022-09-19&end=2022-10-23&country=ir

There is also a weird thing currently happening where, because the Snowflake bridge handles so many connections, other Tor relays may interpret it as a DoS attacker. I don't know exactly what the effect would be, or to what extent it's already happening.

• https://bugs.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/40223

  We discussed a few days ago how 12 super active snowflake back-end bridges on one IP address could trigger the "too many connections plus too many circuits from one IP address" anti-ddos defenses at Tor relays.

• https://bugs.torproject.org/tpo/core/tor/40636#note_2819763

  Snowflake appear to open numerous connections too. We'll have to be careful not to block it.

• http://meetbot.debian.net/tor-meeting/2022/tor-meeting.2022-10-13-15.58.log.html#l-44

  I was not previously aware that snowflake-01's high usership might be interpreted by other relays as a DoS signal

[Hadi-1624]

I'd like to test this in Tehran. Do i need to specifically use the bridge provided by @wkrp?

[mehdifirefox]

I'd like to test this in Tehran. Do i need to specifically use the bridge provided by @wkrp?

I tested. Work well. Is in the Beta version right now The new version will coming soon.

[Hadi-1624]

I just did a quick test with the beta, using the internal snowflake bridges on shatel ISP network, it connects, however, whatsapp calls won't work at all, downloads won't work at all, such as playstore downloads and updates. Whatsapp text and voice messages are sent and received very fast.

Is it not possible to use whatsapp calls on tor?

[Hadi-1624]

on MCI mobile network it almost never connects at all. if it connects, it is so slow it's completely unuseable, i only managed to receive 150kbit in 5 minutes.

[mehdifirefox]

on MCI mobile network it almost never connects at all. if it connects, it is so slow it's completely unuseable, i only managed to receive 150kbit in 5 minutes.

من دیدم وصل شد دیگه تست نکردم..چون نسخه بتا بود یکم واسه من گیر داشت من فک میکنم نسخه بعدی بهتر میشه . در کل الان بهترین روش v2ray هست .سرورهای رایگان هم کانالای تلگرام هستن

I saw the connection it was not tested anymore. I think the next version will get better. Overall is the best V2ray method right now.

[free-the-internet]

on MCI mobile network it almost never connects at all. if it connects, it is so slow it's completely unuseable, i only managed to receive 150kbit in 5 minutes.

I would gather some reports in the next days about this.

I just did a quick test with the beta, using the internal snowflake bridges on shatel ISP network, it connects, however, whatsapp calls won't work at all, downloads won't work at all, such as playstore downloads and updates. Whatsapp text and voice messages are sent and received very fast.

Is it not possible to use whatsapp calls on tor?

I don't think calls are going to work, the reason maybe are DDoS on tor network (see this), as well as very high latency that is not suitable for almost real time calling.

Besides, I remember you had timeout with your v2ray proxy; maybe it's because your domain is blocked. Check it with ping at least.

[wkrp]

Tor Browser 11.5.5, released today, was intended to enable uTLS by default for Snowflake. But a bug went undetected before release that actually stops Snowflake from working. If you are affected by the bug, you will see this message in the Tor log:

[WARN] SOCKS arguments can't be more than 510 bytes (535).

The Tor applications team is preparing a new release to fix the problem. If you have already upgraded to version 11.5.5, there is fortunately an easy workaround. You just have to enter a custom bridge line.

• In Tor Browser for desktop, go to ≡ (hamburger menu) → Settings → Connection → Bridges, then click the Add a Bridge Manually... button.
• In Tor Browser for Android, go to ⚙️ (settings) → Config Bridge. Toggle Use a Bridge to "on" and tap Provide a Bridge I know.

Then paste in the following bridge line:

snowflake 192.0.2.3:80 2B280B23E1107BB62ABFC40DDCC8824814F80A72 fingerprint=2B280B23E1107BB62ABFC40DDCC8824814F80A72 url=https://snowflake-broker.torproject.net.global.prod.fastly.net/ front=cdn.sstatic.net ice=stun:stun.l.google.com:19302,stun:stun.altar.com.pl:3478,stun:stun.antisip.com:3478,stun:stun.bluesip.net:3478,stun:stun.dus.net:3478,stun:stun.epygi.com:3478,stun:stun.sonetel.com:3478,stun:stun.sonetel.net:3478,stun:stun.stunprotocol.org:3478,stun:stun.uls.co.za:3478,stun:stun.voipgate.com:3478,stun:stun.voys.nl:3478 utls-imitate=hellorandomizedalpn

After you upgrade to the next release, you can remove the custom bridge line and go back to the built-in snowflake bridge.

The short description of what went wrong is that the encoding tor uses to pass parameters to a pluggable transport only has a limited capacity of 510 bytes. Recent additions to the parameter list (such as the new utls-imitate parameter) increased the length of the line past the limit. The bridge line above removes one STUN server from ice= parameter to shorten the length of the parameters list.

Unrelated good news: the 11.5.5 release has a new working meek-azure bridge to replace the old one which had been offline since 2022-10-04.

[wkrp]

Tor Browser 11.5.6 has been released. It fixes the problem with Snowflake from 11.5.5. If you manually entered a Snowflake bridge line as a workaround, you can revert back to the built-in Snowflake configuration. Version 11.5.6 has uTLS enabled by default for Snowflake.

[hexrot]

@wkrp Thanks a lot! I’ll continue with flakes, and unsubscribe. :-)

Am 27.10.2022 um 17:27 schrieb wkrp @.***>:

Tor Browser 11.5.6 https://blog.torproject.org/new-release-tor-browser-1156/ has been released. It fixes the problem with Snowflake from 11.5.5. If you manually entered a Snowflake bridge line as a workaround, you can revert back to the built-in Snowflake configuration. Version 11.5.6 has uTLS enabled by default for Snowflake.

— Reply to this email directly, view it on GitHub https://github.com/net4people/bbs/issues/131#issuecomment-1293705773, or unsubscribe https://github.com/notifications/unsubscribe-auth/A3WELUEKKWZDJWCI5LW4S2LWFKNP3ANCNFSM6AAAAAAQ7A6GC4. You are receiving this because you were mentioned.

[free-the-internet]

@wkrp Do you see any problems or unusual drops in snowflake usage? Especially traffic from Iran. Today I see decrease in snowflake traffic in my relays.

[mehdifirefox]

The speed is very low

On Wed, 16 Nov 2022, 15:31 free-the-internet, @.***> wrote:

@wkrp https://github.com/wkrp Do you see any problems or unusual drops in snowflake usage? Especially traffic from Iran. Today I see decrease in snowflake traffic in my relays.

— Reply to this email directly, view it on GitHub https://github.com/net4people/bbs/issues/131#issuecomment-1316894768, or unsubscribe https://github.com/notifications/unsubscribe-auth/AF26MY3HU5VAJOJKXPH4QRTWITELBANCNFSM6AAAAAAQ7A6GC4 . You are receiving this because you commented.Message ID: @.***>

[free-the-internet]

The speed is very low

Disconnect and connect again and test, maybe this time you can connect to a better proxy.

[wkrp]

Do you see any problems or unusual drops in snowflake usage? Especially traffic from Iran. Today I see decrease in snowflake traffic in my relays.

No, I do not see any change in the past few days. In fact, since November 1, traffic has recovered to about half the level from before the sudden blocking, possibly as a result of Orbot for Android .

The speed is very low

I encourage you to try the snowflake-02 bridge according to the instructions in https://github.com/net4people/bbs/issues/152. I don't know why, but it seems to work more smoothly for me. My best guess is that it has something to do with the DDoS mitigations that are currently deployed on Tor. Because the snowflake-01 bridge has so many users, other relays may interpret its connections as being part of the DDoS attack. I haven't done any tests to try and confirm this guess.

[free-the-internet]

@wkrp Thanks. Some questions: In snowflake proxy instances, do we have any option to advertise how many client we can serve? In general how broker decides how many users should be allocated to a proxy instance? Also, is there any configuration on the proxy instance so that "we" can forward to bridge-02, instead of clients? Unfortunately many people just once install Orbot and then use it as it is. So, I am really interested in helping them if I can do something on my side.

FYI, All in all I see 1 M bytes / sec transfer rate on my server that has a 1Gbps link speed. Certainly my proxy instance can do better.

[wkrp]

In snowflake proxy instances, do we have any option to advertise how many client we can serve? In general how broker decides how many users should be allocated to a proxy instance?

I think there is some logic in the broker that tries to distribute clients evenly over proxies, but I am not familiar with the details.

In general, there's no need to worry if your proxy has excess capacity. If your proxy has excess capacity, then other proxies have excess capacity too. It would not really speed things up to shift users from one to another.

Also, is there any configuration on the proxy instance so that "we" can forward to bridge-02, instead of clients?

No. The difficulty lies with how Tor clients work. The bridge line includes a relay fingerprint, and clients authenticate the bridge to ensure it is the relay they expected to connect to. So the choice of what bridge to use has to be done by the client.

There were some alternative designs considered; for example supporting more than one fingerprint per bridge line, or having the client not authenticate its first hop. How it works now, with the client telling the broker which bridge it wants to use, has its own disadvantages, but it emerged as the best option after discussion. You can read a history of the design discussion here: https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/28651

[hexrot]

My european flakes instances in four browser types: since many days the usual traffic.

Am 16.11.2022 um 22:20 schrieb wkrp @.***>:

In snowflake proxy instances, do we have any option to advertise how many client we can serve? In general how broker decides how many users should be allocated to a proxy instance?

I think there is some logic in the broker that tries to distribute clients evenly over proxies, but I am not familiar with the details.

In general, there's no need to worry if your proxy has excess capacity. If your proxy has excess capacity, then other proxies have excess capacity too. It would not really speed things up to shift users from one to another.

Also, is there any configuration on the proxy instance so that "we" can forward to bridge-02, instead of clients?

No. The difficulty lies with how Tor clients work. The bridge line includes a relay fingerprint, and clients authenticate the bridge to ensure it is the relay they expected to connect to. So the choice of what bridge to use has to be done by the client.

There were some alternative designs considered; for example supporting more than one fingerprint per bridge line, or having the client not authenticate its first hop. How it works now, with the client telling the broker which bridge it wants to use, has its own disadvantages, but it emerged as the best option after discussion. You can read a history of the design discussion here: https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/28651

— Reply to this email directly, view it on GitHub https://github.com/net4people/bbs/issues/131#issuecomment-1317681827, or unsubscribe https://github.com/notifications/unsubscribe-auth/A3WELUBRH2V5MCIVGHU7F3TWIVFYHANCNFSM6AAAAAAQ7A6GC4. You are receiving this because you were mentioned.