Properly configured Shadowsocks servers reportedly blocked in China?

gfw-report commented 3 years ago

Since March 2021, at least three users have reported that their Shadowsocks-libev servers became not pingable (temporarily in some cases) even after following the recommended setup in this tutorial (中文版). The tutorial documented how to setup a Shadowsocks-libev server that could defend against all known active probing attacks from the GFW as well as the partitioning oracle attack.

We document their reports in this post and we are taking immediate actions to investigate the causes.

At the same time, if you ever setup your servers by following our tutorial, we welcome you to report the status of your servers, no matter they are blocked or not. You can choose to:

post under this thread;
or post in the comment section(中文评论区);
or contact us privately (more contact information can be found at the footer of gfw.report).

Edited on May 5, 2021: add details from additional user report

The first report says that on the night of March 3, 2021, the server was not responding temporarily and since the morning of March 4, the server had not been pingable from at least two different network. The server remained not pingable for at least six hours until the user decided to discard the server. It appears that the user had setup the Shadowsocks server as recommended:

Used the latest version of Shadowsocks-libev at that time: shadowsocks-libev 3.3.5-1 751 latest/edge max-c-lv -;
Used an AEAD cipher: chacha20-ietf-poly1305;
Used a strong password generated by openssl rand -base64 18;
Used tcp_only mode;
Used ufw to only allow traffic to port 22 and the Shadowsocks port 24xxx;

The second report says that the server was not responding for half an hour on March 14, 2021, and has been working properly since then.

The third report says that the server was not pingable for approximately an hour on April 20, 2021, and have been working properly since then.

We were not able to find more reports on the recent blocking of Shadowsocks. Specifically, we looked up the open and closed Github issues in the following places:

gfw-report commented 3 years ago

自2021年3月起，至少三名用户汇报了他们的Shadowsocks服务器在使用了我们教程建议的配置后仍然出现不能ping通的情况。按照我们教程配置的Shadowsocks服务器本应可以抵御所有已知的来自GFW的主动探测以及（还未被GFW采用的）partitioning oracle攻击。

限于数据点有限，我们不能确定是审查者使用了新的检测方法，还是这只是个别用户的网络问题引发的误会。我们将迅速调查原因。与此同时，如果您曾按照这个教程配置Shadowsocks服务器，我们希望您能提供您的服务器的情况，包括被封锁了，或是运行正常的。您可以选择：

在这篇帖子下留言；
或是使用这个评论区;
再或是私下地联系我们（我们的联系方式在gfw.report的页脚）。

gfw-report commented 3 years ago

After calling for more users to report the status of their Shadowsocks servers from this post and from Twitter, one more user reported that "the Shadowsocks server configured according to the tutorial has been running stably for more than two months without any problem".

We have been encouraging users to report any blocking they encountered; however, it is also important to know their servers are working properly.

gfw-report commented 3 years ago

An up-to-date Outline server has been blocked in China. This server, located outside of China, has been running properly for more than two months, before it got blocked. The server possibly first got blocked around Jun 24 2021. This phenomena is interesting and possibly alarming because an up-to-date Outline server should have been able to defend against all known active probing attacks from the GFW.

We confirmed this blocking on Jun 27 2021 10 AM CST. We tested the blocking by 1) sending ICMP pings to the server (ping $ip) and 2) by making TCP connections to server ports with and without Outline listening to (nc $ip $port -v). We capture the traffic on server using tcpdump.

The result showed that only the Outline listening port was blocked, not the entire IP address. Particularly, ICMP pings from China could still receive echo reply; and SYN pings to non-Outline ports from China could still receive RST/ACK.

The blocking was single directional, meaning the censor only dropped packets from server to the client; SYN packets from China could still reach the server.

gfw-report commented 3 years ago

We encourage users to report the status of their Outline servers, including both blocked and unblocked status.

We were not able to find more reports on the recent blocking of Outline or other Shadowsocks implementations. Specifically, we looked up the open and closed Github issues in the following places:

Outline:

Other Shadowsocks implementations:

The only blocking report was this one, where the server using shadowsocks-rust + xray-plugs got blocked on June 8 2021.

Have you received any blocking report recently, @fortuna ?

gfw-report commented 3 years ago

July 1 will be the 100th anniversary of the Chinese Communist Party. Previous work shows that the censorship will usually be strengthened and advanced around these political sensitive period of time.

While we will continue monitoring the censorship on Outline and other circumvention tools, we also encourage more users, developers and researchers to pay attention to the censorship in China this week.

moranno commented 3 years ago

I'm using shadowsocks with chacha20-ietf-poly1305 by https://github.com/XTLS/Xray-core. Only the listening port was blocked 40 mins ago.

fortuna commented 3 years ago

@gfw-report I haven't heard of private servers being blocked. I know of a few providers that did not observe any issues recently.

nthlink is a notable exception. They have always seen blocked servers, but their system is designed to deal with that via aggressive server rotation. However, as a centralized service, they are vulnerable to enumeration attacks, so it may not be Shadowsocks detection.

This is the traffic from China as reported by the Outline servers that opted to share metrics: Lines are daily averages for 1, 7 and 28-day windows.

There aren't clear drops, except maybe a bit around April 19 and June 6, 2021. Keep in mind that the metrics are not from a random sample, so they may not be representative. The numbers are biased in an unknown way.

Finally, I would like to point out that the choice of client does make a difference. The Outline client as well as the current shadowsocks-libev merge the socks header and initial user data in the same initial packet, which prevents fingerprinting based on packet size. Clients using an old version of ss-libev or other implementations may not be doing that and can expose the server.

fortuna commented 3 years ago

Another important thing to point out: if you are using plugins, that's not Shadowsocks anymore. I don't know how fingerprintable the different plugins are, so keep in mind that they could make you more vulnerable.

wkrp commented 3 years ago

I received a report about a shadowsocks-rust server being blocked in early June. The symptoms matched what was reported in https://github.com/net4people/bbs/issues/69#issuecomment-869099906: unidirectional, with only server-to-client traffic being blocked.

At https://github.com/shadowsocks/shadowsocks-rust/pull/556#issuecomment-864432692 there is the hypothesis that the replay filter itself is a feature that the GFW uses to detect Shadowsocks servers. The Shadowsocks server ignores client handshakes that reuse a previously observed salt. The replay cache is periodically reset. So an attack might look like this:

Observe and capture a legitimate, correctly authenticated handshake.
Replay the capture immediately; it should be ignored by the Shadowsocks server because it reuses a salt.
Wait for the replay cache to be emptied (you can do this by counting connections to the server).
Replay the capture again; this time it should be accepted and the Shadowsocks server should send back a non-empty response.

There's currently an experiment running to test the effect of the replay filter on detectability. shadowsocks-rust with the replay filter enabled was repeatedly blocked (different ports on the same host) in the past week. Disabling the replay filter stops the server from being blocked. This suggests that the GFW is using the behavior of the replay filter in response to active probes (in shadowsocks-rust, at least, and maybe in other implementations) as a classification feature. It may be possible to frustrate this new kind of active probing by changing the way the server remembers and reacts to replays.

fortuna commented 3 years ago

@wkrp Interesting finding. It would be great if someone else was able to reproduce the experiment, and include the Outline Server in the test, since it behaves a bit differently.

My understanding is that the biggest issue for fingerprinting is modified replays, where you get different behavior when you modify certain specific bytes. Replay protection is a way to protect against that, but there are other ways. For instance, the coalescing of IV, SOCKS header and initial data means that if you modify any byte in the first packet you will still get the same result (timeout). You don't need the replay filter in that case.

However, you will get different results if you modify the second packet (really the second encrypted chunk, which usually matches the packet). For example:

Full replay: get data back, closed connection
Replay with first packet modified: timeout
Replay with second packet modified: some data back, then timeout (maybe closed connection).

Is that a reasonable behavior? Can the attacker use that information somehow? If it's not useful to the attacker, then we don't need the filter.

One important thing to keep in mind is that, if you don't have the replay filter, a single client that doesn't coalesce the initial data is enough to expose the server. The attacker can leverage the connections from that single client to fingerprint the server and be confident that the server is Shadowsocks.

Edit: Ignore the last part. Modifying any byte in the IV, SOCKS or initial data will still lead to a timeout these days.

gfw-report commented 3 years ago

Thank you for reporting on the blocking, @maker2002.

While we have no evidence that the blocking is caused by the active probing attack, our initial testing indeed suggests that the Xray implementation of Shadowsocks has active probing weaknesses. These active probing attacks have been previously proposed by Frolov et al. and found being used by the GFW (Alice et al.).

We have opened an issue here: https://github.com/XTLS/Xray-core/issues/625

It is worth noting that Outline servers have fixed this weakness for a long time and we tested and confirmed that such weakness did not exist in the latest version of Outline server. In other words, this weakness cannot be the causes of the blocking of this Outline server: https://github.com/net4people/bbs/issues/69#issuecomment-869099906

gfw-report commented 3 years ago

Thank you for your summary and insights, @wkrp.

We left our observations and thoughts here: https://github.com/shadowsocks/shadowsocks-rust/pull/556#issuecomment-870285895

https://github.com/net4people/bbs/issues/69#issuecomment-870034503

It would be great if someone else was able to reproduce the experiment, and include the Outline Server in the test, since it behaves a bit differently.

We will design and conduct experiments to validate/falsify @Mygod 's hypothesis. We will include Outline.

My understanding is that the biggest issue for fingerprinting is modified replays, where you get different behavior when you modify certain specific bytes.

That's a key insight.

moranno commented 3 years ago

There is another report about shadowsocks-rust with xray-plugin got blocked here: https://github.com/shadowsocks/shadowsocks-rust/issues/549#issue-914457016

And raw shadowsocks blocked in 3 days: https://github.com/shadowsocks/shadowsocks-rust/issues/549#issuecomment-869287337

I have used shadowsocks-rust + v2ray-plugin for almost 1 month. It still goes smoothly. While without any plugins, I got port blocked within 3 days.

xiaokangwang commented 3 years ago

I believe the best thing to do now is to reproduce the replay filter toggle experiment with a tcpdump packet capture. This can provide us with detailed intelligence about the behaviour of censorship tools. Currently, the user report based intelligence gathering is not very useful other than providing clues.

If the replay filter is proven to be the vulnerability exploited by the the GFW. The method to resolve it isn't the removal of this security feature, but to return random data of a calculated length to the attacker's client instead of shutting down the connection or returning no data when a replay is detected. An attacker without the required secret key cannot tell the difference between random data or valid server responses(when done carefully). Thus, returning random data achieve the same goal as turning off the replay filter without security complications. We should always try to reduce the compromise on the security aspect of proxy implementation.

(This post express the personal opinions of the author, instead of the affiliated V2Ray/V2Fly organization.)

fortuna commented 3 years ago

From https://github.com/net4people/bbs/issues/69#issuecomment-870478271:

If the replay filter is proven to be the vulnerability exploited by the the GFW. The method to resolve it isn't the removal of this security feature, but to return random data of a calculated length to the attacker's client instead of shutting down the connection or returning no data when a replay is detected.

@xiaokangwang This may open up new fingerprinting possibilities: send multiple repeated packets, look at the response size entropy. If random, then it's likely Shadowsocks, especially if varies widely. The average packet size may also give a clue. This reminds me of how ShadowsocksR gets detected (random packet sizes).

xiaokangwang commented 3 years ago

From #69 (comment):

If the replay filter is proven to be the vulnerability exploited by the the GFW. The method to resolve it isn't the removal of this security feature, but to return random data of a calculated length to the attacker's client instead of shutting down the connection or returning no data when a replay is detected.

@xiaokangwang This may open up new fingerprinting possibilities: send multiple repeated packets, look at the response size entropy. If random, then it's likely Shadowsocks, especially if varies widely. The average packet size may also give a clue. This reminds me of how ShadowsocksR gets detected (random packet sizes).

Thanks @fortuna . That's correct, this is an issue I initially overlooked.

There are two ways to solve this: The first one is to remember the original initial response length and replay random data of that length on replay. The second one is to maintain a database of destinations and their distribution of the lengths of the first reply. Both methods have performance issues and aren't perfect.

Maybe the best way to resolve these kinds of issues in the long term will be to design a new protocol(or protocols) and migrate these kinds of issues at the protocol level.

Designing a new protocol that incorporates server-side random into sessions can eliminate the need for replay checking. However, it still needs to consider how should it respond to an invalid message from a remote peer.

A little out of topic thought, the root issue here is: how should a censorship resistance tool react when it encounters an invalid message. There are a lot of different behaviours that leaks information: It can read indefinitely, read a certain length, or do not read any additional data. It can write data of a certain length to the attacker, or not. It can close the connection immediately, or wait a time out, or wait indefinitely. These behaviours should vary depending on the status of the connection and the nature of the error. Is this the first message received on the connection, or a subsequent one? Is this message invalid because of an invalid authentication code, or a replay of a previous session? A more systemic analysis could be useful in solving this issue in long run.

fortuna commented 3 years ago

From https://github.com/net4people/bbs/issues/69#issuecomment-870300101:

It would be great if someone else was able to reproduce the experiment, and include the Outline Server in the test, since it behaves a bit differently.

We will design and conduct experiments to validate/falsify @Mygod 's hypothesis. We will include Outline.

Thanks @gfw-report!

If you use the stock Outline Server using the Outline Manager or the install script, you will get replay protection enabled.

If you need to disable it, you will have to use outline-ss-server directly. The replay protection is disabled by default in outline-ss-server. You can increase the number of recorded salts with --replay_history.

Note that the protection against replays of server data is always on, regardless of the --replay_history flag.

gfw-report commented 3 years ago

An up-to-date Shadowsocks-libev server has been blocked in China. This server, located outside of China, has been running properly since Jun 24 2021, before it got blocked on July 20. The server was configured strictly following this tutorial.

We confirmed this blocking by 1) sending ICMP pings to the server (ping $ip) and 2) by making TCP connections to server ports with and without Shadowsocks-libev listening to (nc $ip $port -v). We capture the traffic on server using tcpdump.

The result showed that only the Shadowsocks-libev listening port was blocked, not the entire IP address. Particularly, ICMP pings from China could still receive echo reply.

The blocking was single directional, meaning the censor only dropped packets from server to the client; SYN packets from China could still reach the server.

//cc @madeye @Mygod @zonyitoo @riobard @database64128 @RPRX

fortuna commented 3 years ago

@gfw-report Did the server have replay protection enabled or disabled?

How is the experiment going?

gfw-report commented 3 years ago

Did the server have replay protection enabled or disabled?

Yes, the Shadowsocks-libev server has replay protection enabled.

How is the experiment going?

Starting from July 12, we ran two outline servers on two different IP addresses within the same subnet outside of China. The server were configured following your instructions. The only difference is that one server has replay protection enable, while the other has replay protection disabled.

We then make two connections to each server every 15 seconds from China. The underlying proxies traffic is HTTP and TLS. Neither of the servers has been blocked yet.

Mygod commented 3 years ago

You might need to crank up the number of connections. Three to four days at most should be enough to trigger from my experience.

fortuna commented 3 years ago

Keep in mind that the Outline server and shadowsocks-libev have slightly different behaviors. I wonder if that is making a difference in detection.

@gfw-report: Was the blocked shadowsocks-libev server receiving the same pattern of traffic?

MatthK commented 3 years ago

My wife just moved back to the Mainland temporarily and we couldn't communicate using Signal anymore. I first tried to install the TLS-Proxy but that didn't work. A OpenVPN connection also didn't help. Then I came across this thread and gave it a try to install the Shadowsocks server using the tutorial linked above (https://gfw.report/blog/ss_tutorial/en/).

Installation was smooth and quite flawless. Using the OpenVPN connection she managed to download the Outline client and I could send her the ss-link via WhatsApp (I specifically avoided WeChat). She could connect to my Shadowsocks server and now we can use the Signal messenger again. Even a 3-way video call worked pretty well, even though it says it's not perfect for streaming.

The client also works on her PC now and she can access her Google mails and other company stuff she needs.

Since the OpenVPN is flaky but Shadowsocks seems to work just fine. Since she's the only one to use my server, I would hope it'll fly under the radar and won't get blocked, but yeah. Let's stay optimistic.

Just for information. The server is in Hong Kong and runs on a Raspi 4 with Ubuntu 20.04.

Thank you guys for that clear tutorial and all this work. Highly appreciated to stay connected.

MatthK commented 3 years ago

It seems my optimism was a bit too early. The VPN now seems to be no longer working. When doing a DNS Leak Test, my wife gets various IP addresses, instead of the single one from my Unbound server on my network. Is it possible that the key(s) somehow got leaked through insecure transmission (I sent her the ss:// link via WhatsApp, and avoid to send anything confidential via WeChat). Or how else could there be a DNS leak?

Would it help to just change the password? Or password and URL? Any other tips?

cjhenck commented 3 years ago

Hi @MatthK - I’m the Product Manager for Outline. Can you confirm the DNS Leak Test problem was occurring with the Outline Client? If so, can you give us more details such as on which platforms, with which leak test, and whether the problem persists if she restarts the client?

I’m trying to understand whether the client is crashing on her devices and that’s the source of the test discrepancy.

xiaokangwang commented 3 years ago

Hi, @MatthK

When doing a DNS Leak Test, my wife gets various IP addresses, instead of the single one from my Unbound server on my network.

DNS Leak Test detects if there is a misconfiguration of the DNS setting, not the proxy itself. It is used to detect whether your DNS traffic(which includes your abridged browsing history) is sent to an unintended server. You may update your system DNS, and browser proxy settings to prevent DNS leaks. It does not show who can your proxy or who have used it. It does not show whether it has been blocked or sabotaged.

If you suspect the connection credential is leaked, you can update the password to prevent other people from connecting it with the old password. This does not prevent censorship agency from blocking your proxy with the connection credential since blocking it would only require the IP address(and optionally the port number) for an ss proxy. Your connection credential may leak during transit or after it arrived on your device. On some platform, software installed on the system or the operating system itself will scan your files and upload them to the vendor's server.

Why do you think it is no longer working? Is it because of a DNS leak, or you can no longer connect to it?

P.S.: I don't think this is supposed to be a user support ticket system. (Since we are already here you can submit more feedback to help us find the true issue. I have no intention to suppress discussion.)

MatthK commented 3 years ago

"I don't think this is supposed to be a user support ticket system."

OK, point taken.

cjhenck commented 3 years ago

P.S.: I don't think this is supposed to be a user support ticket system.

Likewise acknowledged. Apologies. For Outline questions feel free to use our tracker or Reddit r/outlinevpn.

On topic - I’m still curious about if the test results from @MatthK mean that the proxy was not working and thus we cannot rely on the previous data point that it hasn’t been blocked. (Sorry for the triple negatives.) Is there other evidence that it is working? (Visiting blocked sites, Signal working, etc.)

database64128 commented 3 years ago

@cjhenck It's clearly working. Outline client doesn't have a routing system like many other popular censorship circumvention solutions in China. When it's connected, everything gets proxied through the remote shadowsocks server, so you either have working Internet, or nothing works.

wkrp commented 3 years ago

I don't have a problem with helping to diagnose user issues, as long as it's relevant to the topic. Anything specific to Outline configuration is probably better handled in an Outline forum. But like @cjhenck, I am curious the two facts that @MatthK posted, which may or may not have the same cause:

The Outline connection stopped working.
A DNS leak test showed DNS queries originating from multiple IP addresses, not the IP address of the DNS caching resolver which the Outline server is (presumably) configured to use.

If the Outline connection stopped working, there could be many causes. There could be a problem at the server, or at the client, or there could be some kind of interference in the middle (such as the server being blocked by a firewall). What specifically is going wrong is something you can troubleshoot with the Outline devs. But it would be consistent with other observations that the GFW has somehow identified the server as an Outline server and blocked its IP address and port number.

An easier way to check if requests are going through the Outline proxy (easier than doing a DNS leak test) is visit an IP-checker web site (e.g.), or just Google "my IP address", and see if it matches the IP address of the Outline server. If it does not match, but the client is able to browse various sites, then there's a good chance there's some kind of misconfiguration or the Outline client just isn't running. As @database64128 says, it's unlikely that Outline would leave you with a "half-working" setup.

I don't know how the DNS leak test works, but it's possible that even if the Outline server is configured to use a designated resolver, the test reports the IP addresses of upstream recursive resolvers. I wouldn't take this test as conclusive, unless you also have results from when the VPN was working that show a difference. But the mention of DNS leaks does bring to mind a hypothesis for how Shadowsocks servers might be discovered:

Hypothesis: The GFW takes advantage of misconfigured clients with leaky DNS to help discover circumvention servers. It identifies source IP addresses that (1) make a DNS query for a name that would normally be blocked, then (2) begin a TCP session with an unclassifiable protocol. When this happens often enough, it blocks the destination IP address / port of the TCP session.

However, this hypothesis has a major flaw, which is that a Shadowsocks client with leaky DNS is likely not to work well, because of the GFW's DNS poisoning. The client would query a DNS name, get a poisoned (useless) IP address in response, give that IP address to the Shadowsocks server through the tunnel, and the Shadowsocks server would simply fail to connect. A configuration that leaks DNS would probably be noticed by the user quickly, at least in China.

Tor emits a warning ("Your application is giving Tor only an IP address. Applications that do DNS resolves themselves may leak information.") when it gets a SOCKS request without a DNS name to resolve. A Shadowsocks client could similarly warn when it gets ATYP=1 or ATYP=4.

database64128 commented 3 years ago

However, this hypothesis has a major flaw, which is that a Shadowsocks client with leaky DNS is likely not to work well, because of the GFW's DNS poisoning.

Such leaks are actually pretty common with misconfigured V2Ray setups. Most users configure V2Ray with both domain rules and IP rules. A typical setup is to first check if the domain is in the Chinese domain list. If not, resolve the domain using the configured DNS servers and then check if the resolved IP is a Chinese IP. If not, then V2Ray makes the request using the original hostname (not the resolved IP address) via the proxy.

This setup would've been fine if the routing rules ensure that traffic to these configured DNS servers goes through the proxy. However, many users add Chinese DNS servers in hopes that it would prevent unlisted Chinese sites from being proxied, which is a totally unnecessary but very dangerous action. These DNS servers are in the Chinese IP list, and therefore DNS queries are made with direct connection. And because the requests are made with the original target address preserved by default (not using resolved IP), when connecting to an endpoint with a "poisoned" DNS response, as long as the response is a non-Chinese IP, it would appear to be working just fine, because by default this invalid IP address is only used in V2Ray's internal router.

To avoid DNS leaks, users should be advised against configuring V2Ray with Chinese DNS servers. Examples with Chinese DNS servers in the official documentation should be removed or annotated with warnings. V2Ray does support placing some constraints to prevent certain DNS servers from being used to resolve certain domains. But it can be quite tricky to make it right.

This is even worse with clients like Clash, where DNS queries don't go through its internal router, and always use direct connection. Clash is extremely popular among users of commercial censorship circumvention services.

xiaokangwang commented 3 years ago

@MatthK Are you intended to use that Unbound server as your DNS server instead of the one provided by ISP or VPS provider? Can you still use the proxy to browse unrestricted Internet?

It is possible that @MatthK intended to use a custom DNS server, and this DNS setting is overridden by the proxy software or socks5 protocol remote domain name resolution. I need more information to know whether the DNS server used is provided by a Chinese ISP or the VPS provider. If it is from the VPS provider, then @MatthK didn't configure his computer to use that Unbound DNS server, and the proxy is working as intended. If the DNS server is provided by a Chinese ISP, then a DNS leak happened.

The proxy connection for web traffic is likely to be working as intended. If the ss server is blocked, the DNS leak test will not be able to complete. The DNS leak test requires a working connection, with or without a DNS leak.

gfw-report commented 3 years ago

Strangely, the two Outline servers mentioned in this comment have still not been blocked yet as of August 6, 2021.

cjhenck commented 3 years ago

I have only been lightly following this thread and missed that @MatthK was trying to use alternative DNS servers. Outline currently hard codes the DNS servers in the client. We've had requests to allow clients to override it but my personal feeling is that Outline's client strength is simplicity rather than advanced functionality (where shadowsocks-android and others shine). However, I do want to eventually remove the dependency on third-party DNS and rely on a DNS forwarder on the VPS so the server operator can specify the recursive, but we don't have the bandwidth at the moment to implement that.

fortuna commented 3 years ago

@MatthK, thanks for the report, but we need more information.

The VPN now seems to be no longer working.

What makes you conclude that?

When doing a DNS Leak Test, my wife gets various IP addresses, instead of the single one from my Unbound server on my network.

Getting multiple IPs doesn't mean a DNS leak. Those tests are misleading, they expect the IPs to match your ISP. With Outline, they won't match. You should expect to see a backend IP from OpenDNS, Cloudflare or Quad9. This IP may not even match the IP or the resolver you talk to because the resolution may be delegated.

My best guess is that the client was not working. The Windows and Linux clients have some issues handling network changes, which is hard since they don't provide a VPN API we can use. If that's the case, disconnecting and reconnecting should work. You may need to close the app first.

MatthK commented 3 years ago

OK, here's a bit of background.

I have currently two shadowsocks server running at my home in Hong Kong. Each runs on a Raspberry Pi with Ubuntu 20.04 LTS. They each use different passwords (openssl rand - base64 24). Each has a different domain using a ddns service, each a different port in the 30000s.

I also have a Pihole installed which uses a local unbound server as the forwarding resolver. The Router is blocking any DNS requests via NAT rules (Port 53 and 853) and points them to my unbound config. If you're on my network, you pretty much get DNS from my little Raspberry, unless you use DNS over HTTPS. I prefer to keep as little a digital trace as possible, so that's why I prefer to do the DNS resolving myself.

When I try to configure various clients at my home to use 8.8.8.8 as the DNS Server and then do the DNS leak test, I always only see my own public IP as the result. Single address, never multiples. So I assume, my setup does the intended job.

The initial setup (of the Shadowsock server) was done, cos the pre-installed OpenVPN wouldn't let her connect to Signal, which is our families messenger of choice. So when I say "it's (not) working", then I usually mean that connecting to read and write messages in Signal is working (or not). My wife also managed to install it on her notebook, which allowed her to connect to her GMail account. Besides that, we don't use the VPN for much else. After the initial setup, using Signal was again possible, and we even managed to get a decent 3-way video call to work. Now a few days ago, she complained and said she can't use Signal anymore. While she can get a connection to the SS server, Signal remains mute. I asked her then, to do the DNS leak test, which showed a bunch of mainland China IP addresses/domains. Here's the partial list she got:

14.152.40.32   None   China Telecom   Guangzhou, China
219.128.128.102 102.128.128.219.broad.st.gd.dynamic.163data.com.cn   China Telecom, China
47.106.96.189   None   Hangzhou Alibaba Advertising Co., Ltd.   Shenzhen, China

I asked her to verify that the DNS server in her Outline profile points to my pihole (192.168.7.3) but had to realize, that there's no way to configure that in the iPhone Outline app. On my Android Shadowsocks client, I can configure this.

Yesterday she could connect again when I sent her a link to my second server. This morning, it doesn't work again. She can connect to the server, even received messages in Signal, but can't send any. I asked her to restart the app, and even restart her phone.

Now here comes the strange part. Although my wife was about to give up, I asked her to do the IP check and DNS leak tests again. A couple of tries with connecting to both servers, and suddenly it works again. All DNS leak tests show my home address only and Signal works, both for receiving and sending messages. I feel quite embarrassed, because it looks more like it was a user fault rather than a systematic problem. Or it was some random fluke (unlikely, I know). And unfortunately I can't try it myself, as I'm not in the mainland. I did experience though previously (more than two years ago when I still was myself in the mainland) some randomness with my OpenVPN connections to my home. Sometimes it did work, while a half an hour later it suddenly wouldn't anymore, to then work again a day or so later.

PS: This is not meant to get support. I'm just trying to point to some potential issues. It could be, that I have screwed up some things, but I'm just trying to point out (potential) problems that might affect others that really need to rely on this solution.

PPS: I obviously didn't share the SS-Links through Wechat, as I perceive this channel as compromised. I did send her the links only through WhatsApp (less preferred channel) or then directly in Signal (once it was working). But talking about VPN and Signal in Wechat might have triggered some flags and could have put her phone under special watch?

database64128 commented 3 years ago

@MatthK The Outline Client UI on iOS can be glitchy sometimes. It might display "connected", but the settings app says "not connected". Make sure to double check with the system VPN settings. For iOS, I would recommend eycorsican/leaf. It's a lot more reliable than Outline Client on iOS devices. The App Store link is in the project's README.

MatthK commented 3 years ago

Ok, that would at least explain to some extent the seen behaviour. If Outline claims it has a connection, when in fact the system is not, then the DNS leak test would obviously show the above results.

RebelliousWhiz commented 3 years ago

The blocking was single directional, meaning the censor only dropped packets from server to the client; SYN packets from China could still reach the server.

Server: shadowsocks-rust v1.11.2 Client: Outline

I think I have encountered a similar situation. I run shadowsocks service for many friends (in China). Every once in a while (log starting last year), one of my friends would report to me that their service was not usable.

However, other services using other ports like other users' shadowsocks, SSH 22, HTTP 80 443, ICMP are fine. Then I asked the cloud service provider, they told me everything was fine. This problem can be solved by changing IP, so I didn't care much.

Last night I decided to figure it out. A friend told me his port was not accessible again. So I moved the IP to another NIC, removed the shadowsocks on this port, opened SSH port, then I SSH to this specific port.

The first few SSH tries were not working. I was angry typing the ticket to the cloud provider. Then I tried one access with IP:port instead of DOMAIN:port, the port was accessible again.

This didn't make sense to me last night... But since you brought it up here, I think it might be the same case.

I will continue to test and try to observe this problem again.

Could anyone tell me how to tell if it's a single directional problem? I will try to catch some log next time I find it.

xiaokangwang commented 3 years ago

single directional

@RebelliousWhiz A single directional block cannot be easily diagnosed with a software log. Packet capturing is the most reliable method for detecting this kind of restriction. To create a packet dump, run tcpdump -w $(filename) , reproduce the issue, and send this file to an expert for analysis.

Speaking of a single directional block, is it possible for us to create a tool that workaround this with source address forging?

wkrp commented 3 years ago

Speaking of a single directional block, is it possible for us to create a tool that workaround this with source address forging?

ReQrypt (previous discussion here) is kind of like that, but in the wrong direction for the GFW. It effectively spoofs in the client→server direction, and allows packets to flow naturally in the server→client direction.

It will work against IP blocking when the firewall blocks SYNs to forbidden IPs, but not when it blocks SYN/ACKs from forbidden IPs (as the GFW does, last time I checked).

gfw-report commented 2 years ago

On November 6, 2021, a user reported that a Shadowsocks-libev server (3.3.5-1, edge channel) configured according to this tutorial got blocked after a week of use. In particular, the config file is as follows:

{
    "server":["::0","0.0.0.0"],
    "server_port":[REDACTED],
    "encryption_method":"chacha20-ietf-poly1305",
    "password":"[REDACTED]",
    "mode":"tcp_and_udp",
    "fast_open":false
}

Since neither the server nor the service got restarted during the week-long usage, it is less likely that the replay filter got reset.

The blocking is done by dropping packets as the user reported:

Inside China:

nc: connect to [REDACTED] port [REDACTED] (tcp) failed: Operation timed out

Outside China:

Connection to [REDACTED] [REDACTED] port [tcp/*] succeeded!

The user also reported seeing "a lot of TCP Retransmission" on the server side, indicating that it is likely that the SYN+ACK packets from server to client got dropped.

gfw-report commented 2 years ago

Since the GFW has been reportedly doing port blocking, rather than IP blocking, the following small trick may mitigate the incovenience when port blocking happens.

A user can run the following command on server to redirect both TCP and UDP traffic ranging from port 12000 to 12010, to port 8388:

sudo iptables -t nat -A PREROUTING -p tcp --dport 12000:12010 -j REDIRECT --to-port 8388
sudo iptables -t nat -A PREROUTING -p udp --dport 12000:12010 -j REDIRECT --to-port 8388

Remember to:

replace 12000:12010 with a different range that only you know (we suggest any port from 1024 to 65535);
replace 8388 with the Shadowsocks port you used.

This way, whenever a port on server got blocked, a user only needs to change the configurations of server_port on the clients, without having to login to the server, change configuration, and then restart Shadowsocks. For example, if you had been using port 12000 but it got blocked, all you need to do is changing the server_port from 12000 to 12001 on your client.

When properly configured, the output should look like this:

sudo iptables -t nat -L PREROUTING -nv --line-number

Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 REDIRECT   tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpts:12000:12010 redir ports 8388
2        0     0 REDIRECT   udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp dpts:12000:12010 redir ports 8388

Note that setting a PREROUTING rule on ephermeral ports (/proc/sys/net/ipv4/ip_local_port_range) will not disrupt normal outgoing connections that use those ephermeral ports as source ports.

fortuna commented 2 years ago

With outline-ss-server, you can set up the same credential on multiple ports, without having to deal with iptables.

The config format is quite straightforward (example).

gfw-report commented 2 years ago

Between August 19 and 22, 2022, we received twelve user reports on the blocking of their properly configured Shadowsocks and Outline servers. Before getting blocking, their servers had been running properly for a relatively long time, ranging from 25 days to 13 months. Two Chinese discussion threads are available at here and here.

From the user reports and our testing on two blocked servers that users granted us access to, this new blocking is different from the dynamic blocking of fully encrypted traffic since November 2021 in many ways. Below is a table for comparison:

	dynamic blocking	this new static blocking
effective date	from November 2021 to date	between August 19 and 22, 2022
blocking IP or port	only used ports of the servers	entire server IP addresses
disruption method	dropping client-to-server packets	dropping server-to-client packets
blocked protocols	only TCP (not UDP or ICMP)	all protocols
blocking duration	usually 120s or 180s	days, if not weeks
triggering conditions	sending random traffic	unknown
triggering server IPs	some IP-prefixes of popular VPS providers	hosts under the same /30 of the blocked hosts are not blocked dynamically or statically.

Some additional details of this new blocking incident include:

Two users reported that, before their servers got blocked completely, there were a few hours when they could access the servers "via cellular, but not WiFi".
At least two servers were blocked sometime between 3:00 AM and 5:00 AM Beijing Time on August 19, 2022.
At least two servers have been unblocked by the afternoon on August 24, 2022.
In some cases, users didn't share the circumvention servers with anyone else.
In some cases, the 30-day usage of the blocked servers is less than 50 GB.

Can you corroborate this new blocking incident from the traffic metrics of Outline? @fortuna

RebelliousWhiz commented 2 years ago

I can also confirm this block. But I noticed the problem starting before Pelosi's visit to Taiwan.

During that time, I suspected a few of my users were sending sensitive content through the internet, and I tried to isolate a few aggressive users. After a few attempts, IP still got blocked after a few days or so, and it has been consistently behaving like that.

Changing IP can only solve the problem temporarily. Now I have to rely on my users' feedback to update the IPs.

I have a few servers that can be used to identify the problem. If any of the developers need resources to test the find out the problem, please contact me.

RebelliousWhiz commented 2 years ago

I have been providing stable services to my friends and family for quite a long time (stable for almost two years). I am pretty sure there's nothing wrong with the server configuration, IP latency, or package drop issue.

I can also confirm there's no interference from Chinese malware like 360 virus scanner or Tencent PC protector. I noticed those software would restrict Shadowsocks traffic a long time ago, so every user of mine will be instructed to remove those software upon installation.

ilhaformosa commented 2 years ago

Earlier this week, two of my servers' IPs were blocked in mainland China by GFW. Both servers were deployed using Outline Server, and has been stably operated for over a year and 6 months respectively. It is worth mentioning that these two servers use CN2 GIA route for connections between US and China.

A newly assigned IP were blocked within a hour after another Outline Server deployment, which is never seen before.

huanbaiwan commented 2 years ago

Shadowsocks-libev 还能用不？？

RebelliousWhiz commented 2 years ago

Shadowsocks-libev 还能用不？？

I suggest remaining calm as we do not know the exact reason for the block. Further observation is required. If changing IP can solve the problem temporarily, please try to take the temporary solution as needed.

No need to panic as IP blocks can be random or unreasonable.

In the meantime, I also suggest taking necessary security actions like hardening your Linux defense (like public key authentication, complicated password, self-deployed cloud images instead of cloud-provided images), using more secured shadowsocks encryption (like AES-256-GCM), or even applying an extra layer of VPN on top of your shadowsocks server to hide your fingerprint (like ExpressVPN or Surfshark).

net4people / bbs

Properly configured Shadowsocks servers reportedly blocked in China? #69