net4people / bbs

Forum for discussing Internet censorship circumvention
3.19k stars 75 forks source link

RST injection of 1.1.1.1:443 in China, 2023-09-05 to 2023-09-20 #285

Open wkrp opened 9 months ago

wkrp commented 9 months ago

This thread is to consolidate reports from other threads, particularly https://github.com/net4people/bbs/issues/280#issuecomment-1706267069 and #284, as well as information received privately.

In summary: since 2023-09-05 (about 06:00 UTC), there is new RST injection of https://1.1.1.1/ (which, notably, is a DNS over HTTPS resolver. 1.1.1.1 may have been blocked in other ways in some regions before; but there was definitely a change starting 2023-09-05. The RST injection resembles some of the "classic" RST injection that has been documented in China in the past, but there are some differences. It also differs from the well-known SNI filtering, in that the injection is triggered by the client's TCP SYN packet, not the TLS Client Hello.

Blocking of 8.8.8.8:443 (Google's DNS over HTTP resolver) is unrelated. It is blocked by packet dropping / null routing, not RST injection (ref).

wkrp commented 9 months ago

I will make a separate post with specific details of the nature of the RST injection.

The fact of the GFW terminating TCP connections by injecting 3 RST packets has been documented repeatedly. However, this recent injection regarding 1.1.1.1:443 is different from what I remember having been reported before. (Please correct me if I am wrong.) The main difference is that, instead of the 3 injected packets all having different sequence numbers, separated by certain intervals, in this case all 3 have the same sequence number. Also, I believe that both the client's SYN and the server's SYN+ACK triggers 3 RSTs, for 6 injected packets in total.

"Ignoring the Great Firewall of China" (2006) described 3 RST packets being sent in both directions (3 to the client, 3 to the server), with sequence numbers base, base+1460, base+4380.

The first three reset packets had sequence values that corresponded to the sequence number at the start of the GET packet, that value plus 1460 and that value plus 4380 (3 × 1460). We believe that the firewall sends three different values to try and ensure that the reset is accepted by the sender, even if the sender has already received ACKs for “full-size” (1460 byte) packets from the destination. Setting the sequence value of the reset packet “correctly” is necessary because many implementations of TCP/IP now apply strict checks that the value is within the expected “window”.

We also examined this blocked connection from the point of view of the Chinese webserver… As can be seen, when the “bad” packet was detected, the firewall also sent resets to the Chinese machine

"Your State is Not Mine" (2017) reports something similar, which they call a "type-2 reset". The difference is that now the RST+ACK flags are set, not just RST. They also describe a "type-1 reset", which is sent 1 at a time, not 3, and does not have the ACK flag set. The spacing between sequence numbers is the same base, base+1460, base+4380 as in 2006. The "type-2 resets" are also noted to have IP TTLs and TCP window sizes that are incremental (base, base+1, base+2) within a salvo of 3.

Once any sensitive content is detected, the GFW injects RST (type-1) and RST/ACK (type-2) packets to both the corresponding client and the server to disrupt the ongoing connection and sustains the disruption for a certain period (90 seconds as per our measurements). …

According to previous work [3, 25] and our measurements, RST (type-1) and RST/ACK (type-2) are likely from two types of GFW instances that usually exist together. … Type-1 reset has only the RST flag set, and random TTL value and window sizes, while type-2 reset has the RST and ACK flags set, and cyclically increasing TTL value and window sizes.

Once a sensitive keyword detected, the GFW sends one type-1 RST and three type-2 RST/ACK with sequence numbers X, X+1460 and X+4380 (X is the current server-side sequence number).

Some other pieces of research that discuss GFW RST injection: https://censorbib.nymity.ch/#Crandall2007a, https://censorbib.nymity.ch/#Park2010a, https://censorbib.nymity.ch/#Xu2011a.

The pcap I have looked at (captured at the client in China) has two noteworthy features:

  1. There are 3 RST+ACK packets after the client's SYN. All 3 have a seq number equal to 0 and an ack number equal to the client's initial sequence number + 1.
  2. There are another 3 RST+ACK packets after the server's SYN+ACK. (Because this form of censorship uses only injection, and not packet dropping, the firewall does not prevent the client's SYN from reaching the real server, nor the server's real SYN+ACK from reaching the client.) All 3 have a seq number equal to the server's initial sequence number + 1, and an ack number equal to the client's initial sequence number + 1.

As noted above, the fact that sequence numbers are the same within each salvo of 3 injected packets is different from what has been seen before. It also appears to me that both the client's SYN and the server's SYN+ACK are triggers for injection. The injected packets are sent in both directions, so each side is likely to see 3 + 3 = 6 total injected RST+ACK packets.

Apart from the sequence numbers, TCP window sizes and IP TTLs are incremental within each salvo of 3, which is consistent with "Your State is Not Mine." Here is a summary of the pcap, showing 3 connection attempts:

trial # timestamp direction flags seq ack win ttl comment
1 0.000 C→S SYN cseq1 0 64240 64 client's real SYN
1 0.008 S→C RST+ACK 0 cseq1+1 4229 65 injected
1 0.008 S→C RST+ACK 0 cseq1+1 4231 67 injected
1 0.008 S→C RST+ACK 0 cseq1+1 4230 66 injected
1 0.248 S→C SYN+ACK sseq1 cseq1+1 65160 47 server's real SYN+ACK
1 0.248 C→S RST cseq1+1 0 0 64 client's real RST
1 0.250 S→C RST+ACK sseq1+1 cseq1+1 162 191 injected
1 0.250 S→C RST+ACK sseq1+1 cseq1+1 163 192 injected
1 0.250 S→C RST+ACK sseq1+1 cseq1+1 164 193 injected
2 2.288 C→S SYN cseq2 0 64240 64 client's real SYN
2 2.296 S→C RST+ACK 0 cseq2+1 3909 145 injected
2 2.296 S→C RST+ACK 0 cseq2+1 3910 146 injected
2 2.296 S→C RST+ACK 0 cseq2+1 3911 147 injected
2 2.508 S→C SYN+ACK sseq2 cseq2+1 65160 47 server's real SYN+ACK
2 2.508 C→S RST cseq2+1 0 0 64 client's real RST
2 2.511 S→C RST+ACK sseq2+1 cseq2+1 187 216 injected
2 2.511 S→C RST+ACK sseq2+1 cseq2+1 188 217 injected
2 2.511 S→C RST+ACK sseq2+1 cseq2+1 189 218 injected
3 3.428 C→S SYN cseq3 0 64240 64 client's real SYN
3 3.435 S→C RST+ACK 0 cseq3+1 3631 67 injected
3 3.435 S→C RST+ACK 0 cseq3+1 3632 68 injected
3 3.435 S→C RST+ACK 0 cseq3+1 3633 69 injected
3 3.668 S→C SYN+ACK sseq3 cseq3+1 65160 47 server's real SYN+ACK
3 3.668 C→S RST cseq3+1 0 0 64 client's real RST
3 3.670 S→C RST+ACK sseq3+1 cseq3+1 199 228 injected
3 3.670 S→C RST+ACK sseq3+1 cseq3+1 200 229 injected
3 3.670 S→C RST+ACK sseq3+1 cseq3+1 201 230 injected

Notice that the injected packets' seq values are equal within a salvo, and that win and ttl values are incremental (e.g. win=3909,3910,3911; ttl=145,146,147). It's also interesting that the window sizes are in the 3000–5000 range for the C→S direction, and in the 100–300 range in the S→C direction; I don't know if that pattern holds in general.

At https://github.com/net4people/bbs/issues/280#issuecomment-1712403769 it was reported that the injected RSTs' seq value could all be either 0 or a fixed value. (See the follow-up note about relative sequence numbers: what was reported as a sequence number of "1" actually means the client's initial sequence number + 1.) If my guess is right, the cases with seq=0 were triggered by the client's SYN, and the cases where seq is some other fixed value were triggered by the server's SYN+ACK.

RPRX commented 9 months ago

有一点需要补充,和以前不同的是,这两年我们已经能明显感受到 城市墙 的存在,不同地区,甚至同一地区的不同运营商都有不同的审查策略、执行封锁的方式,也就是说 @flowerinsnowdh 的测试目前仅代表一个地区,我们还需收集来自更多地区的反馈。

One thing to add, unlike before, in the past two years already we can clearly feel the presence of city walls, different regions and even different operators in the same region have different censorship strategies and ways of enforcing blocking, which means that @flowerinsnowdh's test is only representative of one region currently, and we still need to collect feedback from more regions.

RPRX commented 9 months ago

以及关于多个 RST,我有一个猜想:它们会不会是由不同层级的 GFW 分别发送的?比如市发一个、省发一个、出入境发一个。

As well as regarding multiple RSTs, I have a guess: could they be sent separately by different levels of GFW? For example, one sent by the city, one by the province, and one by the ingress/egress.

flowerinsnowdh commented 9 months ago

有一点需要补充,和以前不同的是,这两年我们已经能明显感受到 城市墙 的存在,不同地区,甚至同一地区的不同运营商都有不同的审查策略、执行封锁的方式,也就是说 @flowerinsnowdh 的测试目前仅代表一个地区,我们还需收集来自更多地区的反馈。

One thing to add, unlike before, in the past two years already we can clearly feel the presence of city walls, different regions and even different operators in the same region have different censorship strategies and ways of enforcing blocking, which means that @flowerinsnowdh's test is only representative of one region currently, and we still need to collect feedback from more regions.

我刚刚测试了一下用自己的网络 Wireshark 抓包,发现了变来变去无非类似下面三种情况

I just tested it with my own network Wireshark packet captures, and found out that the variation is similar to the following three cases

| 方向 *direction* | 时间(秒) *time (s)* | 报文 *message* | Sequence | Acknowledgment | Window | | :---: | :--------: | :----------------------: | :------: | :------------: | :-----: | | C➡️S | 0 | [SYN] | `a` | | `b` | | S➡️C | 0.009318 | [RST, ACK] | 0 | `a+1` | `c` | | C➡️S | 0.513842 | TCP Retransmission [SYN] | `a` | | `a` | | S➡️C | 0.522604 | [RST, ACK] | 0 | `a+1` | `c+32` | | C➡️S | 1.030284 | TCP Retransmission [SYN] | `a` | | `a` | | S➡️C | 1.039229 | [RST, ACK] | 0 | `a+1` | `c+50` | | C➡️S | 1.542319 | TCP Retransmission [SYN] | `a` | | `a` | | S➡️C | 1.551972 | [RST, ACK] | 0 | `a+1` | `c+63` | | C➡️S | 2.056196 | TCP Retransmission [SYN] | `a` | | `a` | | S➡️C | 2.065267 | [RST, ACK] | 0 | `a+1` | `c+115` |
| 方向 *direction* | 时间(秒) *time (s)* | 报文 *message* | Sequence | Acknowledgment | Window | | :---: | :--------: | :---------------------------------: | :------: | :------------: | :-----: | | C➡️S | 0 | [SYN] | `d` | | `b` | | S➡️C | 0.009161 | [RST, ACK] | 0 | `d+1` | `f` | | C➡️S | 0.515497 | TCP Retransmission [SYN] | `d` | | `b` | | S➡️C | 0.660272 | TCP Port numberes reused [SYN, ACK] | `e` | `d+1` | `b` | | C➡️S | 0.660403 | [ACK] | `d+1` | `e+1` | `g` | | C➡️S | 0.660776 | [PSH, ACK] Client Hello | `d+1` | `e+1` | `g` | | S➡️C | 0.670956 | [RST, ACK] | `e+1` | `d+1` | `f+28` | | S➡️C | 0.671687 | [RST, ACK] | `e+1` | `d+1` | `f+538` | | S➡️C | 0.671687 | [RST, ACK] | `e+1` | `d+1` | `f+539` | | S➡️C | 0.671687 | [RST, ACK] | `e+1` | `d+1` | `f+540` | | S➡️C | 0.806085 | [RST] | `e+1` | | 0 | | S➡️C | 0.806545 | [RST] | `e+1` | | 0 |
| 方向 *direction* | 时间(秒) *time (s)* | 报文 *message* | Sequence | Acknowledgment | Window | | :---: | :--------: | :--------------------------------------------: | :------: | :------------: | :----: | | C➡️S | 0 | [SYN] | `h` | | `b` | | S➡️C | 0.010296 | [RST, ACK] | 0 | `h+1` | `g` | | C➡️S | 0.521354 | TCP Retransmission [SYN] | `h` | | `b` | | S➡️C | 0.654938 | TCP Port numberes reused [SYN, ACK] | `i` | `h+1` | `b` | | C➡️S | 0.655161 | [ACK] | `h+1` | `i+1` | `m` | | C➡️S | 0.655302 | [PSH, ACK] TLSv1.3 Client Hello | `h+1` | `i+1` | `m` | | S➡️C | 0.665834 | [RST, ACK] | `i+1` | `h+1` | `g+32` | | S➡️C | 0.788712 | [ACK] | `i+1` | `h+262` | `n` | | S➡️C | 0.791007 | [ACK] TLSv1.3 Server Hello, Change Cipher Spec | `i+1` | `h+262` | `n+1` | | S➡️C | 0.791007 | [ACK] | `i+1437` | `h+262` | `n+1` | | S➡️C | 0.791007 | [ACK] TLSv1.3 Application Data | `i+2873` | `h+262` | `n+1` | | S➡️C | 0.791007 | [RST, ACK] | `i+1` | `h+262` | `g+34` | | S➡️C | 0.791007 | [RST, ACK] | `i+1437` | `h+262` | `g+35` | | S➡️C | 0.791007 | [RST, ACK] | `i+2873` | `h+262` | `g+36` |

好像 RST 报文不能完全阻断 TCP 连接,有时服务端也能成功发来 SYN ACK,RST 有时也指向了不同的目标,但是没有访问成功的案例,最终都会显示为连接重置

It seems that the RST message does not completely block the TCP connection, sometimes the server can also successfully send SYN ACK, RST sometimes also points to a different target, but there is no access to a successful case, will eventually show up as a connection reset

wkrp commented 9 months ago

I received two reports that match the pattern I described above: 3 RST+ACKs in response to the client's SYN, 3 RST+ACKs in response to the server's SYN+ACK, sequence numbers identical within a group of 3, incremental TTL and window size, no packet dropping.

But I also saw one other pcap that was a little different. In response to the client's SYN, there was only 1 RST+ACK, not 3, and there was some kind of packet dropping happening, because no SYN+ACK was received from the server. However, in a small number of cases, the client's SYN got through with no injection, and in those cases, the server's SYN+ACK elicited the same pattern of multiple RST+ACKs with incremental TTL and window size as described earlier. (I saw a case with 2 RST+ACKs and one with 3 RST+ACKs in response to the server's SYN+ACK.) In that pcap, it looks as though there are two different filters operating separately in each direction:

This other pcap, with different filtering in the C→S and S→C directions, matches the first two examples in what @flowerinsnowdh posted immediately above.

  1. In the first of the three examples, every client SYN gets 1 injected RST+ACK. There is no SYN+ACK from the server.
  2. In the second of the three examples, the first client SYN gets 1 injected RST+ACK. But the second client SYN gets through and the real server sends a SYN+ACK at time 0.654938. The client sends an empty ACK and then a Client Hello, but then there are 4 injected RST+ACK packets. The last 3 of the injected RST+ACKs match the pattern described earlier: all received at the same time, incremental window sizes (f+538, f+539, f+540). The first of the injected RST+ACKs doesn't fit the same pattern (and has a different timestamp); maybe it is in response to the client's ACK or Client Hello, not the server's SYN+ACK?

The third of the three examples is a little different.

  1. The client's first SYN gets 1 injected RST+ACK, as in the other two example example. But the client's next SYN gets through, and the client's Client Hello and the server's Server Hello both get through. There's 1 RST+ACK in response to the C→S traffic, then 3 RST+ACKs in response to the S→C traffic, but this time the sequence numbers are all different: i+1, i+1437, i+2873. This is similar to the sequence number spacing described in past work, though the intervals are different: +0, +1536, +2872 rather than +0, +1460, +4380. The window sizes are incremental as before: j+34, j+35, j+36.

So it appears at least 3 behaviors may be possible:

  1. 3 RST+ACK, constant sequence numbers. May be injected in response to either SYN or SYN+ACK.
  2. 1 RST+ACK in response to SYN, with packet dropping. In the case that the SYN does not get blocked, the SYN+ACK gets 3 RST+ACK as in case (1).
  3. 3 injected RST+ACK, sequence numbers separated by intervals. Maybe a result of both the SYN and the SYN+ACK not getting blocked, targeting data-carrying packets in the middle of a connection, not the start(?).
wkrp commented 9 months ago

以及关于多个 RST,我有一个猜想:它们会不会是由不同层级的 GFW 分别发送的?比如市发一个、省发一个、出入境发一个。

As well as regarding multiple RSTs, I have a guess: could they be sent separately by different levels of GFW? For example, one sent by the city, one by the province, and one by the ingress/egress.

In the case of the salvos of 3 RST+ACKs, I think it's unlikely that they sent by separate systems, because the arrival timestamps are so close (equal down to the millisecond in https://github.com/net4people/bbs/issues/285#issuecomment-1712442646 and https://github.com/net4people/bbs/issues/285#issuecomment-1712492809), and because the incremental IP TTL and TCP window size would be difficult to synchronize across different injection systems.

But in the case just above where it seems like there's different injection behavior in the C→S and S→C directions (1 RST+ACKs versus 3, packet dropping versus no), there I can easily believe that they are separate injection systems, based on what I have seen so far.

RPRX commented 9 months ago

Timestamps 非常相近的情况,还有可能是不同层级的 GFW 把封锁策略部署到了边缘(为了减轻出入境的压力等),比如说每一层都想给 https://1.1.1.1 发个 RST,而边缘没有先把这些策略合并,只是依次执行,如果有网站是两个 RST 则这一猜想更有可能为真

In cases where the timestamps are very similar, it is also possible that different tiers of GFW have deployed blocking policies to the edge (to take pressure off the entry and exit points, etc.), e.g. each tier is trying to send an RST to https://1.1.1.1 and the edge is just executing them sequentially without merging them first, which is a more likely scenario if there is a site with two RSTs

klzgrad commented 9 months ago

1 RST+ACK in response to SYN, with packet dropping

Why would it use such a method? Packet dropping would imply some buffering of packets before decision can be made about them, which is more costly than injecting resets from tapping firewalls. Also, in my testing the single RST/ACKs injected in response to SYNs have their mean RTTs approximately the same or slightly more than the RTT of the last hop in China, meaning these single RST/ACKs are still sent from near the last hop instead of some kind of edge nodes where inband firewalls can do more heavy duty blcoking.

Another interesting tidbit: I use this one-liner for i in $(seq 1000); do nc -v 1.1.1.1 443; done to quickly try and wait until the connection gets pass the first RST/ACK in response to SYN to see if there is anything interesting later in the connection. Then I manually type in GET / HTTP/1.1, resulting in this:

nc: connect to 1.1.1.1 port 443 (tcp) failed: Connection refused
nc: connect to 1.1.1.1 port 443 (tcp) failed: Connection refused
Connection to 1.1.1.1 443 port [tcp/https] succeeded!
GET / HTTP/1.1

HTTP/1.1 400 Bad Request
Server: cloudflare
...

In this case the connection is totally unmolested by resets. But if I use for i in $(seq 1000); do nc -v 1.1.1.1 443 <http-get.bin; done, I do receive resets. I haven't done any automated scripting to test this more systematically, but my quick guess is these resets "targeting data-carrying packets in the middle of a connection" have very short connection lifetime: If there is a few seconds of delay without packets after a SYN/ACK, there would be no further resets because the connection is considered expired. Very short lifetime would imply nearly depleted firewall resources enforcing this policy.

mmmray commented 9 months ago

I use this one-liner for i in $(seq 1000); do nc -v 1.1.1.1 443; done to quickly try and wait until the connection gets

curious if you are you reusing the same client port number here?

wkrp commented 9 months ago

1 RST+ACK in response to SYN, with packet dropping

Why would it use such a method? Packet dropping would imply some buffering of packets before decision can be made about them, which is more costly than injecting resets from tapping firewalls.

I agree, it doesn't make sense for there to be both RST injection and packet dropping.

It was suggested to me that the SYN+ACK might be getting dropped by a local NAT, rather than the GFW. The injected RST removes the NAT mapping, so the SYN+ACK gets dropped.

Something like this:

A timing digram showing the GFW injecting a RST+ACK towards both the client and the server; the RST+ACK arrives at the client's NAT, and the NAT drops the server's SYN+ACK that arrives shortly later.

klzgrad commented 9 months ago

curious if you are you reusing the same client port number here?

No, at least not shown in Wireshark.

Edit: My testing point is behind a CGNAT so it is possible the port translated by the CGNAT and visible to the GFW is reused. If that is the case, then it's possible the GFW also maintains a regular connection lifetime but ignores the rest of the connection after resets on both directions while the CGNAT reuses the port for a new connection.

bmixonba commented 9 months ago

1 RST+ACK in response to SYN, with packet dropping

Why would it use such a method? Packet dropping would imply some buffering of packets before decision can be made about them, which is more costly than injecting resets from tapping firewalls.

I agree, it doesn't make sense for there to be both RST injection and packet dropping.

It was suggested to me that the SYN+ACK might be getting dropped by a local NAT, rather than the GFW. The injected RST removes the NAT mapping, so the SYN+ACK gets dropped.

Something like this:

A timing digram showing the GFW injecting a RST+ACK towards both the client and the server; the RST+ACK arrives at the client's NAT, and the NAT drops the server's SYN+ACK that arrives shortly later.

You might be able to test this out by spamming SYNs from the client through the NAT. As long as a SYN makes it after the RST+ACK and before the SYN+ACK and matches the TCP connection (i.e., ports, ips, SEQ/ACK_SEQ), the SYN/ACK may reach the client. I'd expect so any way for something Linux or freebsd-like.

mmmray commented 9 months ago

Edit: My testing point is behind a CGNAT so it is possible the port translated by the CGNAT and visible to the GFW is reused. If that is the case, then it's possible the GFW also maintains a regular connection lifetime but ignores the rest of the connection after resets on both directions while the CGNAT reuses the port for a new connection.

something along those lines was also my thinking, and it would make sense together with the observation made in https://github.com/net4people/bbs/issues/275 -- the censorship system in the paper additionally uses sequence numbers to identify TCP sessions, and I suspect it is because they cannot maintain a strongly-consistent session table/flow tracking table in a central place due to the scale they're operating on. if there was a strongly-consistent table then I would have expected the SYN after the RST to actually be identified as a new connection.

assuming you have no way to determine which client port is visible outside of CGNAT (maybe cloudflare can reflect it for you or you try another server that does that), I wonder if you are hitting a race condition in flow tracking, and putting sleep between the nc changes the result? for example are you still able to connect if you wait 3 seconds after the RST before establishing a new connection

one thing I cannot wrap my head around is if all of this is consistent or inconsistent with @wkrp's findings

klzgrad commented 9 months ago

way to determine which client port is visible outside of CGNAT

The CGNAT in my testing cycles through 4096 externally visible ports per IP. But this seems to have no correlation with the frequency of getting a HTTP/1.1 400 Bad Request response from 1.1.1.1.

The following is just for fun. I use this script to collect some survival rates if there is some wait after connecting:

import socket
import time
import sys
from collections import defaultdict

if len(sys.argv) != 3:
    print(f'Usage: f{sys.argv[0]} clienthello.bin wait_after_connect')
    sys.exit(1)

clienthello = open(sys.argv[1], mode='rb').read()

wait_after_connect = float(sys.argv[2])
count = defaultdict(int)
for i in range(10000):
    with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
        try:
            s.connect(('1.1.1.1', 443))
        except ConnectionRefusedError as e:
            continue
        except ConnectionResetError as e:
            continue

        time.sleep(0.1)
        try:
            s.recv(1, socket.MSG_PEEK | socket.MSG_DONTWAIT)
        except ConnectionResetError as e:
            continue
        except BlockingIOError as e:
            pass

        time.sleep(wait_after_connect)

        try:
            s.sendall(clienthello)
        except ConnectionResetError as e:
            print(i, 'reset during send')
            count['reset during send'] += 1
            continue

        data = b''

        s.settimeout(0.5)
        reset = False
        while True:
            try:
                data += s.recv(10240)
            except TimeoutError as e:
                break
            except ConnectionResetError as e:
                print(i, 'reset after send')
                count['reset after send'] += 1
                reset = True
                break
        if reset:
            continue

        data_descr = f'{len(data)} bytes'
        print(i, data_descr)
        count[data_descr] += 1
print(dict(count))

and have these results:

python3 test.py clienthello.bin 0
592 reset after send
1367 reset after send
2193 reset after send
4046 reset after send
4079 reset after send
4090 reset after send
4091 reset after send
4093 reset after send
4119 reset after send
4133 reset after send
4140 reset after send
4148 reset after send
4153 reset after send
4159 reset after send
4172 reset after send
4179 reset after send
4215 reset after send
4222 reset after send
4233 reset after send
4242 reset after send
4247 reset after send
4249 reset after send
4253 reset after send
4258 reset after send
4267 reset after send
4272 reset after send
4273 reset after send
4275 reset after send
4276 reset after send
4295 reset after send
4318 reset after send
4559 reset after send
5096 reset after send
5652 reset after send
5935 reset after send
5969 reset after send
5984 reset after send
6001 reset after send
6008 reset after send
6012 reset after send
6014 reset after send
6178 reset after send
6408 reset after send
8318 reset after send
8360 reset after send
8385 reset after send
8400 reset after send
8421 reset after send
8431 reset after send
8435 reset after send
8454 reset after send
8456 reset after send
8479 reset after send
8481 reset after send
8489 reset after send
8497 reset after send
8524 4262 bytes
8544 reset after send
8554 reset after send
8559 4262 bytes
8579 4262 bytes
8585 reset after send
8595 reset after send
8600 reset after send
8607 reset after send
8623 reset after send
8631 reset after send
8652 reset after send
8679 reset after send
8684 reset after send
9148 reset after send
9556 reset after send
9591 reset after send
9609 reset after send
9613 reset after send
9616 reset after send
9619 reset after send
9636 reset after send
9672 reset after send
9677 reset after send
9685 reset after send
9693 reset after send
9726 reset after send
{'reset after send': 80, '4262 bytes': 3}

python3 test.py clienthello.bin 0.5
81 4262 bytes
2674 4262 bytes
2705 4262 bytes
2735 4262 bytes
2774 4262 bytes
2779 4262 bytes
2787 reset after send
2795 reset after send
2845 4262 bytes
3011 reset after send
3446 reset after send
3455 reset after send
3457 reset after send
3482 reset after send
6904 4262 bytes
6921 4262 bytes
6936 4262 bytes
6942 4262 bytes
6943 4262 bytes
6948 4262 bytes
6968 4262 bytes
6976 4262 bytes
7000 reset after send
7007 reset after send
7013 reset after send
7052 reset after send
7053 reset after send
7469 4262 bytes
7479 reset after send
7486 reset after send
7488 reset after send
7518 reset after send
7537 4262 bytes
7703 reset after send
7713 reset after send
7751 reset after send
7956 reset after send
9708 4262 bytes
{'4262 bytes': 18, 'reset after send': 20}

python3 test.py clienthello.bin 1
148 4262 bytes
1551 4262 bytes
5139 reset after send
6232 reset after send
7291 4262 bytes
7550 4262 bytes
7700 4262 bytes
7712 4262 bytes
7905 2800 bytes
7992 4262 bytes
8000 2800 bytes
8014 4262 bytes
8020 4262 bytes
8040 4262 bytes
8047 4262 bytes
8055 4262 bytes
8061 4262 bytes
8071 4262 bytes
9146 4262 bytes
9360 reset after send
9974 reset after send
9984 reset after send
{'4262 bytes': 15, 'reset after send': 5, '2800 bytes': 2}

python3 test.py clienthello.bin 2
1592 4262 bytes
1602 4262 bytes
1798 reset after send
3043 reset after send
4792 4262 bytes
5260 4262 bytes
5266 4262 bytes
5294 4262 bytes
5297 4262 bytes
5646 4262 bytes
5663 4262 bytes
5782 reset after send
6770 reset after send
7728 4262 bytes
7834 2800 bytes
9332 4262 bytes
9336 4262 bytes
9351 4262 bytes
9360 2800 bytes
9428 reset after send
9666 reset after send
9690 reset after send
9821 reset after send
{'4262 bytes': 13, 'reset after send': 8, '2800 bytes': 2}

python3 test.py clienthello.bin 5
3749 4200 bytes
3765 reset after send
3825 4262 bytes
3988 4262 bytes
4392 4262 bytes
5733 4262 bytes
7498 2800 bytes
8133 reset after send
8395 0 bytes
{'4200 bytes': 1, 'reset after send': 2, '4262 bytes': 4, '2800 bytes': 1, '0 bytes': 1}

python3 test.py clienthello.bin 10
485 4262 bytes
697 reset after send
1566 4262 bytes
2298 4262 bytes
3192 4262 bytes
4501 reset after send
6159 4262 bytes
7304 4262 bytes
8344 reset after send
9761 0 bytes
9914 4262 bytes
{'4262 bytes': 7, 'reset after send': 3, '0 bytes': 1}

Increasing the delay after connecting doesn't improve the survival rate after higher than 1 second. It seems there is some temporal locality in congestion(?) instead of just connection expiration in flow tracking.

for example are you still able to connect if you wait 3 seconds after the RST before establishing a new connection

For this:

python3 test.py clienthello.bin 1
947 reset after send
1753 reset after send
2724 4262 bytes
4040 4262 bytes
4714 4262 bytes
5295 4262 bytes
5300 reset after send
5450 4262 bytes
...
RPRX commented 9 months ago

此前我们观察到 GFW 发的 RST 包与正常包的 TTL 有明显差异,或许我们可以简单地丢弃 TTL 异常的 RST 包,给 GFW 加点成本

不过该方法对双向 RST 不太好使(如果目标网站不受控),以及境内段层层 NAT 可能会响应 RST,但是值得测试一下

Previously we observed that the RST packets sent by GFW had a significantly different TTL than normal packets, so perhaps we could simply discard the RST packets with the abnormal TTL and add some cost to GFW

However, this method doesn't work well for bi-directional RST (if the target site is uncontrolled), and the layers of NAT in the internal segments may respond to RST, but it's worth a test

mmmray commented 9 months ago

as you said, it also is not clear to me how any NAT alongside the terminated connection can be convinced to ignore the RST

it is likely that between censor and server, there is no NAT. I wonder if it is somehow possible to 1) drop RST on the server as you describe 2) re-connect from the server to the client using NAT traversal

https://tailscale.com/blog/how-nat-traversal-works/ is very interesting

however, generally i feel that all these tricks are full of detectable features, and at the same time fragile against network topology, so maybe the entire line of thinking is a dead end in general (i.e. we should accept the RST when it comes and instead try not to trigger it, as we already do)

RPRX commented 9 months ago

that all these tricks are full of detectable features, and at the same time fragile against network topology

是这样没错,这些小 trick 就只是像 @gfw-report 的那些方法一样,给 GFW 加点成本,若大范围传播、引起 GFW 的注意后可能很快就会失效。所以这些方法作为长期主力肯定是不行的,但这些测试可以 have fun 使我们对 GFW 这个黑箱有更多的了解。

就像很多伊朗朋友希望 Xray 加的 各种分片,我觉得它们特征明显,但有时就是能用,以及最近 @5e2t 又继续测试的 河南 RST

That's right, these little tricks are just a way to add cost to GFW like the @gfw-report methods, and they will probably fail quickly if they spread widely and get the attention of GFW. So these methods will not work as a long term mainstay, but these tests can have fun and give us a better understanding of the black box that is GFW.

Like the various fragments that many of my Iranian friends want Xray to add, which I think are obvious, but sometimes just work, and the Henan RST that @5e2t has continued to test recently.

RPRX commented 9 months ago

其实这里提到的 3 个 RST 看起来基本上是 GFW 特意设置的冗余策略,但我一直在想有没有一种可能,一个数据包会依次受到多级 GFW(或多个未经合并的规则)的审查、触发多次操作。

比如,@5e2t 发现他那里(河南)访问某些域名会收到 4 个 RST,其中一个 RST 与众不同:https://github.com/XTLS/Xray-core/issues/2426#issuecomment-1719448029

In fact, the 3 RSTs mentioned here seem to be basically a deliberate redundancy policy set up by GFW, but I've been wondering if it's possible for a single packet to be scrutinized by multiple levels of GFW (or multiple unconsolidated rules) in sequence, triggering multiple actions.

For example, @5e2t found that he was getting 4 RSTs for accessing certain domains there (Henan), one of which was different: https://github.com/XTLS/Xray-core/issues/2426#issuecomment-1719448029

wkrp commented 9 months ago

其实这里提到的 3 个 RST 看起来基本上是 GFW 特意设置的冗余策略

the 3 RSTs mentioned here seem to be basically a deliberate redundancy policy set up by GFW

Yes, the fact that there are 3 RSTs is not surprising (to me): it's been that way for as long as I can remember, at least as far back as 2006. The thing that seemed unusual to me, this time, is that the sequence numbers in the RST were equal, instead of being spaced out as has been documented before.

但我一直在想有没有一种可能,一个数据包会依次受到多级 GFW(或多个未经合并的规则)的审查、触发多次操作。

I've been wondering if it's possible for a single packet to be scrutinized by multiple levels of GFW (or multiple unconsolidated rules) in sequence, triggering multiple actions.

For DNS injection, at least, it is certainly the case that there are multiple middleboxes that independently inject false DNS responses. See the "Triplet Censors". It is possible to get 2 or 3 injected DNS responses for a single DNS query.

@5e2t 发现他那里(河南)访问某些域名会收到 4 个 RST,其中一个 RST 与众不同

@5e2t found that he was getting 4 RSTs for accessing certain domains there (Henan), one of which was different

I did a check just now to see what kind of RSTs I would get for simple HTTP Host censorship. It's easy to test because it's bidirectional. I used this command to send a request for a blocked domain name through the firewall from the outside (www.cqa.cn is the web site of an airport in China):

curl -v --connect-to ::www.cqa.cn http://torproject.org/

Interestingly, from this command I got seven RSTs, with apparently three different network fingerprints (note flags, win, ipid, ttl). It looks like a group of 1, a group of 3, and another group of 3. All of the sequence numbers were equal. TCP window sizes and IP TTLs were not incremental.

time direction flags seq ack win ipid ttl
0.49277 S→C RST 657250197 0 45685 0 70
0.49854 S→C RST+ACK 657250197 2473353066 3727 18048 150
0.49856 S→C RST+ACK 657250197 2473353066 3727 18048 150
0.50038 S→C RST+ACK 657250197 2473353066 3727 18048 150
0.50042 S→C RST+ACK 657250197 2473353066 647 57475 70
0.50044 S→C RST+ACK 657250197 2473353066 647 57475 70
0.50045 S→C RST+ACK 657250197 2473353066 647 57475 70

Maybe the "same sequence numbers" feature is not new with 1.1.1.1; maybe the sequence numbers stopped being spaced out some time in the past and it just wasn't documented.

wkrp commented 9 months ago

I got a report that RST injection of 1.1.1.1:443 has stopped as of 2023-09-20. What do you see?

klzgrad commented 9 months ago

Stopped here too.

flowerinsnowdh commented 9 months ago

I got a report that RST injection of 1.1.1.1:443 has stopped as of 2023-09-20. What do you see?

现在 https://1.1.1.1/ 可以访问了,https://8.8.8.8/ 仍不可访问

Now https://1.1.1.1/ is accessible, https://8.8.8.8/ is still inaccessible

RPRX commented 9 months ago

I got a report that RST injection of 1.1.1.1:443 has stopped as of 2023-09-20. What do you see?

当时我试了,可以访问了,但最近有报告称它又被针对了 https://github.com/net4people/bbs/issues/292#issuecomment-1742192257 ,你们测测看。

I tried it at the time and was able to access it, but recently there have been reports of it being targeted again https://github.com/net4people/bbs/issues/292#issuecomment-1742192257, test it out guys.

5e2t commented 9 months ago

@RPRX root@host:~# curl -v https://1.1.1.1

5e2t commented 9 months ago

@RPRX The traffic between me and 1.1.1.1 is through Shanghai exit GFW, I get RST in 30ms, the delay between me and Shanghai exit GFW is also 30ms.