WT & masscan. Only the first open port is reported

[SkyperTHC]

This problem does never show up if I'm using WG on both sides. It shows up when I'm using WT on the EXIT-NODE The bug is re-produceable (i tested 20x the same scenario and it showed up reliably).

I'm testing the current github from branch tcp-fix. I'm scanning for 2 ports on a single target only. Both ports are open on the target. Only the first one is found.

On the origin server (running wireguard)

masscan --interface wgExit --rate 1 -p31337,22 --open-only 45.33.32.156

Packets on origin server (wireguard):

13:09:42.916290 IP 172.16.0.2.49782 > 45.33.32.156.31337: Flags [S], seq 699471134, win 1024, length 0
13:09:43.073437 IP 45.33.32.156.31337 > 172.16.0.2.49782: Flags [S.], seq 3059500064, ack 699471135, win 27584, options [mss 1380], length 0
13:09:43.073500 IP 172.16.0.2.49782 > 45.33.32.156.31337: Flags [R], seq 699471135, win 0, length 0
13:09:43.916450 IP 172.16.0.2.49782 > 45.33.32.156.31337: Flags [R], seq 699471135, win 1200, length 0

13:09:44.916482 IP 172.16.0.2.49782 > 45.33.32.156.22: Flags [S], seq 2369022976, win 1024, length 0

Observe that the TCP SYN-ACK from port 22 is never send to the origin server. We only the the outgoing SYN.

Packet on Exit-Node to target (running wiretap):

15:09:42.918608 IP 192.145.44.201.49070 > 45.33.32.156.31337: Flags [S], seq 299369726, win 16060, length 0
15:09:43.070073 IP 45.33.32.156.31337 > 192.145.44.201.49070: Flags [S.], seq 619693497, ack 299369727, win 65160,  length 0
15:09:43.070131 IP 192.145.44.201.49070 > 45.33.32.156.31337: Flags [.], ack 1, win 8, length 0
15:09:43.222053 IP 45.33.32.156.31337 > 192.145.44.201.49070: Flags [F.], seq 1, ack 1, win 510, options [nop,nop,TS val 2032056807 ecr 1905127897], length 0
15:09:43.224657 IP 192.145.44.201.49070 > 45.33.32.156.31337: Flags [.], ack 2, win 8, length 0

15:09:44.918731 IP 192.145.44.201.36134 > 45.33.32.156.22: Flags [S], seq 292390217, win 16060, length 0
15:09:45.070204 IP 45.33.32.156.22 > 192.145.44.201.36134: Flags [S.], seq 3384812886, ack 292390218, win 65160, length 0
15:09:45.070277 IP 192.145.44.201.36134 > 45.33.32.156.22: Flags [.], ack 1, win 8, length 0
15:09:45.227020 IP 45.33.32.156.22 > 192.145.44.201.36134: Flags [P.], seq 1:45, ack 1, win 510, length 44: SSH: SSH-2.0-OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.13
15:09:45.227053 IP 192.145.44.201.36134 > 45.33.32.156.22: Flags [.], ack 45, win 8, length 0

15:09:48.071226 IP 192.145.44.201.49070 > 45.33.32.156.31337: Flags [F.], seq 1, ack 2, win 8, length 0
15:09:48.223085 IP 45.33.32.156.31337 > 192.145.44.201.49070: Flags [.], ack 2, win 510, length 0
15:09:50.070754 IP 192.145.44.201.36134 > 45.33.32.156.22: Flags [R.], seq 1, ack 45, win 8, length 0

The socket information on the exit node shows that the data is in the recv-q (44 bytes, "OpenSSH_..."):

tcp       44      0 192.145.44.201:36134    45.33.32.156:22        

The data stays in the RECV-Q for 5 seconds before the socket is closed.

The expected behaviour is for WT to:

1. Send SYN/ACK to origin-host when TCP to target completes
2. Send data received from target to origin-host

My gut feeling is that WT misses that the TCP connection has been established successfully and then kills it after 5 seconds.

It is odd because it works when I'm using nmap instead of masscan:

 nmap -p31337,22 -Pn -sS 45.33.32.156

The only difference that I can see is that nmap uses the Linux kernel (rather than raw TCP) and thus the SYN is re-send 2 times:

# FIRST SYN to port 22
13:16:49.292759 IP 172.16.0.2.42152 > 45.33.32.156.22: Flags [S], seq 2471212292, win 1024, options [mss 1460], length 0

13:16:49.292816 IP 172.16.0.2.42152 > 45.33.32.156.31337: Flags [S], seq 2471212292, win 1024, options [mss 1460], length 0
13:16:49.451170 IP 45.33.32.156.31337 > 172.16.0.2.42152: Flags [S.], seq 2053963635, ack 2471212293, win 27584, options [mss 1380], length 0
13:16:49.451203 IP 172.16.0.2.42152 > 45.33.32.156.31337: Flags [R], seq 2471212293, win 0, length 0

# SECOND SYN to port 22. Why is this needed? 
13:16:51.092230 IP 172.16.0.2.42154 > 45.33.32.156.22: Flags [S], seq 2471081222, win 1024, options [mss 1460], length 0

13:16:51.250134 IP 45.33.32.156.22 > 172.16.0.2.42154: Flags [S.], seq 2537989258, ack 2471081223, win 27584, options [mss 1380], length 0
13:16:51.250179 IP 172.16.0.2.42154 > 45.33.32.156.22: Flags [R], seq 2471081223, win 0, length 0

My gut says that WT somehow needs this 2nd SYN?

[luker983]

I ran into this as well. The problem is that Wiretap uses gvisor's DNAT rules to intercept TCP connections. When two SYNs come in from the same source address and port it creates a conflict in the NAT table.

Aside from reimplementing NAT or a different transparent proxy solution, I think the proper way to do this is implementing Implicit Source Port Mapping (https://www.netfilter.org/documentation/HOWTO/NAT-HOWTO-6.html) in netstack. I raised an issue upstream: https://github.com/google/gvisor/issues/9030.

nmap is sending a second SYN with a new source port so no NAT conflict applies. Unfortunately if you try to scan more ports nmap will suffer the same problem.

Potential workarounds while a proper solution is developed:

• Use connect scans
• Configure nmap or masscan in such a way that SYN scans iterate through source ports before reusing a source port?

[SkyperTHC]

It doesnt happen often - you made me fall off my chair. Well spotted. It's crazy that gvisor's DNAT has such an obvious bug.

The specs state that the same SRC-IP:SRC-PORT can be used to open different and multiple tcp connection. A TCP connection is identified by "SRC-IP + SRC-PORT + DST-IP + DST-PORT" and not just by SRC-IP and SRC-PORT.

It can be easily tested:

nc -p 4444 45.148.244.66 31337
nc -p 4444 45.148.244.66 31338

ss -antp
ESTAB    0      0         172.16.0.2:4444  45.148.244.66:31338 users:(("nc",pid=181,fd=3))
ESTAB    0      0         172.16.0.2:4444  45.148.244.66:31337 users:(("nc",pid=86,fd=3))

Et voila, two fully established TCP connection with SRC-IP, SRC-PORT and DST-IP identical...just DST-Port differs.

More here:

• 1993 UNIX® System V Network Programming
• 1986 Design of the UNIX Operating System, 1st edition

I hear Jon Postel laughing all the way from his grave...

[luker983]

The dnat-to-forwarder branch fixes this problem by no longer relying on DNAT.