Getting stuck at streamConn

[phuongvietvu0306]

I have followed all the instructions and run the example but I cannot get it to work. I setup go-tproxy on a local server and have my computer setting default gateway to that server. I see log printing "Accepting TCP connection from [myComputerIP:port] with the destination of [destinationIP:port]" However the browser keeps spinning and waiting. I put some log printing in the example and it seems like it got stuck in the streamConn (copy 0 bytes and nil error). Can you please help me fixing this issue Thank you

[KatelynHaworth]

Are you able to provide some sample code that you are working with?

Also what operating system is the code running on (e.g. Ubuntu 16.04, etc)?

[phuongvietvu0306]

I am currently exactly the example code that you provide. I am running it on my server Ubuntu 16.04 and 1 more PC desktop also running Ubuntu 16.04 Do you need some more information

[KatelynHaworth]

On the ubuntu server, is IP forwarding enabled along with the appropriate IPTables rules for TProxy?

[phuongvietvu0306]

Yes I think I have all the iptables rules added correctly. I try to keep it as correctly as the example without modifying anything, even my binding port. My iptables rules are:

iptables -t mangle -N DIVERT
iptables -t mangle -A PREROUTING -p tcp -m socket -j DIVERT
iptables -t mangle -A DIVERT -j MARK --set-mark 1
iptables -t mangle -A DIVERT -j ACCEPT
iptables -t mangle -A PREROUTING -p tcp --dport 80 -j TPROXY --tproxy-mark 0x1/0x1 --on-port 8080

I am a little uncertain if I have to modify any of these to fit my environment:

ip rule add fwmark 1 lookup 100
ip route add local 0.0.0.0/0 dev lo table 100

In addition to your instruction. I also added:

sudo sysctl -w net.ipv4.ip_forward=1
sudo sysctl -w net.ipv6.conf.all.forwarding=1

[phuongvietvu0306]

In addition, as the the example iptables rules above, I think it only forward port 80 right? I tested on my browser, any HTTPS url can still work correctly. Currently only HTTP websites on port 80 are not working. If I add forward port 443 in the iptables, then I HTTPS websites are not working anymore

[KatelynHaworth]

I put some log printing in the example and it seems like it got stuck in the streamConn (copy 0 bytes and nil error)

Do you know which stream this was that had a 0 byte copy, was it conn->remoteConn or remoteConn->conn?

[phuongvietvu0306]

I have modified this part in "handleTCPConn" in the example:

streamConn := func(name string, dst io.Writer, src io.Reader) {
    n, err := io.Copy(dst, src)
    fmt.Println("Streaming", name, n, err)
    streamWait.Done()
}

go streamConn("remote to conn", remoteConn, conn)
go streamConn("conn to remote", conn, remoteConn)

And the result is the following. The websites that I tested on is still live and good.

Streaming remote to conn 0 readfrom tcp 10.3.4.151:60554: write tcp 10.3.4.151:60554: write: connection timed out
Streaming conn to remote 0 <nil>
Streaming remote to conn 0 readfrom tcp 10.3.4.151:60551: write tcp 10.3.4.151:60551: write: connection timed out
Streaming conn to remote 0 <nil>
Streaming remote to conn 0 readfrom tcp 10.3.4.151:60552: write tcp 10.3.4.151:60552: write: connection timed out
Streaming conn to remote 0 <nil>
Streaming remote to conn 0 readfrom tcp 10.3.4.151:60555: write tcp 10.3.4.151:60555: write: connection timed out
Streaming conn to remote 0 <nil>

[KatelynHaworth]

Awesome, that helps a lot more, one thing to note though about the streamConn function, the first argument is the destination and the second is the source, so that error is while streaming data from the client to the remote.

That being said, it feels like there may be asymmetric routing happening on your network, so the data is leaving TProxy with the address of your client machine but when it is coming back from the remote host your router is sending it directly to your client rather than sending it to the TProxy server.

Do you possibly have a diagram of the network layout for this test?

[phuongvietvu0306]

Awesome, that helps a lot more, one thing to note though about the streamConn function, the first argument is the destination and the second is the source, so that error is while streaming data from the client to the remote.

So do I have to switch remoteConn and conn? And the result should be like this? I changed like this and it is still not working:

go streamConn("remote to conn", conn, remoteConn)
go streamConn("conn to remote", remoteConn, conn)

result:

Streaming conn to remote 0 readfrom tcp 10.3.4.151:60667: write tcp 10.3.4.151:60667: write: connection timed out
Streaming remote to conn 0 <nil>

[KatelynHaworth]

Changing the order won't change much. again I think it may be asymmetric routing, are you able to do a packet capture on something further upstream to see if a response is being sent from remote?

[phuongvietvu0306]

That being said, it feels like there may be asymmetric routing happening on your network, so the data is leaving TProxy with the address of your client machine but when it is coming back from the remote host your router is sending it directly to your client rather than sending it to the TProxy server.

Do you possibly have a diagram of the network layout for this test?

I'm not sure what you mean because I'm not very specialized in network. I have 2 test environments: laptop -> server and laptop -> PC

• I'm use my laptop as client, set default gateway to my server. Both are in the same subnet and IP Range. Laptop is Macbook and Server is Ubuntu 16.04
• I'm use my laptop as client, set default gateway to my PC. Both are in the same subnet and IP Range. Laptop is Macbook and PC is Ubuntu 16.04

[KatelynHaworth]

The core problem with asymmetric routing is that if the upstream router has a direct connection to the client PC, rather than sending response packets to the TProxy server it will send it directly to the client PC.

For TProxy to work, all network traffic has to flow through it so the upstream router can't send it packets directly to the client.

There are two ways to accomplish this:

1) Use two networks so that the upstream router is on one and the client is on another, the TProxy server is setup to be the router for the client's network (this is how the vagrant example works)

2) Setup the TProxy server with two network interfaces, one attached to the network that hosts the router, the other directly connected to the client PC. From there, setup a bridge with the two interface and configure the TProxy rule on the bride interface.

[KatelynHaworth]

@phuongvietvu0306 Have you had progress in getting this to work? I would like to close this issue if it is no longer valid

[KatelynHaworth]

Closing as stale

[learnerBing]

can we add a client as another vagrant virtual machine, to the extent of the example code? So there is no setup required for the initial setup?