Traffic does not reach server

Hello!

I am trying to understand how TPROXY works and I keep getting infinite loops (or traffic stuck, in this case), no matter the application that I use TPROXY with. I stumbled upon your repository and I was hoping that you could shed some light on what I'm doing wrong. I'm sorry in advance if my question is out of place.

This is my setup on CentOS 7 3.10.0-1160.83.1.el7.x86_64:

ip a s
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens192: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:50:56:8e:06:d8 brd ff:ff:ff:ff:ff:ff
    inet 10.17.133.214/21 brd 10.17.135.255 scope global noprefixroute dynamic ens192
       valid_lft 81182sec preferred_lft 81182sec
    inet6 fe80::8eda:d7f8:1055:4ab/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever

lsmod | grep -i tproxy
xt_TPROXY              17327  1 
nf_defrag_ipv6         35104  2 xt_socket,xt_TPROXY
nf_defrag_ipv4         12729  3 xt_socket,xt_TPROXY,nf_conntrack_ipv4

Routing rules:

ip rule add fwmark 1 lookup 100
ip route add local 0.0.0.0/0 dev lo table 100

ip route list table 100
local default dev lo scope host 

ip rule list
0:  from all lookup local
32762:  from all fwmark 0x1 lookup 100
32763:  from all fwmark 0x1 lookup 100
32764:  from all fwmark 0x1 lookup 100
32765:  from all fwmark 0x1 lookup 100
32766:  from all lookup main
32767:  from all lookup default

I have these iptables rules in place:

iptables -t mangle -L -nv
Chain PREROUTING (policy ACCEPT 2351 packets, 481K bytes)
 pkts bytes target     prot opt in     out     source               destination         
 255K  140M DIVERT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            socket
    2   112 TPROXY     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:80 TPROXY redirect 0.0.0.0:8080 mark 0x1/0x1

Chain INPUT (policy ACCEPT 2915 packets, 524K bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 369 packets, 44806 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain POSTROUTING (policy ACCEPT 369 packets, 44806 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain DIVERT (1 references)
 pkts bytes target     prot opt in     out     source               destination         
 255K  140M MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            MARK set 0x1
 255K  140M ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0

How I reproduce this (output provided after running all the steps):

I have an app that listens on port 80 on the machine:
```
python -m SimpleHTTPServer 80
Serving HTTP on 0.0.0.0 port 80 ...
```
Nothing gets here.

I start the go-tproxy example on the same machine:

./example 
2023/04/25 16:10:44 Starting GoLang TProxy example
2023/04/25 16:10:44 Binding TCP TProxy listener to 0.0.0.0:8080
2023/04/25 16:10:44 Binding UDP TProxy listener to 0.0.0.0:8080
2023/04/25 16:11:40 Accepting TCP connection from 10.17.132.130:40448 with destination of 10.17.133.214:80

Then it gets stuck.

From another machine, I run:

curl -v http://10.17.133.214:80/serverfile
*   Trying 10.17.133.214:80...
* Connected to 10.17.133.214 (10.17.133.214) port 80 (#0)
> GET /serverfile HTTP/1.1
> Host: 10.17.133.214
> User-Agent: curl/7.81.0
> Accept: */*
>

Then it gets stuck.

I was expecting that the traffic will reach the webserver and it will appear as it was coming from that other machine(the one I ran curl from), instead of the local machine, even though it was routed through the go-proxy application.

Please help me figure out is wrong in my usage or understanding of TPROXY, if you've got the time.

Thank you very much!

Hey @greenro With TPROXY there a thing about the routing path of all connections from and to the destination server/service. TPROXY is a network level intervention and there for the routes should be symmetrical between the client and the server to some degree. First cleanup the ip rules from 4 exactly the same to only 1 just to cleanup the TPROXY system routing rules. Then we need to understand the network infrastructure in your test setup. To try and understand all of this please attach the output of the next commands from all the nodes in the setup:

ip address
ip route
ip rule list
ip neigh
iptables-save

I have seen you are using a 10.17.128.0/21 CIDR which has the range between: 10.17.128.1 to 10.17.135.254 which means that you are doing something weird in your setup. How many machines are your setup? .. you should have 3 hosts ie client, TPROXY-Router, server. The client and the server cannot sit on the same network segment unless special static routing rules are present (for very unique edge cases) and usually the TPROXY-Router is sitting either as a designated router between the client and the server or somewhere on the route path between the client and the server.

If you wish to see an example setup of a squid TPROXY setup with WCCPv2 that can illustrate to some degree "a setup" take a peek at: http://wiki.squid-cache.org/ConfigExamples/UbuntuTproxy4Wccp2

I will try to help you as much as I can with this.

Hello @elico! Thank you so much for the help!

The setup in my original question consists of two machines on the same network:

CentOS: has the TPROXY setup and the server
Ubuntu: only runs the curl command and has the traffic redirected to the CentOS machine. Nothing else was specially configured for this setup.

I had to set up a new, clean environment with two machines:

Ubuntu20.04-1: the client - no additional configurations done

Output of commands asked:

# ip address
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
      link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
      inet 127.0.0.1/8 scope host lo
         valid_lft forever preferred_lft forever
      inet6 ::1/128 scope host 
         valid_lft forever preferred_lft forever
2: ens160: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
      link/ether 00:50:56:8e:40:29 brd ff:ff:ff:ff:ff:ff
      altname enp3s0
      inet 10.17.135.188/21 brd 10.17.135.255 scope global dynamic noprefixroute ens160
         valid_lft 67412sec preferred_lft 67412sec
      inet6 fe80::ef9e:b2d9:b2b9:c62b/64 scope link noprefixroute 
         valid_lft forever preferred_lft forever

# ip route
default via 10.17.128.1 dev ens160 proto dhcp metric 100 
10.17.128.0/21 dev ens160 proto kernel scope link src 10.17.135.188 metric 100 
169.254.0.0/16 dev ens160 scope link metric 1000 

# ip rule list
0:  from all lookup local
32766:  from all lookup main
32767:  from all lookup default

# ip neigh
10.17.128.1 dev ens160 lladdr 00:50:56:95:6f:83 REACHABLE
172.23.22.3 dev ens160 lladdr 00:50:56:95:6f:83 STALE
10.17.135.248 dev ens160 lladdr 00:50:56:8e:a9:6a STALE

# iptables-save
# *no output*

Ubuntu20.04-2: the server Configs ran:

ip -f inet rule add fwmark 1 lookup 100
ip -f inet route add local default dev ens160 table 100

echo 1 > /proc/sys/net/ipv4/ip_forward
echo 0 > /proc/sys/net/ipv4/conf/default/rp_filter
echo 0 > /proc/sys/net/ipv4/conf/all/rp_filter
echo 0 > /proc/sys/net/ipv4/conf/ens160/rp_filter

Output of commands asked:

# ip address
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
      link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
      inet 127.0.0.1/8 scope host lo
         valid_lft forever preferred_lft forever
      inet6 ::1/128 scope host 
         valid_lft forever preferred_lft forever
2: ens160: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
      link/ether 00:50:56:8e:a9:6a brd ff:ff:ff:ff:ff:ff
      altname enp3s0
      inet 10.17.135.248/21 brd 10.17.135.255 scope global dynamic noprefixroute ens160
         valid_lft 85641sec preferred_lft 85641sec
      inet6 fe80::9c38:a94a:e614:f211/64 scope link noprefixroute 
         valid_lft forever preferred_lft forever

# ip route
default via 10.17.128.1 dev ens160 proto dhcp metric 100 
10.17.128.0/21 dev ens160 proto kernel scope link src 10.17.135.248 metric 100 
169.254.0.0/16 dev ens160 scope link metric 1000 

# ip rule list
0:  from all lookup local
32765:  from all fwmark 0x1 lookup 100
32766:  from all lookup main
32767:  from all lookup default

# ip neigh
10.17.128.1 dev ens160 lladdr 00:50:56:95:6f:83 REACHABLE

# iptables-save
# Generated by iptables-save v1.8.4 on Wed May  3 16:07:11 2023
*mangle
:PREROUTING ACCEPT [523:125151]
:INPUT ACCEPT [43671:422610580]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [22433:1291448]
:POSTROUTING ACCEPT [22433:1291448]
:DIVERT - [0:0]
-A PREROUTING -p tcp -m socket -j DIVERT
-A PREROUTING -p tcp -m tcp --dport 80 -j TPROXY --on-port 3129 --on-ip 0.0.0.0 --tproxy-mark 0x1/0x1
-A DIVERT -j MARK --set-xmark 0x1/0xffffffff
-A DIVERT -j ACCEPT
COMMIT
# Completed on Wed May  3 16:07:11 2023
# Generated by iptables-save v1.8.4 on Wed May  3 16:07:11 2023
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
COMMIT
# Completed on Wed May  3 16:07:11 2023

Both machines are on the same network.

On the server I ran:

python3 -m http.server 80
Serving HTTP on 0.0.0.0 port 80 (http://0.0.0.0:80/) ...

And then:

./example 
2023/05/03 16:26:24 Starting GoLang TProxy example
2023/05/03 16:26:24 Binding TCP TProxy listener to 0.0.0.0:3129
2023/05/03 16:26:24 Binding UDP TProxy listener to 0.0.0.0:3129
2023/05/03 16:30:05 Accepting TCP connection from 10.17.135.188:34064 with destination of 10.17.135.248:80

On the client I ran:

curl http://10.17.135.248/serverfile
curl: (52) Empty reply from server

What I had before trying TPROXY: I have an app on Ubuntu20.04-2 which acts as a local proxy. Incoming traffic from Ubuntu20.04-1 is being redirected (iptables -j REDIRECT) to my app, and my app then re-injects the traffic. The issue is that, after reinjection, the source IP address will be Ubuntu20.04-2's address. client->iptables(REDIRECT)->proxy(my app)->server - (wrong source IP)

What I want to achieve with TPROXY: I want that when Ubuntu20.04-1 client accesses the app on Ubuntu20.04-2 through the TPROXY, TPROXY will redirect the traffic to my app, which will then re-inject the traffic and send it to the reserver. I want the server to see the original IP address of the client, and not the address of Ubuntu20.04-2. client->iptables(TPROXY)->proxy(my app)->server - (correct source IP)

If you got any more ideas for me to try, please shoot. I am thinking that my requirements are not really fit for use with TPROXY, however I want to make sure of that before dropping TPROXY and trying something else.

Thanks again for the help!

I have not found a solution as of yet. I am closing this as I don't think it is suited for my use case.

@greenro All The Bests.

KatelynHaworth / go-tproxy

Traffic does not reach server #13