pf route-to : very slow upload speed (less than 0.2 Mb/s)

[cattyhouse]

Download speed is ok (direct or via tun) Upload speed to ip in chnroute is ok (direct) Upload speed to the rest of ip very slow (through tun)

• hev conf

tunnel:
  name: utun
  mtu: 1500
  multi-queue: true
  ipv4: 198.18.0.1
  post-up-script: /opt/pf.tproxy/bin/up.sh
  pre-down-script: /opt/pf.tproxy/bin/down.sh
socks5:
  port: 1080
  address: 127.0.0.1
  udp: 'udp'
misc:
  limit-nofile: 655350

• pf conf

pass out quick on en0 inet proto tcp from any to TROJAN_SERVER_IP port = 443 flags S/SA keep state
pass out quick on en0 proto tcp from any to <direct> flags S/SA keep state
pass out quick on en0 proto udp from any to <direct> keep state
pass out on en0 route-to utun6 inet proto tcp all flags S/SA keep state
pass out on en0 route-to utun6 inet proto udp all keep state

EDIT

issue fixed by turning of tso sudo sysctl net.inet.tcp.tso=0

[cattyhouse]

the issue is not about hev-socks-tunnel, because i tried to use route add instead of pf rules, the upload is fine.

[cattyhouse]

i think the route-to has to be taken care of. but i don't know how, what is the difference of packet flow between download and upload via tun interface?

[heiher]

try to set mtu = 8500.

[cattyhouse]

try to set mtu = 8500.

same.

[heiher]

Okay. I suggest you use mtu = 8500.

For this issue, are you sure the upload traffics through tun by pf?

[cattyhouse]

Okay. I suggest you use mtu = 8500.

For this issue, are you sure the upload traffics through tun by pf?

i think so, when i do scp 100M_file root@host_not_in_direct_table: , based on this rule pass out on en0 route-to utun6 inet proto udp all keep state, it should go through tun

[heiher]

Can you verify it by packet capture?

[cattyhouse]

sudo tcpdump -n -i any dst host host_not_in_direct_table and dst port 22:

14:48:46.281212 IP en0_ip.49405 > host_not_in_direct_table.22: Flags [.], seq 130845:132305, ack 10825, win 65535, length 1460

[cattyhouse]

if i disable pf with pfctl -d. then use pure route like route delete default , route add default utun_ip_addr, upload is ok.

[heiher]

Can you simplify pf rules like this, and then what happens?

[cattyhouse]

Can you simplify pf rules like this, and then what happens?

that rule is not complete, because out via lo will also be route-to utun, e.g. 127.0.0.1.any -> 127.0.0.1.socs5_port will be route-to utun, then it is a dead loop

[heiher]

I mean to keep the pf rules simple. for example, just proxy the nobody user to prevent dead loop.

[cattyhouse]

same slow upload speed. by the way, this user method still have to set skip on lo0

sudo pfctl -s rules

No ALTQ support in kernel
ALTQ related functions disabled
pass out route-to utun6 inet proto tcp all user != 4294967294 flags S/SA keep state
pass out route-to utun6 inet proto udp all user != 4294967294 keep state

[cattyhouse]

so, let's continue the pf route-to issue, i did some other experiment, run the trojan-go socks5 server on one my of LAN machine, and set that as hev's upstream, trojan-go debug shows whenever there is a udp dns(udp 1.1.1.1:53) look up, the dst trojan-go sees is meta 0.0.0.0, but tcp dns (like dig +tcp @1.1.1.1 ) look up show correct dst.

can you take a look at the source code of pf route-to logic in here? git clone --depth 1 https://github.com/apple-oss-distributions/xnu. the total xnu kernel source code size is about 193M

i believe this is also the case on FreeBSD. however OpenBSD has a different PF than macOS and FreeBSD, e.g. on OpenBSD the route-to arg has to be redirhost = address [ "/" mask-bits ]

[heiher]

I can't reproduce it on my virtual machine (macOS 11.7). Even with pf route-to, both upload and download speeds are around 500Mbps.

[cattyhouse]

I can't reproduce it on my virtual machine. Even with pf route-to, both upload and download speeds are around 500Mbps.

ok, it is weird. what is the tested OS?

but now i switched to using route method:

• route add trojan server to router ip
• route add inverted chnroute+private (that's all the ips not from china and private, tool used ip-dedup -i) to utun's ip, the list is about 14000 lines, it takes 15s to add to routing table.

this way, no upload speed issue. and as long as hev does not quit, the routing entries will not get flushed when switch Wi-Fi network (although need to change trojan server's route again)

[heiher]

I can't reproduce it on my virtual machine. Even with pf route-to, both upload and download speeds are around 500Mbps.

ok, it is weird. what is the tested OS?

macOS 11.7

but now i switched to using route method:

* route add trojan server to router ip

* route add inverted chnroute+private (that's all the ips not from china and private, tool used `ip-dedup -i`) to utun's ip, the list is about 140000 items, it takes 15s to add to routing table.

Interesting issue. Can you find similar issues on the Internet?

this way, no upload speed issue. and as long as hev does not quit, the routing entries will not get flushed when switch Wi-Fi network (although need to change trojan server's route again)

[cattyhouse]

I can't reproduce it on my virtual machine (macOS 11.7). Even with pf route-to, both upload and download speeds are around 500Mbps.

i guess your socks5 server is hev-socks5-server? and udp mode is set to tcp

[heiher]

I can't reproduce it on my virtual machine (macOS 11.7). Even with pf route-to, both upload and download speeds are around 500Mbps.

i guess your socks5 server is hev-socks5-server? and udp mode is set to tcp

Yeah. It's hev-socks5-server. and udp mode is set to udp.

[cattyhouse]

I can't reproduce it on my virtual machine (macOS 11.7). Even with pf route-to, both upload and download speeds are around 500Mbps.

i guess your socks5 server is hev-socks5-server? and udp mode is set to tcp

Yeah. It's hev-socks5-server. and udp mode is set to udp.

can you try trojan-go client mode? or trojan-r client mode? this is more like a real world use case.

go mod tidy -go=$(go env GOVERSION | sed 's/^go//')
go get -u
CGO_ENABLED=0 GOARCH=amd64 GOOS=darwin go build -tags "client forward" -trimpath -ldflags="-s -w -buildid=" -o trojan-go-mac-x86_64

{
    "run_type": "client",
    "local_addr": "127.0.0.1",
    "local_port": 1080,
    "remote_addr": "x.x.x.x",
    "remote_port": 443,
    "password": [
        "some_password"
    ],
    "ssl": {
        "sni": "example.com"
    },
    "log_level": 0
}

[cattyhouse]

Interesting issue. Can you find similar issues on the Internet?

route-to is rarely used like this. i ask in #freebsd #openbsd irc channel, they have no clue.

[cattyhouse]

no need to try trojan-go, i just run hev-socks5-server on a LAN machine : 192.168.11.2, same issue, if your macOS 11.7 works, then my macOS 14.3 bug?

• speedtest -s ID

Server: Frontier - Los Angeles, CA (id: 14236)
Download:   309.86 Mbps (data used: 364.8 MB)
Upload:     0.07 Mbps (data used: 129.5 kB)

Server: China Telecom JiangSu 5G - Zhenjiang (id: 36663)
Download:   311.25 Mbps (data used: 563.7 MB)
Upload:     0.17 Mbps [=========/          ] 47%   - latency: 13.59 ms

• hev-socks5-tunnel

  
  tunnel:
  name: utun
  mtu: 8500
  multi-queue: true
  ipv4: 198.18.0.1
  post-up-script: /opt/pf.tproxy/bin/up.sh
  pre-down-script: /opt/pf.tproxy/bin/down.sh
  socks5:
  port: 1080
  address: 192.168.11.2
  udp: 'udp'
  misc:
  limit-nofile: 655350

- pf
```sh
set skip on lo0

pass out quick inet proto tcp from any to 192.168.11.2 flags S/SA keep state
pass out quick inet proto udp from any to 192.168.11.2 keep state
pass out route-to utun4 inet proto tcp all flags S/SA keep state
pass out route-to utun4 inet proto udp all keep state

[heiher]

I think I understand the difference between pf route-to and normal route. This may be the root cause of slow transmission.

Although the upload speed on my virtual machine is not very slow, it is still slower than the normal route. 500Mbps vs 1000Mbps. The difference is that pf route-to cannot make the operating system network stack correctly obtain the MTU of utun, so that the MTU of the TCP flow is only 1500, while the normal route MTU is 8500.

No idea about how to solve it.

[cattyhouse]

Wonder if there are any sysctl options to tune that?

[cattyhouse]

close as fixed, see the first comment.