Transparently proxying UDP traffic with pf?

[Orum]

This is a similar symptom to my earlier issue (#1473), but as the circumstances have drastically changed I thought I'd open a new issue. To summarize the changes, I am no longer running sslocal on the host (Linux/iptables) machine; it's instead being run on my FreeBSD router which uses pf.

Obviously, this requires different firewall rules to work, but there is little if any documentation I could find for transparently proxying via pf within the shadowsocks-rust project. What I've done here is largely guesswork, but this is the rule I came up with to transparently proxy both TCP & UDP to sslocal (and then on to ssserver): rdr on $int_if inet proto { tcp, udp } to !<private> -> 127.0.0.1 port 1080

sslocal is being run on the router via the following command: sslocal -b 127.0.0.1:1080 -U --protocol redir -s 192.168.x.y:8388 -m none --tcp-redir pf --udp-redir pf

...and finally, ssserver is run with -U -m none -b 192.168.x.y:8388

This "works" in the sense that both TCP & UDP traffic are being redirected to sslocal (unlike in #1473 where only TCP traffic was ever reaching sslocal), which is then sending the traffic on to sserver. However, once again, only TCP traffic is being fully proxied correctly.

UDP traffic does arrive at sslocal without any doubt, as it shows up when running sslocal with -vvv like so (timestamps removed for brevity, and the host system's address being substituted here with simply <host>):

TRACE  tokio-runtime-worker ThreadId(05) shadowsocks::relay::udprelay::proxy_socket: crates/shadowsocks/src/relay/udprelay/proxy_socket.rs:235: UDP server client send to 127.0.0.1:1080, control: UdpSocketControlData { client_session_id: 7043831170210357480, server_session_id: 0, packet_id: 1, user: None }, payload length 96 bytes, packet length 103 bytes
TRACE  tokio-runtime-worker ThreadId(05) shadowsocks_service::local::redir::udprelay: crates/shadowsocks-service/src/local/redir/udprelay/mod.rs:307: received UDP packet from <host>:44849, destination 127.0.0.1:1080, length 96 bytes
TRACE  tokio-runtime-worker ThreadId(05) shadowsocks_service::local::net::udp::association: crates/shadowsocks-service/src/local/net/udp/association.rs:433: udp relay <host>:44849 -> 127.0.0.1:1080 (proxied) with 96 bytes

This traffic is then forwarded on to ssserver, which I can verify by watching outgoing traffic on the router via tcpdump. After they arrive at ssserver though, they fail to reach the internet, so clearly something isn't configured correctly.

My best guess as to what is happening here, based on the logs, is that sslocal thinks the destination of the UDP packet is 127.0.0.1 (i.e. the address that sslocal is running on and where incoming UDP traffic on the router is redirected to), and fails to retrieve the original destination address before pf's rdr rule took effect. This wouldn't surprise me as my rdr rule is complete guesswork, so I assume I'm missing something.

If that is indeed the problem, what should the rdr rule look like to transparently proxy UDP traffic with pf? Or am I barking up the wrong tree, and is the problem due to something else entirely?

[ge9]

Did you confirm an infinite loop actually happened? The config seems fine to me. Packets coming out from 192.168.11.191 will be routed on lo0 and then redirected to 127.0.0.1:22222, being received by sslocal.

[zonyitoo]

2024-06-18T23:31:54.173984822Z DEBUG tokio-runtime-worker ThreadId(07) shadowsocks_service::local::net::udp::association: udp association for 192.168.11.191:50143 is closed
2024-06-18T23:31:54.174065558Z DEBUG tokio-runtime-worker ThreadId(09) shadowsocks_service::local::net::udp::association: udp association for 192.168.11.191:15262 is closed
2024-06-18T23:31:54.174056618Z DEBUG tokio-runtime-worker ThreadId(03) shadowsocks_service::local::net::udp::association: udp association for 192.168.11.191:55806 is closed
2024-06-18T23:31:54.174088187Z DEBUG tokio-runtime-worker ThreadId(02) shadowsocks_service::local::net::udp::association: udp association for 192.168.11.191:28037 is closed
2024-06-18T23:31:54.174128695Z DEBUG tokio-runtime-worker ThreadId(06) shadowsocks_service::local::net::udp::association: udp association for 192.168.11.191:12287 is closed
2024-06-18T23:31:54.174117241Z DEBUG tokio-runtime-worker ThreadId(04) shadowsocks_service::local::net::udp::association: udp association for 192.168.11.191:51797 is closed
2024-06-18T23:31:54.174116682Z DEBUG tokio-runtime-worker ThreadId(05) shadowsocks_service::local::net::udp::association: udp association for 192.168.11.191:35430 is closed
2024-06-18T23:31:54.17421446Z DEBUG tokio-runtime-worker ThreadId(09) shadowsocks_service::local::net::udp::association: udp association for 192.168.11.191:54599 is closed
2024-06-18T23:31:54.174220885Z DEBUG tokio-runtime-worker ThreadId(03) shadowsocks_service::local::net::udp::association: udp association for 192.168.11.191:15870 is closed
2024-06-18T23:31:54.174257761Z DEBUG tokio-runtime-worker ThreadId(07) shadowsocks_service::local::net::udp::association: udp association for 192.168.11.191:43392 is closed
2024-06-18T23:31:54.174318663Z DEBUG tokio-runtime-worker ThreadId(06) shadowsocks_service::local::net::udp::association: udp association for 192.168.11.191:15490 is closed

Yes, it do exist.

[ge9]

Ah, did you assigned an additional IP? I assigned both 10.0.2.15 and 10.0.2.25 on the external interface. 10.0.2.15 was assigned by default (by Virtualbox's DHCP) and 10.0.2.25 is manually assigned, thus can be used for proxying.

[zonyitoo]

Nope. 192.168.11.192 was assigned by DHCP. So I should add another IP to em0?

ifconfig em0 192.168.11.20 netmask 255.255.255.0 alias

[zonyitoo]

Ok now, here is the problem. Currently sslocal cannot get the original destination IP address from FreeBSD, so

nc -us 192.168.11.20 192.168.11.1 8888 -v

was successfully redirected to 127.0.0.1:22222, but

2024-06-18T23:54:44.357260908Z DEBUG tokio-runtime-worker ThreadId(02) shadowsocks_service::local::net::udp::association: created udp association for 192.168.11.20:45818
2024-06-18T23:54:44.363167525Z DEBUG tokio-runtime-worker ThreadId(02) shadowsocks_service::local::net::udp::association: 192.168.11.20:45818 -> 127.0.0.1:22222 (proxied) sending 1 bytes failed, error: Connection refused (os error 61)
2024-06-18T23:54:44.36442355Z ERROR tokio-runtime-worker ThreadId(02) shadowsocks_service::local::net::udp::association: udp relay 192.168.11.20:45818 <- ... (proxied) failed, error: Connection refused (os error 61)

Well, the destination address was 127.0.0.1:22222, so it will again generate a loop if a ssserver was actually there.

How can you test for a response UDP packet sent from remote?

[ge9]

Yes, that's the problem on pf, which I don't know how to solve. I was testing UDP behavior with ipfw, not pf.

[zonyitoo]

I just found that pf has a divert-to action, which will preserve the original destination:

https://man.openbsd.org/pf.conf#divert-to

How to configure a rule with divert-to?

[ge9]

No, it's OpenBSD behavior. FreeBSD's divert-to has similar syntax but completely different meaning, as documented in https://man.freebsd.org/cgi/man.cgi?pf.conf(5) . (Also, from my experience on running redsocks in OpenBSD, supporting OpenBSD+divert-to requires a little work on UDP/TCP socket options).

[zonyitoo]

2024-06-19T01:50:51.156486787Z TRACE tokio-runtime-worker ThreadId(07) shadowsocks::relay::udprelay::proxy_socket: UDP server client receive from 192.168.11.186:16700, control: None, packet length 11 bytes, payload length 4 bytes
2024-06-19T01:50:51.165007423Z TRACE tokio-runtime-worker ThreadId(07) shadowsocks_service::local::net::udp::association: udp relay 10.0.2.25:48221 <- 192.168.11.186:16700 (proxied) received 4 bytes
2024-06-19T01:50:51.16521583Z DEBUG tokio-runtime-worker ThreadId(07) shadowsocks::net::sys: IpStackCapability support_ipv4=true
2024-06-19T01:50:51.16532143Z DEBUG tokio-runtime-worker ThreadId(07) shadowsocks::net::sys: IpStackCapability support_ipv6=true
2024-06-19T01:50:51.165420045Z DEBUG tokio-runtime-worker ThreadId(07) shadowsocks::net::sys: IpStackCapability support_ipv4_mapped_ipv6=true
2024-06-19T01:50:51.169426421Z TRACE tokio-runtime-worker ThreadId(07) shadowsocks_service::local::redir::udprelay: udp redir send back data 4 bytes, remote: 192.168.11.186:16700, peer: [::ffff:10.0.2.25]:48221, socket_opts: RedirSocketOpts
2024-06-19T01:50:51.169623373Z TRACE tokio-runtime-worker ThreadId(07) shadowsocks_service::local::net::udp::association: udp relay 10.0.2.25:48221 <- 192.168.11.186:16700 (proxied) with 4 bytes

Just tested on FreeBSD with ipfw. UDP works successfully if: sysctl net.inet6.ip6.v6only=0.

If it is 1:

2024-06-19T01:53:32.434992296Z DEBUG tokio-runtime-worker ThreadId(03) shadowsocks::net::sys: IpStackCapability support_ipv4=true
2024-06-19T01:53:32.43515321Z DEBUG tokio-runtime-worker ThreadId(03) shadowsocks::net::sys: IpStackCapability support_ipv6=true
2024-06-19T01:53:32.435340943Z DEBUG tokio-runtime-worker ThreadId(03) shadowsocks::net::sys: IpStackCapability support_ipv4_mapped_ipv6=true
2024-06-19T01:54:33.632720219Z TRACE tokio-runtime-worker ThreadId(03) shadowsocks::relay::udprelay::proxy_socket: UDP server client receive from 192.168.11.186:16700, control: None, packet length 12 bytes, payload length 5 bytes
2024-06-19T01:54:33.632885883Z TRACE tokio-runtime-worker ThreadId(03) shadowsocks_service::local::net::udp::association: udp relay 10.0.2.25:53661 <- 192.168.11.186:16700 (proxied) received 5 bytes
2024-06-19T01:54:33.633137591Z TRACE tokio-runtime-worker ThreadId(03) shadowsocks_service::local::redir::udprelay: udp redir send back data 5 bytes, remote: 192.168.11.186:16700, peer: [::ffff:10.0.2.25]:53661, socket_opts: RedirSocketOpts
2024-06-19T01:54:33.633245146Z TRACE tokio-runtime-worker ThreadId(03) shadowsocks_service::local::net::udp::association: udp relay 10.0.2.25:53661 <- 192.168.11.186:16700 (proxied) with 5 bytes

No errors was shown. But the client couldn't receive the response.

In the latter case, I think net.inet6.ip6.v6only=1 will disable IPv4-mapped IPv6 IP packets routing to IPv4 clients.

[zonyitoo]

With the test just did in Python:

import socket
s = socket.socket(socket.AF_INET6, socket.SOCK_DGRAM, 0)
s.setsockopt(socket.IPPROTO_IPV6, socket.IPV6_V6ONLY, 0)
s.connect(('::ffff:169.254.1.1', 53))
print(s.getsockname())

When net.inet6.ip6.v6only=1:

Script will always success without any errors. The line of setsockopt(IPV6_V6ONLY) could be removed without affecting the result.

If IPV6_V6ONLY was set to 1, connect() will fail with EINVAL, which is expected.

When net.inet6.ip6.v6only=0:

Behavior was exactly the same.

So there is no way to detect with socket, we may have to read sysctl in code.

[database64128]

The behavior seems quite wrong. Maybe we should let FreeBSD developers know about it.

[Orum]

They offer a convenient way to do that.

In any case, perhaps this issue report on shadowsocks should be split into two separate issues, one for the original issue of using transparent proxying with pf, and a new one for the IPv4/v6 issue.

[zonyitoo]

True. But currently the topic about IPv4-mapped IPv6 support was already come to a conclusion. We should focus on how to support UDP with pf.

[ge9]

The IPv4/v6 issue seems to be fixed now? By the way, I added support for OpenBSD/pf by doing the same thing as I did for redsocks. https://github.com/shadowsocks/shadowsocks-rust/commit/3224af86de450379cbde4975459f292c8789bcee I'll make PR if it's OK.

[zonyitoo]

@ge9 Of course, please make a PR.

[zonyitoo]

Here is an issue listed all known ways to make transparent proxy: https://github.com/2EXP/2exp.github.io/issues/2 .

pfioc_states seems to work on macOS, but some fields are missing on FreeBSD.

[zonyitoo]

https://man.freebsd.org/cgi/man.cgi?query=pf&sektion=4&manpath=OpenBSD

DIOCNATLOOK struct pfioc_natlook    *pnl
           Look up a state table entry by source and destination addresses
           and ports.

           struct pfioc_natlook {
               struct pf_addr   saddr;
               struct pf_addr   daddr;
               struct pf_addr   rsaddr;
               struct pf_addr   rdaddr;
               u_int16_t    rdomain;
               u_int16_t    rrdomain;
               u_int16_t    sport;
               u_int16_t    dport;
               u_int16_t    rsport;
               u_int16_t    rdport;
               sa_family_t  af;
               u_int8_t     proto;
               u_int8_t     direction;
           };

           This  was  primarily  used  to support transparent proxies with
           rdr-to rules.  New proxies should use divert-to rules  instead.
           These  do  not  require access to the privileged /dev/pf device
           and   preserve   the   original   destination    address    for
           [getsockname(2)](https://man.freebsd.org/cgi/man.cgi?query=getsockname&sektion=2&apropos=0&manpath=OpenBSD+7.5).  For  SOCK_DGRAM sockets, the [ip(4)](https://man.freebsd.org/cgi/man.cgi?query=ip&sektion=4&apropos=0&manpath=OpenBSD+7.5) socket op-
           tions IP_RECVDSTADDR and IP_RECVDSTPORT can be used to retrieve
           the destination address and port.

In the document it said IP_RECVDSTADDR and IP_RECVDSTPORT could be used to retrive the destination address and port.

https://github.com/shadowsocks/shadowsocks-rust/blob/8b32d8509ce42763e891f55e7b12d63a223e3908/crates/shadowsocks-service/src/local/redir/udprelay/sys/unix/bsd.rs#L296-L338

But it doesn't work, right? @Orum

       If the IP_RECVDSTADDR option is enabled on  a  SOCK_DGRAM  socket,  the
       [recvmsg(2)](https://man.freebsd.org/cgi/man.cgi?query=recvmsg&sektion=2&apropos=0&manpath=OpenBSD+7.5)  call    will return the destination IP address for a UDP data-
       gram.  The msg_control field in the msghdr structure points to a buffer
       that contains a cmsghdr structure followed  by  the  IP  address.   The
       cmsghdr fields have the following values:

         cmsg_len = CMSG_LEN(sizeof(struct in_addr))
         cmsg_level = IPPROTO_IP
         cmsg_type = IP_RECVDSTADDR

       If  the  IP_RECVDSTPORT  option  is enabled on a SOCK_DGRAM socket, the
       [recvmsg(2)](https://man.freebsd.org/cgi/man.cgi?query=recvmsg&sektion=2&apropos=0&manpath=OpenBSD+7.5) call will return the destination port    for  a  UDP  datagram.
       The  msg_control  field in the msghdr structure points to a buffer that
       contains a cmsghdr structure followed by the  port  in  16-bit  network
       byte order.  The cmsghdr fields have the following values:

         cmsg_len = CMSG_LEN(sizeof(u_int16_t))
         cmsg_level = IPPROTO_IP
         cmsg_type = IP_RECVDSTPORT

Implementation detail: https://reviews.freebsd.org/D9235

According to this implementation, IP_RECVDSTADDR will returns sockaddr_in with address and port, so IP_RECVDSTPORT is not useful in IPv4. IPv6 implementation only has IPV6_RECVDSTADDR and no IPV6_RECVDSTPORT. But it is different from the document (manpage), which said IP_RECVDSTADDR returns in_addr instead of sockaddr_in.

[zonyitoo]

https://github.com/freebsd/freebsd-src/blob/bbecd3148abf68918b1aa5fc7750dd8ec17fea72/sys/netinet/udp_usrreq.c#L289-L300

It is sockaddr_in. So the man doc is wrong?

[ge9]

I didn't know FreeBSD's man page describes UDP transparent proxy treatments. That seems much like OpenBSD's behavior. I may try it.

[zonyitoo]

All the references are from FreeBSD's manpage.

[ge9]

I tried using IP_RECVDSTADDR in FreeBSD but obtained IP was still 127.0.0.1. No difference from IP_ORIGDSTADDR. We can enable both at once and only gain 127.0.0.1 from both...🤔 Also, IP_RECVDSTPORT seems not exist in FreeBSD.

[zonyitoo]

IP_RECVORIGDSTADDR = IP_ORIGDSTADDR, they have the same value.

but obtained IP was still 127.0.0.1

I think the next step is to findout where the value of udp_in[1] is from.

[ge9]

IP_RECVORIGDSTADDR equals IP_ORIGDSTADDR, but IP_RECVDSTADDR is different. Indeed, messages received by recvmsg() are slightly different, but both include 127.0.0.1.`

[zonyitoo]

https://github.com/freebsd/freebsd-src/blob/bbecd3148abf68918b1aa5fc7750dd8ec17fea72/sys/netinet/udp_usrreq.c#L493-L505

udp_in[1] is IP packet's destination address. So I think the next step is: how to let pf send packets to this port without modifying its original destinaion address.

[zonyitoo]

Why we cannot obtain the original destination address from DIOCNATLOOK? If we set a NAT rule for UDP, can we make it work?