Transparently proxying UDP traffic with pf?

[Orum]

This is a similar symptom to my earlier issue (#1473), but as the circumstances have drastically changed I thought I'd open a new issue. To summarize the changes, I am no longer running sslocal on the host (Linux/iptables) machine; it's instead being run on my FreeBSD router which uses pf.

Obviously, this requires different firewall rules to work, but there is little if any documentation I could find for transparently proxying via pf within the shadowsocks-rust project. What I've done here is largely guesswork, but this is the rule I came up with to transparently proxy both TCP & UDP to sslocal (and then on to ssserver): rdr on $int_if inet proto { tcp, udp } to !<private> -> 127.0.0.1 port 1080

sslocal is being run on the router via the following command: sslocal -b 127.0.0.1:1080 -U --protocol redir -s 192.168.x.y:8388 -m none --tcp-redir pf --udp-redir pf

...and finally, ssserver is run with -U -m none -b 192.168.x.y:8388

This "works" in the sense that both TCP & UDP traffic are being redirected to sslocal (unlike in #1473 where only TCP traffic was ever reaching sslocal), which is then sending the traffic on to sserver. However, once again, only TCP traffic is being fully proxied correctly.

UDP traffic does arrive at sslocal without any doubt, as it shows up when running sslocal with -vvv like so (timestamps removed for brevity, and the host system's address being substituted here with simply <host>):

TRACE  tokio-runtime-worker ThreadId(05) shadowsocks::relay::udprelay::proxy_socket: crates/shadowsocks/src/relay/udprelay/proxy_socket.rs:235: UDP server client send to 127.0.0.1:1080, control: UdpSocketControlData { client_session_id: 7043831170210357480, server_session_id: 0, packet_id: 1, user: None }, payload length 96 bytes, packet length 103 bytes
TRACE  tokio-runtime-worker ThreadId(05) shadowsocks_service::local::redir::udprelay: crates/shadowsocks-service/src/local/redir/udprelay/mod.rs:307: received UDP packet from <host>:44849, destination 127.0.0.1:1080, length 96 bytes
TRACE  tokio-runtime-worker ThreadId(05) shadowsocks_service::local::net::udp::association: crates/shadowsocks-service/src/local/net/udp/association.rs:433: udp relay <host>:44849 -> 127.0.0.1:1080 (proxied) with 96 bytes

This traffic is then forwarded on to ssserver, which I can verify by watching outgoing traffic on the router via tcpdump. After they arrive at ssserver though, they fail to reach the internet, so clearly something isn't configured correctly.

My best guess as to what is happening here, based on the logs, is that sslocal thinks the destination of the UDP packet is 127.0.0.1 (i.e. the address that sslocal is running on and where incoming UDP traffic on the router is redirected to), and fails to retrieve the original destination address before pf's rdr rule took effect. This wouldn't surprise me as my rdr rule is complete guesswork, so I assume I'm missing something.

If that is indeed the problem, what should the rdr rule look like to transparently proxy UDP traffic with pf? Or am I barking up the wrong tree, and is the problem due to something else entirely?

[zonyitoo]

I have never tested on FreeBSD, so this project supports FreeBSD is still "theoretically".

I have tested pf on macOS, which supports TCP perfectly just like you did on FreeBSD. But on macOS, UDP's destination must be retrieved from ioc_getstates. This API is far different from FreeBSD's definition, so I think FreeBSD doesn't use the same mechanism to get UDP packets' destination when redirected by pf.

In your last PR, you said on FreeBSD UDP's destination could be retrieved from msg_name returned by recvfrom. Is that correct? I couldn't see any document from FreeBSD talking about it.

The 127.0.0.1:1080 in your case was exactly the value of msg_name returned from recv_dest_from.

[Orum]

In your last PR, you said on FreeBSD UDP's destination could be retrieved from msg_name returned by recvfrom. Is that correct? I couldn't see any document from FreeBSD talking about it.

That wasn't me, so you must be thinking of someone else. While I know how to write rules for pf fairly well, I've never programmed anything to interface with it, so I'm unsure of how the API is supposed to be used.

The good news is that you can access the man pages online. There's also a forum, mailing lists, and a Discord server, all of which feature some form of development/programming discussion.

I'm interested in learning as well, but I'm still learning rust, so it may be some time until I am of use.

[zonyitoo]

Glad to know that you are familiar with FreeBSD and pf. Could you find any doc / references about how to get the "original destination address" of UDP packets?

On Linux, it is quite straight forward: https://man7.org/linux/man-pages/man7/ip.7.html (IP_RECVORIGDSTADDR).

On macOS, I got it from another post: (?) Deleted.

A famous tools mitmproxy only supports TCP on FreeBSD: https://docs.mitmproxy.org/stable/howto-transparent/ .

So it is quite hard to find a reference about how FreeBSD handles UDP redirects and how to get the original destination address programmatically.

[zonyitoo]

And you could also try the ipfw: https://forums.freebsd.org/threads/retrieving-the-destination-port-in-an-ipfwed-udp-packet.37545/

[zonyitoo]

@Orum Do you still interest in making this Project working on FreeBSD? What's the current status?

[Orum]

I am still very interested in making it work. However, I think the best way to do so at this point is to just read through FreeBSD's source code. As such, I'm familiarizing myself with it, but it's going to take some time as I have other obligations and projects demanding my attention, so it's on the back burner for me right now. This is probably the best, or at least, surest way to figure out how to get the address (assuming it's even possible at present) from pf.

Additionally, I'd need to learn much more about rust before I contribute anything other than information to this project, as my knowledge of it is spotty at best. But, if I do figure out the FreeBSD side of the picture, perhaps someone here can handle that side of things once they have the information. I'm also not only willing, but happy to test things to see if they work once they're implemented.

[ge9]

https://github.com/semigodking/redsocks/issues/200 FYI, I succeeded in running transparent proxy in FreeBSD with redsocks+SOCKS5 (sorry they are completely unrelated to shadowsocks) and ipfw, but not with pf. When I tested with pf, the UDP destination address was obtained as the transparent proxy's listen address, just as @Orum mentioned. As I mentioned in the link, in OpenBSD, where pf originates from, using divert-to rule of pf worked perfectly. divert rule seems not ported to FreeBSD's pf.

[Orum]

FreeBSD and OpenBSD diverged in their rules syntax quite a few years back when the latter made some changes that broke older rule sets, IIRC. I've stuck with FreeBSD, which might have an equivalent of divert-to (or not), but I don't really know what it does as I never bothered to keep up with OBSD's syntax as I don't use it.

I also don't really want to switch to ipfw, or any other firewall, as I really value simplicity and clarity of pf's rules' syntax. So many other firewalls are hard to decipher, which is undesirable to say the least when it comes to security.

In any case I will try and dedicate some time to reading through and understanding the pf/bpf code in the future, but that will be some time from now.

[zonyitoo]

@ge9 Have you tried shadowsocks-rust with ipfw? What problem did you see?

[ge9]

I tested it and it worked, but there were a problem. shadowsocks-rust uses IPv4-mapped IPv6 address to return message to IPv4 clients: https://github.com/shadowsocks/shadowsocks-rust/blob/ac67a08f87afe54e844241fb75d8bfc018a0bd03/crates/shadowsocks-service/src/local/redir/udprelay/mod.rs#L199 However, FreeBSD disables IPv4-mapped IPv6 address by default (according to https://github.com/scala-native/scala-native/issues/3630). If I set sysctl net.inet6.ip6.v6only=0, then it worked. Except for this, normal transparent proxy setting will work.

FYI, My settings are following:

• Initial setup (the machine already had 10.0.2.15/24)

  #!/bin/sh
  kldload ipfw
  fwcmd=ipfw
  ifconfig em0 alias 10.0.2.25 netmask 0xffffff00
  $fwcmd add 100 allow all from any to any via lo0
  $fwcmd add 500 fwd 127.0.0.1,22222 tcp from 10.0.2.25 to any
  $fwcmd add 600 fwd 127.0.0.1,22222 udp from 10.0.2.25 to any
  $fwcmd add 700 allow ip from any to any

• sslocal -b 127.0.0.1:22222 -U --protocol redir -s 192.168.1.7:1080 --tcp-redir pf --udp-redir pf -m none
  • (specifying pf seems fine)
• ssserver -U -s 192.168.1.7:1080 -m none (on another PC)
  • note that specifying 0.0.0.0:1080 here doesn't work

[zonyitoo]

There is a SUPPORT_IPV6_TRANSPARENT flag that indicates whether the current platform supports IPv6, if it is true, then shadowsocks-rust will always convert IPv4 to IPv4-mapped-IPv6 and uses only one socket for sending back.

This is always working for Linux, because dual-stack socket was enabled by default for most distribution of Linux.

Is there a way to test if dual-stack is enabled on FreeBSD? By reading sysctl net.inet6.ip6.v6only=0 should be one possible option.

[zonyitoo]

Could you please help to run the test program on FreeBSD and see if any of these syscalls returned errors? https://stackoverflow.com/questions/30184377/how-to-detect-if-dual-stack-socket-is-supported

[zonyitoo]

cc @madeye , FreeBSD doesn't support IPv4-mapped-IPv6 by default. So my previous (years ago) PR that uses IPv6 sockets for sending back UDP packets won't be able to run properly on FreeBSD.

[ge9]

This is the result of the python test program.

Traceback (most recent call last):
  File "/root/test.py", line 4, in <module>
    s.connect(('::ffff:169.254.1.1', 53))
OSError: [Errno 22] Invalid argument

If I set sysctl net.inet6.ip6.v6only=0 and commented out s.setsockopt(socket.IPPROTO_IPV6, socket.IPV6_V6ONLY, 1), it worked. Otherwise, it failed.

In the shadowsocks, this is the error.

2024-06-17T03:45:10.334750916+09:00  WARN udp failed to send back 68 bytes to client 10.0.2.25:44550, from target 3.132.228.249:3478 (proxied), error: Invalid argument (os error 22)

This originates from https://github.com/shadowsocks/shadowsocks-rust/blob/ac67a08f87afe54e844241fb75d8bfc018a0bd03/crates/shadowsocks-service/src/local/redir/udprelay/mod.rs#L203 .

[zonyitoo]

Please test if it is Ok without settingnet.inet6.ip6.v6only=0 manually.

[zonyitoo]

I just found a better solution. Working on it.

[zonyitoo]

Tested on OpenWRT. Please help testing it if it is working correctly on FreeBSD. @ge9

[ge9]

5ba8b7d worked but 765c9e5 or cd25d25 (latest) doesn't (same error).

[zonyitoo]

Please run again with -v and check these debug logs:

https://github.com/shadowsocks/shadowsocks-rust/blob/cd25d25b22a3a9b1fa5805e1482826e6214f797a/crates/shadowsocks/src/net/sys/mod.rs#L143-L181

Are they all true?

[ge9]

here is the log. seems all true.

2024-06-17T06:38:52.894052461+09:00 TRACE  tokio-runtime-worker ThreadId(02) shadowsocks_service::local::redir::udprelay: crates/shadowsocks-service/src/local/redir/udprelay/mod.rs:285: received UDP packet from 10.0.2.25:65158, destination 3.132.228.249:3478, length 28 bytes
2024-06-17T06:38:52.894445849+09:00 DEBUG  tokio-runtime-worker ThreadId(02) shadowsocks_service::local::net::udp::association: crates/shadowsocks-service/src/local/net/udp/association.rs:123: created udp association for 10.0.2.25:65158
2024-06-17T06:38:52.894817377+09:00 TRACE  tokio-runtime-worker ThreadId(02) shadowsocks_service::local::net::udp::association: crates/shadowsocks-service/src/local/net/udp/association.rs:433: udp relay 10.0.2.25:65158 -> 3.132.228.249:3478 (proxied) with 28 bytes
2024-06-17T06:38:52.894896617+09:00 TRACE  tokio-runtime-worker ThreadId(02) mio::poll: /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/mio-0.8.11/src/poll.rs:551: registering event source with poller: token=Token(79171555479936), interests=READABLE | WRITABLE
2024-06-17T06:38:52.895050148+09:00 TRACE  tokio-runtime-worker ThreadId(02) shadowsocks::relay::udprelay::proxy_socket: crates/shadowsocks/src/relay/udprelay/proxy_socket.rs:103: connected udp remote 192.168.1.8:1080 (outbound: 192.168.1.8:1080) with ConnectOpts { user_cookie: None, bind_local_addr: None, bind_interface: None, tcp: TcpSocketOpts { send_buffer_size: None, recv_buffer_size: None, nodelay: false, fastopen: false, keepalive: Some(15s), mptcp: false }, udp: UdpSocketOpts { mtu: None } }
2024-06-17T06:38:52.895129216+09:00 TRACE  tokio-runtime-worker ThreadId(02) shadowsocks::relay::udprelay::proxy_socket: crates/shadowsocks/src/relay/udprelay/proxy_socket.rs:235: UDP server client send to 3.132.228.249:3478, control: UdpSocketControlData { client_session_id: 786820869274048462, server_session_id: 0, packet_id: 1, user: None }, payload length 28 bytes, packet length 35 bytes
2024-06-17T06:38:53.135364801+09:00 TRACE  tokio-runtime-worker ThreadId(02) shadowsocks::relay::udprelay::proxy_socket: crates/shadowsocks/src/relay/udprelay/proxy_socket.rs:445: UDP server client receive from 3.132.228.249:3478, control: None, packet length 75 bytes, payload length 68 bytes
2024-06-17T06:38:53.136228332+09:00 TRACE  tokio-runtime-worker ThreadId(02) shadowsocks_service::local::net::udp::association: crates/shadowsocks-service/src/local/net/udp/association.rs:607: udp relay 10.0.2.25:65158 <- 3.132.228.249:3478 (proxied) received 68 bytes
2024-06-17T06:38:53.136421625+09:00 DEBUG  tokio-runtime-worker ThreadId(02) shadowsocks::net::sys: crates/shadowsocks/src/net/sys/mod.rs:155: IpStackCapability support_ipv4=true
2024-06-17T06:38:53.136798744+09:00 DEBUG  tokio-runtime-worker ThreadId(02) shadowsocks::net::sys: crates/shadowsocks/src/net/sys/mod.rs:164: IpStackCapability support_ipv6=true
2024-06-17T06:38:53.137187564+09:00 DEBUG  tokio-runtime-worker ThreadId(02) shadowsocks::net::sys: crates/shadowsocks/src/net/sys/mod.rs:175: IpStackCapability support_ipv4_mapped_ipv6=true
2024-06-17T06:38:53.13739486+09:00 TRACE  tokio-runtime-worker ThreadId(02) mio::poll: /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/mio-0.8.11/src/poll.rs:551: registering event source with poller: token=Token(79171555480192), interests=READABLE | WRITABLE
2024-06-17T06:38:53.137685127+09:00  WARN  tokio-runtime-worker ThreadId(02) shadowsocks_service::local::net::udp::association: crates/shadowsocks-service/src/local/net/udp/association.rs:620: udp failed to send back 68 bytes to client 10.0.2.25:65158, from target 3.132.228.249:3478 (proxied), error: Invalid argument (os error 22)

[zonyitoo]

Well, so FreeBSD allows bind()ing a IPv4-mapped-IPv6 address, but doesn’t allow sendmsg to it. Programs that are written in Go should also fails with the same reason.

[zonyitoo]

Call connect() in this commit. Please help verify it again.

[ge9]

Hmm it seems still not working...

2024-06-16T23:01:19.392825484Z TRACE  tokio-runtime-worker ThreadId(03) shadowsocks_service::local::redir::udprelay: crates/shadowsocks-service/src/local/redir/udprelay/mod.rs:285: received UDP packet from 10.0.2.25:13870, destination 3.132.228.249:3478, length 28 bytes
2024-06-16T23:01:19.393494564Z DEBUG  tokio-runtime-worker ThreadId(03) shadowsocks_service::local::net::udp::association: crates/shadowsocks-service/src/local/net/udp/association.rs:123: created udp association for 10.0.2.25:13870
2024-06-16T23:01:19.393869192Z TRACE  tokio-runtime-worker ThreadId(03) shadowsocks_service::local::net::udp::association: crates/shadowsocks-service/src/local/net/udp/association.rs:433: udp relay 10.0.2.25:13870 -> 3.132.228.249:3478 (proxied) with 28 bytes
2024-06-16T23:01:19.394037649Z TRACE  tokio-runtime-worker ThreadId(03) mio::poll: /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/mio-0.8.11/src/poll.rs:551: registering event source with poller: token=Token(88188202799232), interests=READABLE | WRITABLE
2024-06-16T23:01:19.394211694Z TRACE  tokio-runtime-worker ThreadId(03) shadowsocks::relay::udprelay::proxy_socket: crates/shadowsocks/src/relay/udprelay/proxy_socket.rs:103: connected udp remote 192.168.1.212:1080 (outbound: 192.168.1.212:1080) with ConnectOpts { user_cookie: None, bind_local_addr: None, bind_interface: None, tcp: TcpSocketOpts { send_buffer_size: None, recv_buffer_size: None, nodelay: false, fastopen: false, keepalive: Some(15s), mptcp: false }, udp: UdpSocketOpts { mtu: None } }
2024-06-16T23:01:19.394416748Z TRACE  tokio-runtime-worker ThreadId(03) shadowsocks::relay::udprelay::proxy_socket: crates/shadowsocks/src/relay/udprelay/proxy_socket.rs:235: UDP server client send to 3.132.228.249:3478, control: UdpSocketControlData { client_session_id: 8860848107368566370, server_session_id: 0, packet_id: 1, user: None }, payload length 28 bytes, packet length 35 bytes
2024-06-16T23:01:19.566423525Z TRACE  tokio-runtime-worker ThreadId(03) shadowsocks::relay::udprelay::proxy_socket: crates/shadowsocks/src/relay/udprelay/proxy_socket.rs:445: UDP server client receive from 3.132.228.249:3478, control: None, packet length 75 bytes, payload length 68 bytes
2024-06-16T23:01:19.567810852Z TRACE  tokio-runtime-worker ThreadId(03) shadowsocks_service::local::net::udp::association: crates/shadowsocks-service/src/local/net/udp/association.rs:607: udp relay 10.0.2.25:13870 <- 3.132.228.249:3478 (proxied) received 68 bytes
2024-06-16T23:01:19.568463729Z DEBUG  tokio-runtime-worker ThreadId(03) shadowsocks::net::sys: crates/shadowsocks/src/net/sys/mod.rs:156: IpStackCapability support_ipv4=true
2024-06-16T23:01:19.568883894Z DEBUG  tokio-runtime-worker ThreadId(03) shadowsocks::net::sys: crates/shadowsocks/src/net/sys/mod.rs:165: IpStackCapability support_ipv6=true
2024-06-16T23:01:19.56980524Z DEBUG  tokio-runtime-worker ThreadId(03) shadowsocks::net::sys: crates/shadowsocks/src/net/sys/mod.rs:173: IpStackCapability support_ipv4_mapped_ipv6=true
2024-06-16T23:01:19.569994091Z TRACE  tokio-runtime-worker ThreadId(03) mio::poll: /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/mio-0.8.11/src/poll.rs:551: registering event source with poller: token=Token(88188202799488), interests=READABLE | WRITABLE
2024-06-16T23:01:19.578334536Z  WARN  tokio-runtime-worker ThreadId(03) shadowsocks_service::local::net::udp::association: crates/shadowsocks-service/src/local/net/udp/association.rs:620: udp failed to send back 68 bytes to client 10.0.2.25:13870, from target 3.132.228.249:3478 (proxied), error: Invalid argument (os error 22)

[zonyitoo]

Replace it with the method just like the one on stackoverflow. Please try again. @ge9

[ge9]

Still not working...

2024-06-16T23:53:16.161455017Z TRACE  tokio-runtime-worker ThreadId(02) shadowsocks_service::local::redir::udprelay: crates/shadowsocks-service/src/local/redir/udprelay/mod.rs:285: received UDP packet from 10.0.2.25:34707, destination 3.132.228.249:3478, length 28 bytes
2024-06-16T23:53:16.161934966Z DEBUG  tokio-runtime-worker ThreadId(02) shadowsocks_service::local::net::udp::association: crates/shadowsocks-service/src/local/net/udp/association.rs:123: created udp association for 10.0.2.25:34707
2024-06-16T23:53:16.162531969Z TRACE  tokio-runtime-worker ThreadId(02) shadowsocks_service::local::net::udp::association: crates/shadowsocks-service/src/local/net/udp/association.rs:433: udp relay 10.0.2.25:34707 -> 3.132.228.249:3478 (proxied) with 28 bytes
2024-06-16T23:53:16.162692883Z TRACE  tokio-runtime-worker ThreadId(02) mio::poll: /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/mio-0.8.11/src/poll.rs:551: registering event source with poller: token=Token(51249433133440), interests=READABLE | WRITABLE
2024-06-16T23:53:16.162828655Z TRACE  tokio-runtime-worker ThreadId(02) shadowsocks::relay::udprelay::proxy_socket: crates/shadowsocks/src/relay/udprelay/proxy_socket.rs:103: connected udp remote 192.168.1.212:1080 (outbound: 192.168.1.212:1080) with ConnectOpts { user_cookie: None, bind_local_addr: None, bind_interface: None, tcp: TcpSocketOpts { send_buffer_size: None, recv_buffer_size: None, nodelay: false, fastopen: false, keepalive: Some(15s), mptcp: false }, udp: UdpSocketOpts { mtu: None } }
2024-06-16T23:53:16.16302868Z TRACE  tokio-runtime-worker ThreadId(02) shadowsocks::relay::udprelay::proxy_socket: crates/shadowsocks/src/relay/udprelay/proxy_socket.rs:235: UDP server client send to 3.132.228.249:3478, control: UdpSocketControlData { client_session_id: 17626901403037311660, server_session_id: 0, packet_id: 1, user: None }, payload length 28 bytes, packet length 35 bytes
2024-06-16T23:53:16.356517518Z TRACE  tokio-runtime-worker ThreadId(02) shadowsocks::relay::udprelay::proxy_socket: crates/shadowsocks/src/relay/udprelay/proxy_socket.rs:445: UDP server client receive from 3.132.228.249:3478, control: None, packet length 75 bytes, payload length 68 bytes
2024-06-16T23:53:16.357477975Z TRACE  tokio-runtime-worker ThreadId(02) shadowsocks_service::local::net::udp::association: crates/shadowsocks-service/src/local/net/udp/association.rs:607: udp relay 10.0.2.25:34707 <- 3.132.228.249:3478 (proxied) received 68 bytes
2024-06-16T23:53:16.358022457Z DEBUG  tokio-runtime-worker ThreadId(02) shadowsocks::net::sys: crates/shadowsocks/src/net/sys/mod.rs:155: IpStackCapability support_ipv4=true
2024-06-16T23:53:16.358794623Z DEBUG  tokio-runtime-worker ThreadId(02) shadowsocks::net::sys: crates/shadowsocks/src/net/sys/mod.rs:164: IpStackCapability support_ipv6=true
2024-06-16T23:53:16.359158356Z DEBUG  tokio-runtime-worker ThreadId(02) shadowsocks::net::sys: crates/shadowsocks/src/net/sys/mod.rs:172: IpStackCapability support_ipv4_mapped_ipv6=true
2024-06-16T23:53:16.361218674Z TRACE  tokio-runtime-worker ThreadId(02) mio::poll: /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/mio-0.8.11/src/poll.rs:551: registering event source with poller: token=Token(51249433133696), interests=READABLE | WRITABLE
2024-06-16T23:53:16.362913582Z  WARN  tokio-runtime-worker ThreadId(02) shadowsocks_service::local::net::udp::association: crates/shadowsocks-service/src/local/net/udp/association.rs:620: udp failed to send back 68 bytes to client 10.0.2.25:34707, from target 3.132.228.249:3478 (proxied), error: Invalid argument (os error 22)

[zonyitoo]

Udp Socket bind([::ffff:3.132.228.249]:3478) was Ok, but then sendmsg([::ffff:10.0.2.25]:34707) failed?

Is that a bug in FreeBSD? :(

Further more, in the test code, sockets bind([::ffff:127.0.0.1]:0) and connect([::ffff:127.0.0.1]:53) were all working well.

For TCP socket, bind([::ffff:127.0.0.1]:0) and then listen() Ok. For client socket, bind([::ffff:127.0.0.1]:0) and then connect() to the server socket Ok.

Getting Headache.

[ge9]

(edited)

By the way, I forgot to test TCP. With ipfw, --udp-redir pf is fine, but --tcp-redir pf did not work. And TCP seems not affected by this IPv4-mapped address issue (it works fine with ipfw and --tcp-redir ipfw).

[zonyitoo]

UDP works with pf but TCP doesn't? That wasn't expected.

[ge9]

It's about testing with ipfw, not pf.

[zonyitoo]

Ok. Please also test the TCP with ipfw. Let's fix all of them together.

[ge9]

I think it's OK that --tcp-redir pf+ipfw doesn't work, because --tcp-redir pf introduces special treatment for redirected TCP packets.

[ge9]

I'm not so familiar with the detail of implementation, but is it not possible to always use IPv4 address for IPv4 clients? (i.e. completely exclude IPv4-mapped IPv6 address)?

[zonyitoo]

Of course it is Ok. If the support_ipv4_mapped_ipv6 is false in the IpStackCapabilities, redir will create different sockets for IPv4 and IPv6. But now the problem is why it is always true.

[ge9]

In my understanding, FreeBSD does support IPv4-mapped addresses, but intentionally disables it with security reason. (according to the link). So, it may not be the problem of capabilities.

[zonyitoo]

IPv4-mapped IPv6 are handled right here:

https://github.com/freebsd/freebsd-src/blob/fc3907ce001ec95ff4f32861bfa84f81c268d179/sys/netinet6/udp6_usrreq.c#L743-L780

It is actually bypasses it to IPv4 implementation.

[zonyitoo]

Was that if ((inp->inp_flags & IN6P_IPV6_V6ONLY) == 0) equal to the sysctl net.inet6.ip6.v6only=0?

[zonyitoo]

For bind, it should always success. Because if the sysctl was not 0, nothing would actually happened.

https://github.com/freebsd/freebsd-src/blob/fc3907ce001ec95ff4f32861bfa84f81c268d179/sys/netinet6/udp6_usrreq.c#L1051-L1067

[zonyitoo]

https://github.com/freebsd/freebsd-src/blob/fc3907ce001ec95ff4f32861bfa84f81c268d179/sys/netinet6/sctp6_usrreq.c#L811-L828

connect() for TCP should also reject IPv4-mapped addresses. Hmm, why.

[zonyitoo]

https://github.com/freebsd/freebsd-src/blob/fc3907ce001ec95ff4f32861bfa84f81c268d179/sys/netinet6/sctp6_usrreq.c#L549-L571

It should also reject in bind(). WHY?? WHY??

Is that because we have set IPV6_V6ONLY with setsockopt? If so, let me try to do that in redir's send-back socket.

[zonyitoo]

Morning. Please help testing if this commit working in FreeBSD. @ge9

[ge9]

I'm afraid that's still not working.

This is UDP log,

2024-06-17T11:58:24.450931472Z TRACE  tokio-runtime-worker ThreadId(02) shadowsocks_service::local::redir::udprelay: crates/shadowsocks-service/src/local/redir/udprelay/mod.rs:285: received UDP packet from 10.0.2.25:11770, destination 3.132.228.249:3478, length 28 bytes
2024-06-17T11:58:24.451373707Z DEBUG  tokio-runtime-worker ThreadId(02) shadowsocks_service::local::net::udp::association: crates/shadowsocks-service/src/local/net/udp/association.rs:123: created udp association for 10.0.2.25:11770
2024-06-17T11:58:24.451986355Z TRACE  tokio-runtime-worker ThreadId(02) shadowsocks_service::local::net::udp::association: crates/shadowsocks-service/src/local/net/udp/association.rs:433: udp relay 10.0.2.25:11770 -> 3.132.228.249:3478 (proxied) with 28 bytes
2024-06-17T11:58:24.452117657Z TRACE  tokio-runtime-worker ThreadId(02) mio::poll: /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/mio-0.8.11/src/poll.rs:551: registering event source with poller: token=Token(79307137466752), interests=READABLE | WRITABLE
2024-06-17T11:58:24.452228565Z TRACE  tokio-runtime-worker ThreadId(02) shadowsocks::relay::udprelay::proxy_socket: crates/shadowsocks/src/relay/udprelay/proxy_socket.rs:103: connected udp remote 192.168.1.212:1080 (outbound: 192.168.1.212:1080) with ConnectOpts { user_cookie: None, bind_local_addr: None, bind_interface: None, tcp: TcpSocketOpts { send_buffer_size: None, recv_buffer_size: None, nodelay: false, fastopen: false, keepalive: Some(15s), mptcp: false }, udp: UdpSocketOpts { mtu: None } }
2024-06-17T11:58:24.452379981Z TRACE  tokio-runtime-worker ThreadId(02) shadowsocks::relay::udprelay::proxy_socket: crates/shadowsocks/src/relay/udprelay/proxy_socket.rs:235: UDP server client send to 3.132.228.249:3478, control: UdpSocketControlData { client_session_id: 18020449999949013623, server_session_id: 0, packet_id: 1, user: None }, payload length 28 bytes, packet length 35 bytes
2024-06-17T11:58:24.624516942Z TRACE  tokio-runtime-worker ThreadId(02) shadowsocks::relay::udprelay::proxy_socket: crates/shadowsocks/src/relay/udprelay/proxy_socket.rs:445: UDP server client receive from 3.132.228.249:3478, control: None, packet length 75 bytes, payload length 68 bytes
2024-06-17T11:58:24.625452815Z TRACE  tokio-runtime-worker ThreadId(02) shadowsocks_service::local::net::udp::association: crates/shadowsocks-service/src/local/net/udp/association.rs:607: udp relay 10.0.2.25:11770 <- 3.132.228.249:3478 (proxied) received 68 bytes
2024-06-17T11:58:24.626366339Z DEBUG  tokio-runtime-worker ThreadId(02) shadowsocks::net::sys: crates/shadowsocks/src/net/sys/mod.rs:155: IpStackCapability support_ipv4=true
2024-06-17T11:58:24.630506251Z DEBUG  tokio-runtime-worker ThreadId(02) shadowsocks::net::sys: crates/shadowsocks/src/net/sys/mod.rs:164: IpStackCapability support_ipv6=true
2024-06-17T11:58:24.632915775Z DEBUG  tokio-runtime-worker ThreadId(02) shadowsocks::net::sys: crates/shadowsocks/src/net/sys/mod.rs:172: IpStackCapability support_ipv4_mapped_ipv6=true
2024-06-17T11:58:24.634410658Z  WARN  tokio-runtime-worker ThreadId(02) shadowsocks_service::local::net::udp::association: crates/shadowsocks-service/src/local/net/udp/association.rs:620: udp failed to send back 68 bytes to client 10.0.2.25:11770, from target 3.132.228.249:3478 (proxied), error: Invalid argument (os error 22)

and this is TCP (successful).

2024-06-17T11:58:48.767080033Z TRACE  tokio-runtime-worker ThreadId(02) mio::poll: /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/mio-0.8.11/src/poll.rs:551: registering event source with poller: token=Token(79307137467008), interests=READABLE | WRITABLE
2024-06-17T11:58:48.767456059Z TRACE  tokio-runtime-worker ThreadId(02) shadowsocks_service::local::redir::tcprelay: crates/shadowsocks-service/src/local/redir/tcprelay/mod.rs:136: got connection 10.0.2.25:22369
2024-06-17T11:58:48.768354218Z TRACE  tokio-runtime-worker ThreadId(02) mio::poll: /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/mio-0.8.11/src/poll.rs:551: registering event source with poller: token=Token(79307137467264), interests=READABLE | WRITABLE
2024-06-17T11:58:48.771289507Z TRACE  tokio-runtime-worker ThreadId(02) shadowsocks::relay::tcprelay::proxy_stream::client: crates/shadowsocks/src/relay/tcprelay/proxy_stream/client.rs:135: connected tcp remote 192.168.1.212:1080 (outbound: 192.168.1.212:1080) with ConnectOpts { user_cookie: None, bind_local_addr: None, bind_interface: None, tcp: TcpSocketOpts { send_buffer_size: None, recv_buffer_size: None, nodelay: false, fastopen: false, keepalive: Some(15s), mptcp: false }, udp: UdpSocketOpts { mtu: None } }
2024-06-17T11:58:48.771878129Z DEBUG  tokio-runtime-worker ThreadId(02) shadowsocks_service::local::utils: crates/shadowsocks-service/src/local/utils.rs:29: established tcp tunnel 10.0.2.25:22369 <-> 3.132.228.249:3478 through sever 192.168.1.212:1080 (outbound: 192.168.1.212:1080)
2024-06-17T11:58:49.291495008Z TRACE  tokio-runtime-worker ThreadId(02) shadowsocks::relay::tcprelay::utils: crates/shadowsocks/src/relay/tcprelay/utils.rs:255: copy bidirection ends, a_to_b: Done(68), b_to_a: Done(0)
2024-06-17T11:58:49.291837509Z TRACE  tokio-runtime-worker ThreadId(02) shadowsocks_service::local::utils: crates/shadowsocks-service/src/local/utils.rs:72: tcp tunnel 10.0.2.25:22369 <-> 3.132.228.249:3478 (proxied) closed, L2R 0 bytes, R2L 68 bytes
2024-06-17T11:58:49.292008481Z TRACE  tokio-runtime-worker ThreadId(02) mio::poll: /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/mio-0.8.11/src/poll.rs:682: deregistering event source from poller
2024-06-17T11:58:49.294304303Z TRACE  tokio-runtime-worker ThreadId(02) mio::poll: /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/mio-0.8.11/src/poll.rs:682: deregistering event source from poller

[ge9]

By the way, testing on my FreeBSD is totally fine for me, but if you want, you can set up FreeBSD on VirtualBox on your own machine, like I do. I'm accessing the machine through SSH and this is comfortable.

[zonyitoo]

Well yes, that’s what I was working on. But I am totally unfamiliar with pf and ipfw, this is one of the huge obstacles in front of me.

[ge9]

Actually I'm not either so familiar with pf and ipfw. My configuration above will be just enough for transparent proxy testing.

[Orum]

...I am totally unfamiliar with pf and ipfw, this is one of the huge obstacles in front of me.

I am happy to help with pf rules, but I don't really know ipfw well enough to offer assistance there. I also don't use IPv4 to 6 translation/proxying (at least on FreeBSD), so I'm not sure I'll be of use for that either.

For basic transparent proxying though, the rules should be fairly straightforward (assuming UDP proxying is supposed to be configured like TCP; as that's one of the issues we're trying to solve here, it may not be). I'll expand a little on the rule in my original post: rdr on $int_if inet proto { tcp, udp } to !<private> -> 127.0.0.1 port 1080

rdr is a redirection rule, meaning it applies to incoming traffic, and will translate portions of the IP headers. In this case, all incoming traffic on the $int_if (a macro, which is basically like a variable in sh/bash, that's defined earlier in the pf.rules) interface that is IPv4 (the inet part) and is either TCP or UDP (proto { tcp, udp }) and doesn't have a destination listed in the "private" table (to !<private>) will be redirected to 127.0.0.1 port 1080 (-> 127.0.0.1 port 1080). The table must be defined after macros, but before the redirection rules. In turn, filtering rules must be at the end of the ruleset, though for testing purposes you shouldn't need anything other than a very simple rule to pass everything. In other words, your full ruleset might look something like this:

# set this to have the name of whatever your internal (i.e. 'LAN')
# interface is, that will be redirected to shadowsocks:
int_if="vlan4"
# set this to your public/WAN interface:
isp_if="vlan2"

# a table of all private/unroutable/reserved addresses
table <private> { 224/3, 0/8, 127/8, 10/8, 172.16/12, 192.168/16, 169.254/16 } const

# various options I usually set
set optimization normal
set block-policy drop
set ruleset-optimization basic

# normalization
scrub all random-id reassemble tcp fragment reassemble

# translation rules; first a basic NAT rule for direct connectivity,
# and then one to redirect clients on $int_if to shadowsocks
nat on $isp_if inet from any to !<private> -> ($isp_if)
rdr on $int_if inet proto { tcp, udp } to !<private> -> 127.0.0.1 port 1080

# define whatever filtering rules here, but for this example I'll just pass everything
pass quick all

Does that make sense? Let me know if anything is unclear.

[zonyitoo]

Status report: I was figuring out why kldload ipfw freezes my SSH connection.

It is quite late tonight, I will continue tomorrow with your setting with pf.

[ge9]

Ah I also had the proble, I should've mentioned If you run the command in the local FreeBSD machine (not via SSH), it will work.

[ge9]

I have one thing to add, actually I'm testing transparent proxy with only one machine, by making the machine send packet to itself through the loopback device. 10.0.2.25 is the source address to which the proxy will be applied. Then we can test proxy by some commands like nc -s 10.0.2.25 192.168.1.1 8888. FYI, this is my FreeBSD pf (not ipfw) rule for testing.

rdr pass on lo0 proto {tcp, udp} from 10.0.2.25 -> 127.0.0.1 port 22222
pass out quick route-to lo0 from 10.0.2.25
pass

[zonyitoo]

BTW, mark-based routing on FreeBSD is also supported, which should be something like Linux's SO_MARK:

https://github.com/shadowsocks/shadowsocks-rust/blob/a27410c52d110f6e8a9c301dd82232ab67b4b5b1/crates/shadowsocks/src/net/sys/unix/bsd/freebsd.rs#L43-L59

It should allow you to easily route the outbound requests sent from sslocal to the actual outbound interface.

But I really don't know how to set it in pf or ipfw.

  SO_USER_COOKIE can be used to set the uint32_t so_user_cookie field  in
       the  socket.   The  value is an uint32_t, and can be used in the kernel
       code that manipulates traffic related to the socket.  The default value
       for the field is 0.  As an example, the value can be used as the skipto
       target or pipe number in ipfw/dummynet.

ipfw document about this SO_USER_COOKIE:

    skipto number | tablearg
           Skip  all  subsequent  rules  numbered  less  than number.  The
           search continues with the first rule numbered number or higher.
           It is possible to use the tablearg keyword with a skipto for  a
           computed  skipto.   Skipto  may  work either in O([log(N)](https://man.freebsd.org/cgi/man.cgi?query=log&sektion=N&apropos=0&manpath=FreeBSD+14.1-RELEASE+and+Ports)) or in
           [O(1)](https://man.freebsd.org/cgi/man.cgi?query=O&sektion=1&apropos=0&manpath=FreeBSD+14.1-RELEASE+and+Ports) depending on amount   of  memory  and/or  sysctl  variables.
           See the "SYSCTL VARIABLES" section for more details.

[zonyitoo]

My VM has IP 192.168.11.191, and sslocal ran with the following command:

./target/debug/ssservice local -b '127.0.0.1:22222' -s '192.168.11.1:8388' -m none --protocol redir -v -U

I want to test UDP redirects locally on FreeBSD,

On the other hand, /etc/pf.conf was setting like this:

rdr pass on lo0 proto {udp} from 192.168.11.191 -> 127.0.0.1 port 22222
pass out quick route-to lo0 from 192.168.11.191
pass

It doesn't work, which seems to create an infinite loop.