Open Orum opened 1 month ago
I have never tested on FreeBSD, so this project supports FreeBSD is still "theoretically".
I have tested pf
on macOS, which supports TCP perfectly just like you did on FreeBSD. But on macOS, UDP's destination must be retrieved from ioc_getstates
. This API is far different from FreeBSD's definition, so I think FreeBSD doesn't use the same mechanism to get UDP packets' destination when redirected by pf
.
In your last PR, you said on FreeBSD UDP's destination could be retrieved from msg_name
returned by recvfrom
. Is that correct? I couldn't see any document from FreeBSD talking about it.
The 127.0.0.1:1080
in your case was exactly the value of msg_name
returned from recv_dest_from.
In your last PR, you said on FreeBSD UDP's destination could be retrieved from msg_name returned by recvfrom. Is that correct? I couldn't see any document from FreeBSD talking about it.
That wasn't me, so you must be thinking of someone else. While I know how to write rules for pf fairly well, I've never programmed anything to interface with it, so I'm unsure of how the API is supposed to be used.
The good news is that you can access the man pages online. There's also a forum, mailing lists, and a Discord server, all of which feature some form of development/programming discussion.
I'm interested in learning as well, but I'm still learning rust, so it may be some time until I am of use.
Glad to know that you are familiar with FreeBSD and pf. Could you find any doc / references about how to get the "original destination address" of UDP packets?
On Linux, it is quite straight forward: https://man7.org/linux/man-pages/man7/ip.7.html (IP_RECVORIGDSTADDR
).
On macOS, I got it from another post: (?) Deleted.
A famous tools mitmproxy only supports TCP on FreeBSD: https://docs.mitmproxy.org/stable/howto-transparent/ .
So it is quite hard to find a reference about how FreeBSD handles UDP redirects and how to get the original destination address programmatically.
And you could also try the ipfw
: https://forums.freebsd.org/threads/retrieving-the-destination-port-in-an-ipfwed-udp-packet.37545/
@Orum Do you still interest in making this Project working on FreeBSD? What's the current status?
I am still very interested in making it work. However, I think the best way to do so at this point is to just read through FreeBSD's source code. As such, I'm familiarizing myself with it, but it's going to take some time as I have other obligations and projects demanding my attention, so it's on the back burner for me right now. This is probably the best, or at least, surest way to figure out how to get the address (assuming it's even possible at present) from pf.
Additionally, I'd need to learn much more about rust before I contribute anything other than information to this project, as my knowledge of it is spotty at best. But, if I do figure out the FreeBSD side of the picture, perhaps someone here can handle that side of things once they have the information. I'm also not only willing, but happy to test things to see if they work once they're implemented.
https://github.com/semigodking/redsocks/issues/200
FYI, I succeeded in running transparent proxy in FreeBSD with redsocks+SOCKS5 (sorry they are completely unrelated to shadowsocks) and ipfw, but not with pf. When I tested with pf, the UDP destination address was obtained as the transparent proxy's listen address, just as @Orum mentioned.
As I mentioned in the link, in OpenBSD, where pf originates from, using divert-to
rule of pf worked perfectly. divert
rule seems not ported to FreeBSD's pf.
FreeBSD and OpenBSD diverged in their rules syntax quite a few years back when the latter made some changes that broke older rule sets, IIRC. I've stuck with FreeBSD, which might have an equivalent of divert-to
(or not), but I don't really know what it does as I never bothered to keep up with OBSD's syntax as I don't use it.
I also don't really want to switch to ipfw, or any other firewall, as I really value simplicity and clarity of pf's rules' syntax. So many other firewalls are hard to decipher, which is undesirable to say the least when it comes to security.
In any case I will try and dedicate some time to reading through and understanding the pf/bpf code in the future, but that will be some time from now.
@ge9 Have you tried shadowsocks-rust with ipfw? What problem did you see?
I tested it and it worked, but there were a problem.
shadowsocks-rust uses IPv4-mapped IPv6 address to return message to IPv4 clients:
https://github.com/shadowsocks/shadowsocks-rust/blob/ac67a08f87afe54e844241fb75d8bfc018a0bd03/crates/shadowsocks-service/src/local/redir/udprelay/mod.rs#L199
However, FreeBSD disables IPv4-mapped IPv6 address by default (according to https://github.com/scala-native/scala-native/issues/3630). If I set sysctl net.inet6.ip6.v6only=0
, then it worked.
Except for this, normal transparent proxy setting will work.
FYI, My settings are following:
#!/bin/sh
kldload ipfw
fwcmd=ipfw
ifconfig em0 alias 10.0.2.25 netmask 0xffffff00
$fwcmd add 100 allow all from any to any via lo0
$fwcmd add 500 fwd 127.0.0.1,22222 tcp from 10.0.2.25 to any
$fwcmd add 600 fwd 127.0.0.1,22222 udp from 10.0.2.25 to any
$fwcmd add 700 allow ip from any to any
sslocal -b 127.0.0.1:22222 -U --protocol redir -s 192.168.1.7:1080 --tcp-redir pf --udp-redir pf -m none
pf
seems fine)ssserver -U -s 192.168.1.7:1080 -m none
(on another PC)
0.0.0.0:1080
here doesn't workThere is a SUPPORT_IPV6_TRANSPARENT
flag that indicates whether the current platform supports IPv6, if it is true
, then shadowsocks-rust will always convert IPv4 to IPv4-mapped-IPv6 and uses only one socket for sending back.
This is always working for Linux, because dual-stack socket was enabled by default for most distribution of Linux.
Is there a way to test if dual-stack is enabled on FreeBSD? By reading sysctl net.inet6.ip6.v6only=0
should be one possible option.
Could you please help to run the test program on FreeBSD and see if any of these syscalls returned errors? https://stackoverflow.com/questions/30184377/how-to-detect-if-dual-stack-socket-is-supported
cc @madeye , FreeBSD doesn't support IPv4-mapped-IPv6 by default. So my previous (years ago) PR that uses IPv6 sockets for sending back UDP packets won't be able to run properly on FreeBSD.
This is the result of the python test program.
Traceback (most recent call last):
File "/root/test.py", line 4, in <module>
s.connect(('::ffff:169.254.1.1', 53))
OSError: [Errno 22] Invalid argument
If I set sysctl net.inet6.ip6.v6only=0
and commented out s.setsockopt(socket.IPPROTO_IPV6, socket.IPV6_V6ONLY, 1)
, it worked. Otherwise, it failed.
In the shadowsocks, this is the error.
2024-06-17T03:45:10.334750916+09:00 WARN udp failed to send back 68 bytes to client 10.0.2.25:44550, from target 3.132.228.249:3478 (proxied), error: Invalid argument (os error 22)
This originates from https://github.com/shadowsocks/shadowsocks-rust/blob/ac67a08f87afe54e844241fb75d8bfc018a0bd03/crates/shadowsocks-service/src/local/redir/udprelay/mod.rs#L203 .
Please test if it is Ok without settingnet.inet6.ip6.v6only=0
manually.
I just found a better solution. Working on it.
Tested on OpenWRT. Please help testing it if it is working correctly on FreeBSD. @ge9
5ba8b7d worked but 765c9e5 or cd25d25 (latest) doesn't (same error).
Please run again with -v
and check these debug logs:
Are they all true
?
here is the log. seems all true.
2024-06-17T06:38:52.894052461+09:00 TRACE tokio-runtime-worker ThreadId(02) shadowsocks_service::local::redir::udprelay: crates/shadowsocks-service/src/local/redir/udprelay/mod.rs:285: received UDP packet from 10.0.2.25:65158, destination 3.132.228.249:3478, length 28 bytes
2024-06-17T06:38:52.894445849+09:00 DEBUG tokio-runtime-worker ThreadId(02) shadowsocks_service::local::net::udp::association: crates/shadowsocks-service/src/local/net/udp/association.rs:123: created udp association for 10.0.2.25:65158
2024-06-17T06:38:52.894817377+09:00 TRACE tokio-runtime-worker ThreadId(02) shadowsocks_service::local::net::udp::association: crates/shadowsocks-service/src/local/net/udp/association.rs:433: udp relay 10.0.2.25:65158 -> 3.132.228.249:3478 (proxied) with 28 bytes
2024-06-17T06:38:52.894896617+09:00 TRACE tokio-runtime-worker ThreadId(02) mio::poll: /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/mio-0.8.11/src/poll.rs:551: registering event source with poller: token=Token(79171555479936), interests=READABLE | WRITABLE
2024-06-17T06:38:52.895050148+09:00 TRACE tokio-runtime-worker ThreadId(02) shadowsocks::relay::udprelay::proxy_socket: crates/shadowsocks/src/relay/udprelay/proxy_socket.rs:103: connected udp remote 192.168.1.8:1080 (outbound: 192.168.1.8:1080) with ConnectOpts { user_cookie: None, bind_local_addr: None, bind_interface: None, tcp: TcpSocketOpts { send_buffer_size: None, recv_buffer_size: None, nodelay: false, fastopen: false, keepalive: Some(15s), mptcp: false }, udp: UdpSocketOpts { mtu: None } }
2024-06-17T06:38:52.895129216+09:00 TRACE tokio-runtime-worker ThreadId(02) shadowsocks::relay::udprelay::proxy_socket: crates/shadowsocks/src/relay/udprelay/proxy_socket.rs:235: UDP server client send to 3.132.228.249:3478, control: UdpSocketControlData { client_session_id: 786820869274048462, server_session_id: 0, packet_id: 1, user: None }, payload length 28 bytes, packet length 35 bytes
2024-06-17T06:38:53.135364801+09:00 TRACE tokio-runtime-worker ThreadId(02) shadowsocks::relay::udprelay::proxy_socket: crates/shadowsocks/src/relay/udprelay/proxy_socket.rs:445: UDP server client receive from 3.132.228.249:3478, control: None, packet length 75 bytes, payload length 68 bytes
2024-06-17T06:38:53.136228332+09:00 TRACE tokio-runtime-worker ThreadId(02) shadowsocks_service::local::net::udp::association: crates/shadowsocks-service/src/local/net/udp/association.rs:607: udp relay 10.0.2.25:65158 <- 3.132.228.249:3478 (proxied) received 68 bytes
2024-06-17T06:38:53.136421625+09:00 DEBUG tokio-runtime-worker ThreadId(02) shadowsocks::net::sys: crates/shadowsocks/src/net/sys/mod.rs:155: IpStackCapability support_ipv4=true
2024-06-17T06:38:53.136798744+09:00 DEBUG tokio-runtime-worker ThreadId(02) shadowsocks::net::sys: crates/shadowsocks/src/net/sys/mod.rs:164: IpStackCapability support_ipv6=true
2024-06-17T06:38:53.137187564+09:00 DEBUG tokio-runtime-worker ThreadId(02) shadowsocks::net::sys: crates/shadowsocks/src/net/sys/mod.rs:175: IpStackCapability support_ipv4_mapped_ipv6=true
2024-06-17T06:38:53.13739486+09:00 TRACE tokio-runtime-worker ThreadId(02) mio::poll: /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/mio-0.8.11/src/poll.rs:551: registering event source with poller: token=Token(79171555480192), interests=READABLE | WRITABLE
2024-06-17T06:38:53.137685127+09:00 WARN tokio-runtime-worker ThreadId(02) shadowsocks_service::local::net::udp::association: crates/shadowsocks-service/src/local/net/udp/association.rs:620: udp failed to send back 68 bytes to client 10.0.2.25:65158, from target 3.132.228.249:3478 (proxied), error: Invalid argument (os error 22)
Well, so FreeBSD allows bind()
ing a IPv4-mapped-IPv6 address, but doesn’t allow sendmsg
to it. Programs that are written in Go should also fails with the same reason.
Call connect()
in this commit. Please help verify it again.
Hmm it seems still not working...
2024-06-16T23:01:19.392825484Z TRACE tokio-runtime-worker ThreadId(03) shadowsocks_service::local::redir::udprelay: crates/shadowsocks-service/src/local/redir/udprelay/mod.rs:285: received UDP packet from 10.0.2.25:13870, destination 3.132.228.249:3478, length 28 bytes
2024-06-16T23:01:19.393494564Z DEBUG tokio-runtime-worker ThreadId(03) shadowsocks_service::local::net::udp::association: crates/shadowsocks-service/src/local/net/udp/association.rs:123: created udp association for 10.0.2.25:13870
2024-06-16T23:01:19.393869192Z TRACE tokio-runtime-worker ThreadId(03) shadowsocks_service::local::net::udp::association: crates/shadowsocks-service/src/local/net/udp/association.rs:433: udp relay 10.0.2.25:13870 -> 3.132.228.249:3478 (proxied) with 28 bytes
2024-06-16T23:01:19.394037649Z TRACE tokio-runtime-worker ThreadId(03) mio::poll: /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/mio-0.8.11/src/poll.rs:551: registering event source with poller: token=Token(88188202799232), interests=READABLE | WRITABLE
2024-06-16T23:01:19.394211694Z TRACE tokio-runtime-worker ThreadId(03) shadowsocks::relay::udprelay::proxy_socket: crates/shadowsocks/src/relay/udprelay/proxy_socket.rs:103: connected udp remote 192.168.1.212:1080 (outbound: 192.168.1.212:1080) with ConnectOpts { user_cookie: None, bind_local_addr: None, bind_interface: None, tcp: TcpSocketOpts { send_buffer_size: None, recv_buffer_size: None, nodelay: false, fastopen: false, keepalive: Some(15s), mptcp: false }, udp: UdpSocketOpts { mtu: None } }
2024-06-16T23:01:19.394416748Z TRACE tokio-runtime-worker ThreadId(03) shadowsocks::relay::udprelay::proxy_socket: crates/shadowsocks/src/relay/udprelay/proxy_socket.rs:235: UDP server client send to 3.132.228.249:3478, control: UdpSocketControlData { client_session_id: 8860848107368566370, server_session_id: 0, packet_id: 1, user: None }, payload length 28 bytes, packet length 35 bytes
2024-06-16T23:01:19.566423525Z TRACE tokio-runtime-worker ThreadId(03) shadowsocks::relay::udprelay::proxy_socket: crates/shadowsocks/src/relay/udprelay/proxy_socket.rs:445: UDP server client receive from 3.132.228.249:3478, control: None, packet length 75 bytes, payload length 68 bytes
2024-06-16T23:01:19.567810852Z TRACE tokio-runtime-worker ThreadId(03) shadowsocks_service::local::net::udp::association: crates/shadowsocks-service/src/local/net/udp/association.rs:607: udp relay 10.0.2.25:13870 <- 3.132.228.249:3478 (proxied) received 68 bytes
2024-06-16T23:01:19.568463729Z DEBUG tokio-runtime-worker ThreadId(03) shadowsocks::net::sys: crates/shadowsocks/src/net/sys/mod.rs:156: IpStackCapability support_ipv4=true
2024-06-16T23:01:19.568883894Z DEBUG tokio-runtime-worker ThreadId(03) shadowsocks::net::sys: crates/shadowsocks/src/net/sys/mod.rs:165: IpStackCapability support_ipv6=true
2024-06-16T23:01:19.56980524Z DEBUG tokio-runtime-worker ThreadId(03) shadowsocks::net::sys: crates/shadowsocks/src/net/sys/mod.rs:173: IpStackCapability support_ipv4_mapped_ipv6=true
2024-06-16T23:01:19.569994091Z TRACE tokio-runtime-worker ThreadId(03) mio::poll: /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/mio-0.8.11/src/poll.rs:551: registering event source with poller: token=Token(88188202799488), interests=READABLE | WRITABLE
2024-06-16T23:01:19.578334536Z WARN tokio-runtime-worker ThreadId(03) shadowsocks_service::local::net::udp::association: crates/shadowsocks-service/src/local/net/udp/association.rs:620: udp failed to send back 68 bytes to client 10.0.2.25:13870, from target 3.132.228.249:3478 (proxied), error: Invalid argument (os error 22)
Replace it with the method just like the one on stackoverflow. Please try again. @ge9
Still not working...
2024-06-16T23:53:16.161455017Z TRACE tokio-runtime-worker ThreadId(02) shadowsocks_service::local::redir::udprelay: crates/shadowsocks-service/src/local/redir/udprelay/mod.rs:285: received UDP packet from 10.0.2.25:34707, destination 3.132.228.249:3478, length 28 bytes
2024-06-16T23:53:16.161934966Z DEBUG tokio-runtime-worker ThreadId(02) shadowsocks_service::local::net::udp::association: crates/shadowsocks-service/src/local/net/udp/association.rs:123: created udp association for 10.0.2.25:34707
2024-06-16T23:53:16.162531969Z TRACE tokio-runtime-worker ThreadId(02) shadowsocks_service::local::net::udp::association: crates/shadowsocks-service/src/local/net/udp/association.rs:433: udp relay 10.0.2.25:34707 -> 3.132.228.249:3478 (proxied) with 28 bytes
2024-06-16T23:53:16.162692883Z TRACE tokio-runtime-worker ThreadId(02) mio::poll: /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/mio-0.8.11/src/poll.rs:551: registering event source with poller: token=Token(51249433133440), interests=READABLE | WRITABLE
2024-06-16T23:53:16.162828655Z TRACE tokio-runtime-worker ThreadId(02) shadowsocks::relay::udprelay::proxy_socket: crates/shadowsocks/src/relay/udprelay/proxy_socket.rs:103: connected udp remote 192.168.1.212:1080 (outbound: 192.168.1.212:1080) with ConnectOpts { user_cookie: None, bind_local_addr: None, bind_interface: None, tcp: TcpSocketOpts { send_buffer_size: None, recv_buffer_size: None, nodelay: false, fastopen: false, keepalive: Some(15s), mptcp: false }, udp: UdpSocketOpts { mtu: None } }
2024-06-16T23:53:16.16302868Z TRACE tokio-runtime-worker ThreadId(02) shadowsocks::relay::udprelay::proxy_socket: crates/shadowsocks/src/relay/udprelay/proxy_socket.rs:235: UDP server client send to 3.132.228.249:3478, control: UdpSocketControlData { client_session_id: 17626901403037311660, server_session_id: 0, packet_id: 1, user: None }, payload length 28 bytes, packet length 35 bytes
2024-06-16T23:53:16.356517518Z TRACE tokio-runtime-worker ThreadId(02) shadowsocks::relay::udprelay::proxy_socket: crates/shadowsocks/src/relay/udprelay/proxy_socket.rs:445: UDP server client receive from 3.132.228.249:3478, control: None, packet length 75 bytes, payload length 68 bytes
2024-06-16T23:53:16.357477975Z TRACE tokio-runtime-worker ThreadId(02) shadowsocks_service::local::net::udp::association: crates/shadowsocks-service/src/local/net/udp/association.rs:607: udp relay 10.0.2.25:34707 <- 3.132.228.249:3478 (proxied) received 68 bytes
2024-06-16T23:53:16.358022457Z DEBUG tokio-runtime-worker ThreadId(02) shadowsocks::net::sys: crates/shadowsocks/src/net/sys/mod.rs:155: IpStackCapability support_ipv4=true
2024-06-16T23:53:16.358794623Z DEBUG tokio-runtime-worker ThreadId(02) shadowsocks::net::sys: crates/shadowsocks/src/net/sys/mod.rs:164: IpStackCapability support_ipv6=true
2024-06-16T23:53:16.359158356Z DEBUG tokio-runtime-worker ThreadId(02) shadowsocks::net::sys: crates/shadowsocks/src/net/sys/mod.rs:172: IpStackCapability support_ipv4_mapped_ipv6=true
2024-06-16T23:53:16.361218674Z TRACE tokio-runtime-worker ThreadId(02) mio::poll: /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/mio-0.8.11/src/poll.rs:551: registering event source with poller: token=Token(51249433133696), interests=READABLE | WRITABLE
2024-06-16T23:53:16.362913582Z WARN tokio-runtime-worker ThreadId(02) shadowsocks_service::local::net::udp::association: crates/shadowsocks-service/src/local/net/udp/association.rs:620: udp failed to send back 68 bytes to client 10.0.2.25:34707, from target 3.132.228.249:3478 (proxied), error: Invalid argument (os error 22)
Udp Socket bind([::ffff:3.132.228.249]:3478)
was Ok, but then sendmsg([::ffff:10.0.2.25]:34707)
failed?
Is that a bug in FreeBSD? :(
Further more, in the test code, sockets bind([::ffff:127.0.0.1]:0)
and connect([::ffff:127.0.0.1]:53)
were all working well.
For TCP socket, bind([::ffff:127.0.0.1]:0)
and then listen()
Ok. For client socket, bind([::ffff:127.0.0.1]:0)
and then connect()
to the server socket Ok.
Getting Headache.
(edited)
By the way, I forgot to test TCP. With ipfw, --udp-redir pf
is fine, but --tcp-redir pf
did not work. And TCP seems not affected by this IPv4-mapped address issue (it works fine with ipfw and --tcp-redir ipfw
).
UDP works with pf
but TCP doesn't? That wasn't expected.
It's about testing with ipfw, not pf.
Ok. Please also test the TCP with ipfw. Let's fix all of them together.
I think it's OK that --tcp-redir pf
+ipfw doesn't work, because --tcp-redir pf
introduces special treatment for redirected TCP packets.
I'm not so familiar with the detail of implementation, but is it not possible to always use IPv4 address for IPv4 clients? (i.e. completely exclude IPv4-mapped IPv6 address)?
Of course it is Ok. If the support_ipv4_mapped_ipv6
is false
in the IpStackCapabilities
, redir will create different sockets for IPv4 and IPv6. But now the problem is why it is always true
.
In my understanding, FreeBSD does support IPv4-mapped addresses, but intentionally disables it with security reason. (according to the link). So, it may not be the problem of capabilities.
IPv4-mapped IPv6 are handled right here:
It is actually bypasses it to IPv4 implementation.
Was that if ((inp->inp_flags & IN6P_IPV6_V6ONLY) == 0)
equal to the sysctl net.inet6.ip6.v6only=0
?
For bind
, it should always success. Because if the sysctl was not 0
, nothing would actually happened.
connect()
for TCP should also reject IPv4-mapped addresses. Hmm, why.
It should also reject in bind()
. WHY?? WHY??
Is that because we have set IPV6_V6ONLY
with setsockopt
? If so, let me try to do that in redir's send-back socket.
Morning. Please help testing if this commit working in FreeBSD. @ge9
I'm afraid that's still not working.
This is UDP log,
2024-06-17T11:58:24.450931472Z TRACE tokio-runtime-worker ThreadId(02) shadowsocks_service::local::redir::udprelay: crates/shadowsocks-service/src/local/redir/udprelay/mod.rs:285: received UDP packet from 10.0.2.25:11770, destination 3.132.228.249:3478, length 28 bytes
2024-06-17T11:58:24.451373707Z DEBUG tokio-runtime-worker ThreadId(02) shadowsocks_service::local::net::udp::association: crates/shadowsocks-service/src/local/net/udp/association.rs:123: created udp association for 10.0.2.25:11770
2024-06-17T11:58:24.451986355Z TRACE tokio-runtime-worker ThreadId(02) shadowsocks_service::local::net::udp::association: crates/shadowsocks-service/src/local/net/udp/association.rs:433: udp relay 10.0.2.25:11770 -> 3.132.228.249:3478 (proxied) with 28 bytes
2024-06-17T11:58:24.452117657Z TRACE tokio-runtime-worker ThreadId(02) mio::poll: /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/mio-0.8.11/src/poll.rs:551: registering event source with poller: token=Token(79307137466752), interests=READABLE | WRITABLE
2024-06-17T11:58:24.452228565Z TRACE tokio-runtime-worker ThreadId(02) shadowsocks::relay::udprelay::proxy_socket: crates/shadowsocks/src/relay/udprelay/proxy_socket.rs:103: connected udp remote 192.168.1.212:1080 (outbound: 192.168.1.212:1080) with ConnectOpts { user_cookie: None, bind_local_addr: None, bind_interface: None, tcp: TcpSocketOpts { send_buffer_size: None, recv_buffer_size: None, nodelay: false, fastopen: false, keepalive: Some(15s), mptcp: false }, udp: UdpSocketOpts { mtu: None } }
2024-06-17T11:58:24.452379981Z TRACE tokio-runtime-worker ThreadId(02) shadowsocks::relay::udprelay::proxy_socket: crates/shadowsocks/src/relay/udprelay/proxy_socket.rs:235: UDP server client send to 3.132.228.249:3478, control: UdpSocketControlData { client_session_id: 18020449999949013623, server_session_id: 0, packet_id: 1, user: None }, payload length 28 bytes, packet length 35 bytes
2024-06-17T11:58:24.624516942Z TRACE tokio-runtime-worker ThreadId(02) shadowsocks::relay::udprelay::proxy_socket: crates/shadowsocks/src/relay/udprelay/proxy_socket.rs:445: UDP server client receive from 3.132.228.249:3478, control: None, packet length 75 bytes, payload length 68 bytes
2024-06-17T11:58:24.625452815Z TRACE tokio-runtime-worker ThreadId(02) shadowsocks_service::local::net::udp::association: crates/shadowsocks-service/src/local/net/udp/association.rs:607: udp relay 10.0.2.25:11770 <- 3.132.228.249:3478 (proxied) received 68 bytes
2024-06-17T11:58:24.626366339Z DEBUG tokio-runtime-worker ThreadId(02) shadowsocks::net::sys: crates/shadowsocks/src/net/sys/mod.rs:155: IpStackCapability support_ipv4=true
2024-06-17T11:58:24.630506251Z DEBUG tokio-runtime-worker ThreadId(02) shadowsocks::net::sys: crates/shadowsocks/src/net/sys/mod.rs:164: IpStackCapability support_ipv6=true
2024-06-17T11:58:24.632915775Z DEBUG tokio-runtime-worker ThreadId(02) shadowsocks::net::sys: crates/shadowsocks/src/net/sys/mod.rs:172: IpStackCapability support_ipv4_mapped_ipv6=true
2024-06-17T11:58:24.634410658Z WARN tokio-runtime-worker ThreadId(02) shadowsocks_service::local::net::udp::association: crates/shadowsocks-service/src/local/net/udp/association.rs:620: udp failed to send back 68 bytes to client 10.0.2.25:11770, from target 3.132.228.249:3478 (proxied), error: Invalid argument (os error 22)
and this is TCP (successful).
2024-06-17T11:58:48.767080033Z TRACE tokio-runtime-worker ThreadId(02) mio::poll: /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/mio-0.8.11/src/poll.rs:551: registering event source with poller: token=Token(79307137467008), interests=READABLE | WRITABLE
2024-06-17T11:58:48.767456059Z TRACE tokio-runtime-worker ThreadId(02) shadowsocks_service::local::redir::tcprelay: crates/shadowsocks-service/src/local/redir/tcprelay/mod.rs:136: got connection 10.0.2.25:22369
2024-06-17T11:58:48.768354218Z TRACE tokio-runtime-worker ThreadId(02) mio::poll: /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/mio-0.8.11/src/poll.rs:551: registering event source with poller: token=Token(79307137467264), interests=READABLE | WRITABLE
2024-06-17T11:58:48.771289507Z TRACE tokio-runtime-worker ThreadId(02) shadowsocks::relay::tcprelay::proxy_stream::client: crates/shadowsocks/src/relay/tcprelay/proxy_stream/client.rs:135: connected tcp remote 192.168.1.212:1080 (outbound: 192.168.1.212:1080) with ConnectOpts { user_cookie: None, bind_local_addr: None, bind_interface: None, tcp: TcpSocketOpts { send_buffer_size: None, recv_buffer_size: None, nodelay: false, fastopen: false, keepalive: Some(15s), mptcp: false }, udp: UdpSocketOpts { mtu: None } }
2024-06-17T11:58:48.771878129Z DEBUG tokio-runtime-worker ThreadId(02) shadowsocks_service::local::utils: crates/shadowsocks-service/src/local/utils.rs:29: established tcp tunnel 10.0.2.25:22369 <-> 3.132.228.249:3478 through sever 192.168.1.212:1080 (outbound: 192.168.1.212:1080)
2024-06-17T11:58:49.291495008Z TRACE tokio-runtime-worker ThreadId(02) shadowsocks::relay::tcprelay::utils: crates/shadowsocks/src/relay/tcprelay/utils.rs:255: copy bidirection ends, a_to_b: Done(68), b_to_a: Done(0)
2024-06-17T11:58:49.291837509Z TRACE tokio-runtime-worker ThreadId(02) shadowsocks_service::local::utils: crates/shadowsocks-service/src/local/utils.rs:72: tcp tunnel 10.0.2.25:22369 <-> 3.132.228.249:3478 (proxied) closed, L2R 0 bytes, R2L 68 bytes
2024-06-17T11:58:49.292008481Z TRACE tokio-runtime-worker ThreadId(02) mio::poll: /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/mio-0.8.11/src/poll.rs:682: deregistering event source from poller
2024-06-17T11:58:49.294304303Z TRACE tokio-runtime-worker ThreadId(02) mio::poll: /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/mio-0.8.11/src/poll.rs:682: deregistering event source from poller
By the way, testing on my FreeBSD is totally fine for me, but if you want, you can set up FreeBSD on VirtualBox on your own machine, like I do. I'm accessing the machine through SSH and this is comfortable.
Well yes, that’s what I was working on. But I am totally unfamiliar with pf and ipfw, this is one of the huge obstacles in front of me.
Actually I'm not either so familiar with pf and ipfw. My configuration above will be just enough for transparent proxy testing.
...I am totally unfamiliar with pf and ipfw, this is one of the huge obstacles in front of me.
I am happy to help with pf rules, but I don't really know ipfw well enough to offer assistance there. I also don't use IPv4 to 6 translation/proxying (at least on FreeBSD), so I'm not sure I'll be of use for that either.
For basic transparent proxying though, the rules should be fairly straightforward (assuming UDP proxying is supposed to be configured like TCP; as that's one of the issues we're trying to solve here, it may not be). I'll expand a little on the rule in my original post: rdr on $int_if inet proto { tcp, udp } to !<private> -> 127.0.0.1 port 1080
rdr
is a redirection rule, meaning it applies to incoming traffic, and will translate portions of the IP headers. In this case, all incoming traffic on the $int_if
(a macro, which is basically like a variable in sh/bash, that's defined earlier in the pf.rules
) interface that is IPv4 (the inet
part) and is either TCP or UDP (proto { tcp, udp }
) and doesn't have a destination listed in the "private" table (to !<private>
) will be redirected to 127.0.0.1 port 1080 (-> 127.0.0.1 port 1080
). The table must be defined after macros, but before the redirection rules. In turn, filtering rules must be at the end of the ruleset, though for testing purposes you shouldn't need anything other than a very simple rule to pass everything. In other words, your full ruleset might look something like this:
# set this to have the name of whatever your internal (i.e. 'LAN')
# interface is, that will be redirected to shadowsocks:
int_if="vlan4"
# set this to your public/WAN interface:
isp_if="vlan2"
# a table of all private/unroutable/reserved addresses
table <private> { 224/3, 0/8, 127/8, 10/8, 172.16/12, 192.168/16, 169.254/16 } const
# various options I usually set
set optimization normal
set block-policy drop
set ruleset-optimization basic
# normalization
scrub all random-id reassemble tcp fragment reassemble
# translation rules; first a basic NAT rule for direct connectivity,
# and then one to redirect clients on $int_if to shadowsocks
nat on $isp_if inet from any to !<private> -> ($isp_if)
rdr on $int_if inet proto { tcp, udp } to !<private> -> 127.0.0.1 port 1080
# define whatever filtering rules here, but for this example I'll just pass everything
pass quick all
Does that make sense? Let me know if anything is unclear.
Status report: I was figuring out why kldload ipfw
freezes my SSH connection.
It is quite late tonight, I will continue tomorrow with your setting with pf
.
Ah I also had the proble, I should've mentioned If you run the command in the local FreeBSD machine (not via SSH), it will work.
I have one thing to add, actually I'm testing transparent proxy with only one machine, by making the machine send packet to itself through the loopback device. 10.0.2.25 is the source address to which the proxy will be applied. Then we can test proxy by some commands like nc -s 10.0.2.25 192.168.1.1 8888
.
FYI, this is my FreeBSD pf (not ipfw) rule for testing.
rdr pass on lo0 proto {tcp, udp} from 10.0.2.25 -> 127.0.0.1 port 22222
pass out quick route-to lo0 from 10.0.2.25
pass
BTW, mark-based routing on FreeBSD is also supported, which should be something like Linux's SO_MARK
:
It should allow you to easily route the outbound requests sent from sslocal
to the actual outbound interface.
But I really don't know how to set it in pf
or ipfw
.
SO_USER_COOKIE can be used to set the uint32_t so_user_cookie field in
the socket. The value is an uint32_t, and can be used in the kernel
code that manipulates traffic related to the socket. The default value
for the field is 0. As an example, the value can be used as the skipto
target or pipe number in ipfw/dummynet.
ipfw
document about this SO_USER_COOKIE
:
skipto number | tablearg
Skip all subsequent rules numbered less than number. The
search continues with the first rule numbered number or higher.
It is possible to use the tablearg keyword with a skipto for a
computed skipto. Skipto may work either in O([log(N)](https://man.freebsd.org/cgi/man.cgi?query=log&sektion=N&apropos=0&manpath=FreeBSD+14.1-RELEASE+and+Ports)) or in
[O(1)](https://man.freebsd.org/cgi/man.cgi?query=O&sektion=1&apropos=0&manpath=FreeBSD+14.1-RELEASE+and+Ports) depending on amount of memory and/or sysctl variables.
See the "SYSCTL VARIABLES" section for more details.
My VM has IP 192.168.11.191, and sslocal
ran with the following command:
./target/debug/ssservice local -b '127.0.0.1:22222' -s '192.168.11.1:8388' -m none --protocol redir -v -U
I want to test UDP redirects locally on FreeBSD,
On the other hand, /etc/pf.conf
was setting like this:
rdr pass on lo0 proto {udp} from 192.168.11.191 -> 127.0.0.1 port 22222
pass out quick route-to lo0 from 192.168.11.191
pass
It doesn't work, which seems to create an infinite loop.
This is a similar symptom to my earlier issue (#1473), but as the circumstances have drastically changed I thought I'd open a new issue. To summarize the changes, I am no longer running
sslocal
on the host (Linux/iptables) machine; it's instead being run on my FreeBSD router which uses pf.Obviously, this requires different firewall rules to work, but there is little if any documentation I could find for transparently proxying via pf within the shadowsocks-rust project. What I've done here is largely guesswork, but this is the rule I came up with to transparently proxy both TCP & UDP to
sslocal
(and then on tossserver
):rdr on $int_if inet proto { tcp, udp } to !<private> -> 127.0.0.1 port 1080
sslocal
is being run on the router via the following command:sslocal -b 127.0.0.1:1080 -U --protocol redir -s 192.168.x.y:8388 -m none --tcp-redir pf --udp-redir pf
...and finally,
ssserver
is run with-U -m none -b 192.168.x.y:8388
This "works" in the sense that both TCP & UDP traffic are being redirected to
sslocal
(unlike in #1473 where only TCP traffic was ever reachingsslocal
), which is then sending the traffic on tosserver
. However, once again, only TCP traffic is being fully proxied correctly.UDP traffic does arrive at
sslocal
without any doubt, as it shows up when runningsslocal
with-vvv
like so (timestamps removed for brevity, and the host system's address being substituted here with simply<host>
):This traffic is then forwarded on to
ssserver
, which I can verify by watching outgoing traffic on the router viatcpdump
. After they arrive atssserver
though, they fail to reach the internet, so clearly something isn't configured correctly.My best guess as to what is happening here, based on the logs, is that
sslocal
thinks the destination of the UDP packet is127.0.0.1
(i.e. the address thatsslocal
is running on and where incoming UDP traffic on the router is redirected to), and fails to retrieve the original destination address before pf'srdr
rule took effect. This wouldn't surprise me as myrdr
rule is complete guesswork, so I assume I'm missing something.If that is indeed the problem, what should the
rdr
rule look like to transparently proxy UDP traffic with pf? Or am I barking up the wrong tree, and is the problem due to something else entirely?