Open simonefil opened 1 year ago
Are you running udp-proxy-2020
on both the remote server and home OPNSense router? That is a requirement for a site-to-site VPN with the Roon Discovery protocol.
Edit: I should say I've never personally tried this and so I can't say certain that it will work. My understanding of the Roon Discovery protocol is not perfect, but I believe it to be possible.
Are you running
udp-proxy-2020
on both the remote server and home OPNSense router? That is a requirement for a site-to-site VPN with the Roon Discovery protocol.
Yes I am.
On the remote server with the following parameters: --interface eth0,wg0 --port 9003 --no-listen On my home router with the following parameters: --interface re0,wg1 --fixed-ip wg1@10.200.200.1 --port 9003 (where 10.200.200.1 is IP address of the remote server and re0 is the local LAN interface)
can you capture traffic on the remote end as well?
Also please provide the output of udp-proxy-2020 -l
from both systems.
Sure.
Interfaces on remote server:
Interface: eth0
Interface: wg0
Interface: lo
Interfaces on the router:
Interface: re0
Interface: pppoe0
Interface: wg1
Interface: lo0
pcap files captured on the remote server:
Hmmm... I think I see the problem. Gonna need to figure out how I can reproduce this locally and figure out a fix.
Sure, let me know if you need me to do further testing. Thanks
so I don't have OPNsense, but I do have PFSense which is similar-ish AFAIK. I'd expect the Wireguard implementation at least to be the same...
but it's not. Your Wireguard interfaces are point-to-point. And I believe that is the cause of the errors in the logs:
level=warning msg="Unable to send 362 bytes from re0 out wg1: send: Address family not supported by protocol family"
You're probably going to see a different set of flags than I do for my Wireguard interface:
$ ifconfig tun_wg0
tun_wg0: flags=81c1<UP,RUNNING,NOARP,PROMISC,MULTICAST> metric 0 mtu 1500
description: WG0
options=80000<LINKSTATE>
inet 172.16.12.1 netmask 0xffffff00
groups: wg
nd6 options=101<PERFORMNUD,NO_DAD>
Note that the PROMISC
flag is because udp-proxy-2020
is running. I don't really have a way of testing your config to really know what is going on, but if I were you I'd see if I could figure out how to not run it in PointToPoint mode.
Mmm could you please be more clear about "Your Wireguard interfaces are point-to-point."? My overall wireguard network topology if more like a mesh network (I guess?), all traffic pass through 10.200.200.1, every client can see each other connected client. The router (10.200.200.12) is just one of the multiple clients.
In the specific 10.200.200.1 (Server) - 10.200.200.12 (router) "subsection" it acts as a site-to-multisite, where are the natted clients behind the router can talk to the 10.200.200.1 server.
In fact, the generated wg1.conf file from OPNSense is as follows:
[Interface]
PrivateKey = xxx
Address = 10.200.200.12/24
[Peer]
PublicKey = xxx
Endpoint = aaa.bbb.ccc.ddd:51194
AllowedIPs = 10.200.200.0/24
PersistentKeepalive = 30
With udp-proxy-2020 enabled my ifconfig is the same as yours. If by point-to-point you mean the interface flag it's not the case.
root@OPNsense:/var/log # ifconfig wg1
wg1: flags=81c1<UP,RUNNING,NOARP,PROMISC,MULTICAST> metric 0 mtu 1420
description: Hetzner (opt1)
options=80000<LINKSTATE>
inet 10.200.200.12 netmask 0xffffff00
groups: wg wireguard
nd6 options=109<PERFORMNUD,IFDISABLED,NO_DAD>
When I stop the udp-proxy-2020 server the interface loses the PROMISC flag like it's expected to.
the interface on the remote server:
Interface: wg0
- IP: 10.200.200.1/24 PointToPoint: 10.200.200.1
the interface on the remote server:
Interface: wg0 - IP: 10.200.200.1/24 PointToPoint: 10.200.200.1
Mmm I'm missing something..
The udp-proxy-2020 on my router, which has those logged error messages (Address family not supported by protocol family) and whose interface (wg1) does not have the pointopoint flag is not able route the broadcast packets on the tunnel because the remote interface (on the remote server with linux) has the wg0 interface with the pointopoint flag?
As far as I know, in linux, wireguard always creates the wg0 interface with the POINTOPOINT flag. Please correct me if I'm wrong.
So the log is saying:
level=warning msg="Unable to send 362 bytes from re0 out wg1: send: Address family not supported by protocol family"
But the pcap actually shows all the packets from re0 being sent on wg1 on your OPNSense box. Just not seeing any replies from your Linux server on the remote end over the VPN tunnel. The Address family not supported by protocol family
message is coming from your *BSD kernel so not sure what is going on really.
If you use the --pcap
flag on the remote/Linux box it would be interesting to see if the packets are arriving or not.
Also, on the remote/linux box can you run netstat -an | grep :9003
and report that?
Sure:
root@simone-roon:~# netstat -an | grep :9003
udp 0 0 0.0.0.0:9003 0.0.0.0:*
udp 0 0 0.0.0.0:9003 0.0.0.0:*
udp 0 0 0.0.0.0:9003 0.0.0.0:*
udp 0 0 0.0.0.0:9003 0.0.0.0:*
udp 0 0 0.0.0.0:9003 0.0.0.0:*
udp 0 0 0.0.0.0:9003 0.0.0.0:*
udp 0 0 0.0.0.0:9003 0.0.0.0:*
udp 0 0 0.0.0.0:9003 0.0.0.0:*
PCAP files from the remote linux server:
Traffic for 192.168.0.0/24 is egressing eth0
on your Linux box. Either you need to update your Wireguard AllowedIPs
list or manually add a route to use wg0
?
It looks fine to me. Eth0 (the 192.168.0.0 subnet) is the WAN interface. The whole wireguard-roon setup on the linux server is actually in an LXC container where the host OS port forward the wireguard 51194 udp traffic into the LXC container so that's reachable through the public static IP of the server. The 192.168.x.x is just an internal lan between all the LXC containers that are on the server. (Which acts as a wan for each container)
The traffic TO the linux server wireguard lxc THROUGH the vpn tunnel is then forwarded to the wan interface so that if a wireguard client has setup as allowed IP 0.0.0.0/0 it will still have internet connection through the tunnel.
This is the wg0.conf on the lxc container on the remote linux server (only the interface section, then there are all the peers one of which is my opnsense router 10.200.200.12)
[Interface]
Address = 10.200.200.1/24
SaveConfig = true
PostUp = iptables -A FORWARD -i %i -j ACCEPT; iptables -A FORWARD -o>
PostDown = iptables -D FORWARD -i %i -j ACCEPT; iptables -D FORWARD >
ListenPort = 51194
(Postup and postdown iptables rules are not complete since I'm from mobile and I cannot copy the whole line from the wg0.conf file)
oh, sorry... saw the private IP space and didn't bother to verify they match what is across the VPN. Doesn't seem like Roon is seeing the packets arriving on a software tunnel interface like wg0
and hence isn't replying.
So first, try restarting Roon on the Linux box. Maybe Roon needs to start after the tunnel is up??
If that doesn't work, on the linux side, can you add --fixed-ip eth0@192.168.0.50
or try --fixed-ip lo@127.0.0.1
. Basically tell udp-proxy-2020
to deliver to a different interface.
Honestly, the more I think about this there is a problem I'm not sure how to resolve without you setting a 2nd LXC container and moving Roon to that (away from udp-proxy-2020 and wireguard).
Basically the way udp-proxy-2020
sends packets makes them invisible to the system which sends them. Only different hosts see the packets. There are ways of course to send packets to yourself, but then I can't spoof the correct source IP address of the Roon client. And without that, Roon will learn the wrong IP address and it won't be able to stream music to the device.
Honestly, the more I think about this there is a problem I'm not sure how to resolve without you setting a 2nd LXC container and moving Roon to that (away from udp-proxy-2020 and wireguard).
Basically the way
udp-proxy-2020
sends packets makes them invisible to the system which sends them. Only different hosts see the packets. There are ways of course to send packets to yourself, but then I can't spoof the correct source IP address of the Roon client. And without that, Roon will learn the wrong IP address and it won't be able to stream music to the device.
This seems more than reasonable. Those two LXC will communicate throught the internal LAN (192.168.x.x). I'll keep you updated
FYI, I think I just figured how to inject UDP frames locally and forge the source IP... only it's not cross platform and has to be different on Linux vs. *BSD. So yeah, I think I can make this work, only it's a bit of work. No promises right now when I'll find time, but kinda motivated to make this work so maybe "soon"... at least relative to the heat death of the universe. :D
So I tried as you said:
LXC Container Wireguard+udp-proxy-2020
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.0.51 netmask 255.255.255.0 broadcast 192.168.0.255
inet6 fe80::d049:1ff:feec:414f prefixlen 64 scopeid 0x20<link>
ether ce:83:d6:36:49:a3 txqueuelen 1000 (Ethernet)
RX packets 55984 bytes 75500165 (72.0 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 86611 bytes 79260743 (75.5 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1000 (Local Loopback)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
wg0: flags=209<UP,POINTOPOINT,RUNNING,NOARP> mtu 1420
inet 10.200.200.1 netmask 255.255.255.0 destination 10.200.200.1
unspec 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00 txqueuelen 1000 (UNSPEC)
RX packets 29878 bytes 6236664 (5.9 MiB)
RX errors 26 dropped 0 overruns 0 frame 26
TX packets 57426 bytes 71058028 (67.7 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
udp-proxy-2020 config
ARGS=--interface eth0,wg0 --port 9003 --level debug --logfile /var/log/udp-proxy-2020.log
LXC Container with Roon
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.0.50 netmask 255.255.255.0 broadcast 192.168.0.255
inet6 fe80::c83e:6bff:fee4:aa03 prefixlen 64 scopeid 0x20<link>
ether ce:83:d6:36:49:a3 txqueuelen 1000 (Ethernet)
RX packets 3262 bytes 2118418 (2.0 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 4550 bytes 1667271 (1.5 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1000 (Local Loopback)
RX packets 11520 bytes 829855 (810.4 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 11520 bytes 829855 (810.4 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
Now let's leave alone for a moment the connection with my opnsense router, I tried connecting my smartphone with 4G network + Wireguard (which has always been working) -> Cannot even see the Roon server anymore
LXC Wireguard + udp-proxy-2020 log (partial) - Smartphone wireguard IP 10.200.200.2 :
level=debug msg="wg0: Learned client IP: 10.200.200.2"
level=debug msg="wg0: received packet and fowarding onto other interfaces"
level=debug msg="eth0: sending out because we're not wg0"
level=debug msg="wg0: Learned client IP: 10.200.200.2"
level=debug msg="wg0: received packet and fowarding onto other interfaces"
level=debug msg="eth0: sending out because we're not wg0"
level=debug msg="wg0: Learned client IP: 10.200.200.2"
level=debug msg="wg0: received packet and fowarding onto other interfaces"
level=debug msg="eth0: sending out because we're not wg0"
level=debug msg="wg0: Learned client IP: 10.200.200.2"
level=debug msg="wg0: received packet and fowarding onto other interfaces"
level=debug msg="eth0: sending out because we're not wg0"
level=debug msg="wg0: Learned client IP: 10.200.200.2"
level=debug msg="wg0: received packet and fowarding onto other interfaces"
level=debug msg="eth0: sending out because we're not wg0"
level=debug msg="wg0: Learned client IP: 10.200.200.2"
level=debug msg="wg0: received packet and fowarding onto other interfaces"
level=debug msg="eth0: sending out because we're not wg0"
level=debug msg="processing packet from wg0 on eth0"
level=debug msg="eth0 => 192.168.0.255: packet len: 140"
level=debug msg="processing packet from wg0 on eth0"
level=debug msg="eth0 => 192.168.0.255: packet len: 140"
level=debug msg="processing packet from wg0 on eth0"
level=debug msg="eth0 => 192.168.0.255: packet len: 140"
level=debug msg="processing packet from wg0 on eth0"
level=debug msg="eth0 => 192.168.0.255: packet len: 140"
level=debug msg="processing packet from wg0 on eth0"
level=debug msg="eth0 => 192.168.0.255: packet len: 140"
level=debug msg="processing packet from wg0 on eth0"
level=debug msg="eth0 => 192.168.0.255: packet len: 140"
Multicast packets tested with iperf between 192.168.0.50 and 192.168.0.51 (the two LXC containers) working without any problem.
Am I missing something very stupid?
I'd have to look at the pcaps. Also, it would be very helpful to get a pcap and output of ifconfig
from the roon container.
tcpdump -i eth0 -s0 -w roon.pcap udp port 9003
edited to add -i eth0
to listen on a specific interface
Well I figured out a way to get packets delivered locally if you install Roon on the same server as udp-proxy-2020: udp-proxy-2020-0.1.0-linux-amd64.zip
you'll need to use the new --deliver-local
flag.
That said, I'm not sure why Roon didn't see packets on the tunnel interface naturally? Roon should be listening on all interfaces (0.0.0.0:9003
):
nestat -an | grep :9003
Hey here I am,
Pcap from the two LXC setup
Then I switch in a single LXC setup (like before) and used your new binary, without success:
root@simone-roon:/var/log# udp-proxy-2020 --interface eth0,wg0 --port 9003 --no-listen --level warn --deliver-local
FATAL can't serialize Eth header: (layers.Ethernet) {
BaseLayer: (layers.BaseLayer) {
Contents: ([]uint8) <nil>,
Payload: ([]uint8) <nil>
},
SrcMAC: (net.HardwareAddr) ,
DstMAC: (net.HardwareAddr) (len=6 cap=6) ff:ff:ff:ff:ff:ff,
EthernetType: (layers.EthernetType) IPv4,
Length: (uint16) 0
}
So the pcap-linux-server.zip
seems to contain the pcaps generated by udp-proxy-2020
but not the pcap generated by tcpdump
on the Roon server. But what I can see from the pcaps you provided, the Roon discovery packets are being broadcast on eth0
and in theory the Roon server should see them... but without the other pcap file, I don't know if the issue is that they're not being delivered (perhaps an issue with LXC??) or if the replies are being lost.
The Fatal error is interesting... can you run --level debug
and try agin? I haven't seen that in my testing, but also hard to test every possible combination.
So the pcap-linux-server.zip seems to contain the pcaps generated by udp-proxy-2020 but not the pcap generated by tcpdump on the Roon server
You're right, my bad. Please have some patience, to test this I need to revert back to the 2 lxc configuration.
The Fatal error is interesting... can you run --level debug and try agin? I haven't seen that in my testing, but also hard to test every possible combination.
My bad X2. I didn't update the service file to use the new binary. Now Roon is finally reachable, although players connected to my router are not visible. If I directly connect a single device with Wireguard (ex. smartphone in 4g + wireguard) then it is seen as an output device. This is the pcap on the SINGLE lxc with roon+wireguard+udp-proxy-2020 with added the --deliver-local flag
No worries... this is complicated. You've been doing better than most honestly :)
Tell me if the log file with the debug argument is needed!
So all the pcaps in the latest zip file (single LCX) are empty. Probably need to restart the Roon client(s) to re-initiate discovery?
As for the --level debug
I'm not really sure why that Fatal
error happened, but if it's not happening anymore then we can ignore it.
Anyways, there seems to be two different potential solutions:
--deliver-local
flag--deliver-local
flag.I think we're both getting a little confused with this, so I'd suggest doing whichever option easiest for you. I don't really care and in theory both should work.
You're right it's better to focus on one single setup to better debug the problem and avoid confusion. I'll stick with the single LXC with wireguard+roonserver+udp-proxy-2020 on it
I've tried multiple times (reconnect clients, reboot the lxc and so on) but the pcap files with the new version remain empty.
This is the log file with the debug argument. As you can see udp-proxy-2020 learns the existence of my router.
The binary was started with the following arguments:
--interface eth0,wg0 --port 9003 --no-listen --deliver-local --logfile /var/log/udp-proxy-2020.log --level debug --pcap --pcap-path /tmp
This is the actual ifconfig of the container:
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.0.50 netmask 255.255.255.0 broadcast 192.168.0.255
inet6 fe80::10ac:c7ff:fef3:85b2 prefixlen 64 scopeid 0x20<link>
ether ce:83:d6:36:49:a3 txqueuelen 1000 (Ethernet)
RX packets 3496 bytes 3835677 (3.6 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 3846 bytes 1559961 (1.4 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1000 (Local Loopback)
RX packets 2936 bytes 501534 (489.7 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 2936 bytes 501534 (489.7 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
wg0: flags=209<UP,POINTOPOINT,RUNNING,NOARP> mtu 1420
inet 10.200.200.1 netmask 255.255.255.0 destination 10.200.200.1
unspec 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00 txqueuelen 1000 (UNSPEC)
RX packets 1784 bytes 206368 (201.5 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 1812 bytes 1183160 (1.1 MiB)
TX errors 189 dropped 0 overruns 0 carrier 0 collisions 0
As always I can reach the server with every client but: 1- If the client connect through the wireguard network of the router, no audio devices are detected 2- If the client is connected directly through wireguard (example: laptopt connected to the 4g hotspot of my smartphone and wireguard on the laptop) all the audio devices are discovered.
I think this binary should fix the pcap generation. udp-proxy-2020-0.1.0-linux-amd64.zip
That said, when you're doing the test from the home router, it would be useful to grab the pcaps from that host as well.
Here I am, sorry for the late reply.
The binary you linked still produces empty pcap files. Nonetherless here you are the log (debug level) from both the router and the LXC with included the pcap files from the router instance
So I see some fatal
messages in the LXC logs which are definitely a serious issue. Here is a new binary which has additional logging which should hopefully provide more info:
udp-proxy-2020-0.1.0-linux-amd64.zip
Additionally, it occurs to me you might have better luck with --interface lo,wg0
on the LXC node. You wouldn't use the --deliver-local
flag in that case. Basically, the way udp-proxy-2020
works, packets sent on eth0
or wg0
are not visible to the system where it is running, but the loopback lo
interface should work. In theory at least... this is getting into deep kernel networking that may not be consistent.
any luck @simonefil ?
any luck @simonefil ?
Hi, sorry for the absence. I was on holiday, from tomorrow I'll be available for more test. Anyway, I tested the binary you uploaded last week and it still doesn't generate pcap files. Before proceeding would you kindly dispel a doubt that I have? Until now our focus has been mostly on the LXC side, so the machine which has installed Wireguard and RoonServer, which have always been working as long as whatever client I tried to connect to it, has wireguard installed. Things went wrong when I wanted to NOT use wireguard on every client but directly on the router, so in my complete ignorance the only variable that has changed is the router itself so the problem should be in it and not on the LXC side. What am I missing?
The evidence you've provided so far indicates that the Roon Core is not replying to messages sent by your router at home. Hence that's where I've been focused.
The evidence you've provided so far indicates that the Roon Core is not replying to messages sent by your router at home. Hence that's where I've been focused.
Mmm to corroborate that I did two different tests this time. 1- PC -> Router (Wireguard+udp-proxy-2020) -> LXC (Roon+wireguard+udp-proxy-2020) 2- PC (Wireguard) -> Router -> LXC (Roon+wireguard+udp-proxy-2020)
1st test:
Router args: --port 9003 --interface re0,wg1 --fixed-ip wg1@10.200.200.1 --logfile /var/log/udp-proxy-2020.log --level debug --pcap --pcap-path /tmp
LXC args: --interface lo,wg0 --port 9003 --no-listen --logfile /var/log/udp-proxy-2020.log --level debug --pcap --pcap-path /tmp
PCAP + log of both sides: Router-LXC.zip
2nd test: LXC args like 1st test. On the router wireguard and udp-proxy-2020 are turned off.
PCAP+ log of LXC: PC-LXC.zip
Results: 1- Roon library reachable, no audio devices discovered 2- Roon library reachable, audio devices discovered.
P.s. If you don't mind, thanks to your patience I'd like to make a donation. Would you give me your PP address?
busy with work right now, but donations go here: https://github.com/synfinatic/udp-proxy-2020#how-can-i-say-thanks
Hi, I had running udp-proxy-2020 for ages with the following setup: RoonCore (remote server in a datacenter) -> Wireguard -> Client udp-proxy-2020 installed on the remote server with the following parameters: --interface eth0,wg0 --port 9003 --no-listen
Now, I've changed my configuration because when I'm at home I don't want to have to switch on wireguard on every single client, so with a OPNSense router I achieved the following configuration:
RoonCore (remote server) -> Wireguard -> Home OPNsense router -> Clients. With wireguard installed on the router.
With this setup I can access my Roon library but device audio endpoints are not discovered. Every client can ping the remote server without any issue.
So I tried to configure udp-proxy-2020 on the router too with the following parameters: --interface re0,wg1 --fixed-ip wg1@10.200.200.1 --port 9003 where re0 is the lan interface, without any success. I can' t still see any audio endpoint.
Here the pcap files and the udp-proxy-2020 log on the router udp-proxy-2020.log pcap.zip
I'm not that good at networking so I might be missing something very stupid.
May I ask you for some help?
Thanks,
Simone