split tunnel failing on new installation

[piramiday]

on a freshly installed system the split-tunnel configuration does not work as expected, whereas on a similar system it has been working great for a while. both systems are Ubuntu 20.04 with AARCH64, are up-to-date and have identical packages installed. all the following has been carefully verified and persists even after reinstalling the PIA application, rebooting, and so on.

both systems:

# OS info

$ cat /etc/lsb-release 
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=20.04
DISTRIB_CODENAME=focal
DISTRIB_DESCRIPTION="Ubuntu 20.04.4 LTS"

# PIA info

$ piactl --version
3.3.0+06906

# SplitTunnel config

$ piactl -u dump daemon-settings \
| jq '.|{ splitTunnelEnabled, defaultRoute, routedPacketsOnVPN, splitTunnelDNS}'
{
  "splitTunnelEnabled": true,
  "defaultRoute": true,
  "routedPacketsOnVPN": true,
  "splitTunnelDNS": true
}

# SplitTunnel IP

$ piactl -u dump daemon-settings | jq .bypassSubnets
[
  {
    "mode": "exclude",
    "subnet": "12.34.56.78/32"
  }
]

now, I have connected the PIA application and have saved each system's iptables rules to file. the diff of these two files, let's call them ok.txt and ko.txt is as follows:

$ diff ok.txt ko.txt
105,106d104
< -A piavpn.r.100.transIp -o enp0s3 -j MASQUERADE
< -A piavpn.r.100.transIp -o tun0 -j MASQUERADE
109,110d106
< -A piavpn.r.80.splitDNS -p udp -m cgroup --cgroup 1383 -m udp --dport 53 -j DNAT --to-destination 169.254.169.254:53
< -A piavpn.r.80.splitDNS -p tcp -m cgroup --cgroup 1383 -m tcp --dport 53 -j DNAT --to-destination 169.254.169.254:53
113,114d108
< -A piavpn.r.90.snatDNS -p udp -m cgroup --cgroup 1383 -m udp --dport 53 -j SNAT --to-source 10.0.0.121
< -A piavpn.r.90.snatDNS -p tcp -m cgroup --cgroup 1383 -m tcp --dport 53 -j SNAT --to-source 10.0.0.121
259,264d252
< -A piavpn.r.320.allowDNS -d 169.254.169.254/32 -p udp -m cgroup --cgroup 1383 -m udp --dport 53 -j ACCEPT
< -A piavpn.r.320.allowDNS -d 169.254.169.254/32 -p tcp -m cgroup --cgroup 1383 -m tcp --dport 53 -j ACCEPT
< -A piavpn.r.320.allowDNS -p udp -m cgroup --cgroup 1383 -m udp --dport 53 -j REJECT --reject-with icmp-port-unreachable
< -A piavpn.r.320.allowDNS -p tcp -m cgroup --cgroup 1383 -m tcp --dport 53 -j REJECT --reject-with icmp-port-unreachable
< -A piavpn.r.320.allowDNS -d 169.254.169.254/32 -p udp -m cgroup ! --cgroup 1383 -m udp --dport 53 -j REJECT --reject-with icmp-port-unreachable
< -A piavpn.r.320.allowDNS -d 169.254.169.254/32 -p tcp -m cgroup ! --cgroup 1383 -m tcp --dport 53 -j REJECT --reject-with icmp-port-unreachable

in other words, all of these rules are present in the working system (ok.txt) but are absent from the iptables' rules of the failing system (ko.txt).

specifically, it seems that the PIA application on the failing system is not setting up splitDNS correctly.

any suggestions on what happenened, and how I can make split tunneling work? thanks.

[JonathonH-PIA]

@piramiday Check your kernel configuration - we've found that a lot of aarch64 kernels are built with CONFIG_PROC_EVENTS=n, we need CONFIG_PROC_EVENTS=y in order to provide split tunnel (so we can detect process launches and put them in the right cgroup). Subnet rules don't technically require this, but we don't currently support enabling just subnet split tunnel rules right now, it's all or nothing.

Do both systems have the same kernel?

You can also check if PIA thinks there is an error preventing split tunnel support (the GUI shows this but I'm guessing these are headless systems): piactl -u dump daemon-state | jq .splitTunnelSupportErrors

It'll return [] (empty array) if split tunnel is supported, or ["cn_proc_invalid"] if PIA thinks your kernel lacks CONFIG_PROC_EVENTS and can't provide split tunnel. There are a few other errors you can get, but they are rare (ancient version of iptables, etc.)

If your kernel does lack CONFIG_PROC_EVENTS, the only solution we have is to install (or build) a kernel with CONFIG_PROC_EVENTS; it's currently required for split tunnel. If you get one of the other errors, we can figure out what's different between these two systems. If you only want subnet rules, you might be able to hack out the test for proc events, but be aware PIA won't prevent you from creating app rules that won't completely work (https://github.com/pia-foss/desktop/blob/master/daemon/src/posix/posix_daemon.cpp#L1057)

[piramiday]

thanks for the prompt reply!

the two systems have the same OS and kernel. the splitTunnelSupportErrors seem to be pointing in the right direction! the working system returns an empty list, whereas the failing one returns libnl_invalid.

I have re-checked the APT packages and noticed that libnl-genl-3-200 was present in the former system but missing in the latter. I installed it and -- thankfully -- the iptables differences have disappeared and split tunneling works as expected.

for comparison, the old system had:

$ apt list --installed | grep libnl # always worked

libnl-3-200/focal,now 3.4.0-1 arm64 [installed]
libnl-genl-3-200/focal,now 3.4.0-1 arm64 [installed,auto-removable]
libnl-route-3-200/focal,now 3.4.0-1 arm64 [installed]

while the new system now has:

$ apt list --installed | grep libnl # was failing, now works

libnl-3-200/focal,now 3.4.0-1 arm64 [installed]
libnl-genl-3-200/focal,now 3.4.0-1 arm64 [installed]
libnl-route-3-200/focal,now 3.4.0-1 arm64 [installed]

therefore libnl-genl-3-200 was automatically installed in the former system, but skipped in the latter until manually installed.

I am happy this is now solved, but... is there maybe a bug in the .run installer? how come this crucial library has not been installed? of course, the working system has been connecting for a while and it was not installed using v3.3.0, but an earlier version.

[JonathonH-PIA]

Wow, that's great sleuthing, great find! That was one of the "rare" errors I didn't expect to see :grin:

You're right, it looks like the installer has always been missing libnl-genl-3-200 for Debian/Ubuntu/etc. (https://github.com/pia-foss/desktop/blob/master/extras/installer/linux/linux_installer.sh#L217) I'll get this on our internal tracker to fix.

We probably never noticed it because so many things depend on it normally (such as 802.11 support), so it's a bit challenging to set up a system without pulling in that package! It should definitely be in the installer though, thanks for reporting this!