search

ivpn / desktop-app

Official IVPN Desktop app
https://www.ivpn.net/apps/
GNU General Public License v3.0
363 stars 50 forks source link

[Linux] Raspberry Pi OS: firewall fails on iptables + --match cgroup #223

Closed jordan-ivpn closed 1 year ago

jordan-ivpn commented 1 year ago

Bug report

Describe your environment

Describe the problem

Using Raspberry Pi OS Lite (Release date: September 22nd 2022) based on Debian bullseye from: https://www.raspberrypi.com/software/operating-systems/ https://downloads.raspberrypi.org/raspios_lite_armhf/images/raspios_lite_armhf-2022-09-26/2022-09-22-raspios-bullseye-armhf-lite.img.xz sha256sum: 9bf5234efbadd2d39769486e0a20923d8526a45eba57f74cda45ef78e2b628da

# ivpn firewall -on
Error: failed to change firewall state : failed to execute shell command: exit status 2
root@raspberrypi:/opt/ivpn/etc# ./firewall.sh -enable  
...
+ ip6tables -w 2 -I OUTPUT -j IVPN-OUT
+ ip6tables -w 2 -I INPUT -j IVPN-IN
+ ip6tables -w 2 -I OUTPUT -m cgroup --cgroup 0x4956504e -m comment --comment 'IVPN Split Tunneling' -j ACCEPT
ip6tables v1.8.7 (nf_tables): Couldn't load match `cgroup':No such file or directory

The issue is not present on a working system that uses an older version of the Raspberry Pi OS Desktop (May 2021) based on Debian buster.

Original cgroups hierarchy details on the non-working system are different from the working system:

# findmnt -lo source,target,fstype,options -t cgroup,cgroup2
SOURCE  TARGET                 FSTYPE  OPTIONS
cgroup2 /sys/fs/cgroup         cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot
net_cls /sys/fs/cgroup/net_cls cgroup  rw,relatime,net_cls

Possible to change cgroup hierarchy to match the working system. https://raspberrypi.stackexchange.com/a/137994

Edit /boot/cmdline.txt to add the systemd.unified_cgroup_hierarchy=0 bit at the end, then reboot:

# cat /boot/cmdline.txt 
console=serial0,115200 console=tty1 root=PARTUUID=... rootfstype=ext4 fsck.repair=yes rootwait systemd.unified_cgroup_hierarchy=0

Confirm hierarchy on the non-working system has changed after reboot:

# cat /proc/cmdline 
coherent_pool=1M 8250.nr_uarts=0 snd_bcm2835.enable_compat_alsa=0 snd_bcm2835.enable_hdmi=1 video=HDMI-A-1:1680x1050M@60 vc_mem.mem_base=0x3ec00000 vc_mem.mem_size=0x40000000  console=ttyS0,115200 console=tty1 root=... rootfstype=ext4 fsck.repair=yes rootwait systemd.unified_cgroup_hierarchy=0

After changing the GRUB cmdline on the non-working system:

# findmnt -lo source,target,fstype,options -t cgroup,cgroup2
SOURCE  TARGET                          FSTYPE  OPTIONS
cgroup2 /sys/fs/cgroup/unified          cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate
cgroup  /sys/fs/cgroup/systemd          cgroup  rw,nosuid,nodev,noexec,relatime,xattr,name=systemd
cgroup  /sys/fs/cgroup/cpuset           cgroup  rw,nosuid,nodev,noexec,relatime,cpuset
cgroup  /sys/fs/cgroup/devices          cgroup  rw,nosuid,nodev,noexec,relatime,devices
cgroup  /sys/fs/cgroup/cpu,cpuacct      cgroup  rw,nosuid,nodev,noexec,relatime,cpu,cpuacct
cgroup  /sys/fs/cgroup/net_cls,net_prio cgroup  rw,nosuid,nodev,noexec,relatime,net_cls,net_prio
cgroup  /sys/fs/cgroup/perf_event       cgroup  rw,nosuid,nodev,noexec,relatime,perf_event
cgroup  /sys/fs/cgroup/pids             cgroup  rw,nosuid,nodev,noexec,relatime,pids
cgroup  /sys/fs/cgroup/blkio            cgroup  rw,nosuid,nodev,noexec,relatime,blkio
cgroup  /sys/fs/cgroup/freezer          cgroup  rw,nosuid,nodev,noexec,relatime,freezer

I have changed the cgroup hierarchy on the non-working system to match the hierarchy on the working system (hierarchies are the same), but the issue persists, so the issue does not seem to be systemd.unified_cgroup_hierarchy related.

Non-working system:

# cat /proc/net/ip_tables_matches
udplite
udp
tcp
icmp

Working system (not connected is same as connected with firewall ON + lan allow):

# cat /proc/net/ip_tables_matches
connmark
mark
comment
addrtype
udplite
udp
tcp
addrtype
icmp

Same list on both systems, hierarchy numbers are different. This is from the-non-working system:

cat /proc/cgroups
#subsys_name    hierarchy       num_cgroups     enabled
cpuset  2       1       1
cpu     4       1       1
cpuacct 4       1       1
blkio   8       1       1
memory  0       40      0
devices 3       32      1
freezer 9       1       1
net_cls 5       1       1
perf_event      6       1       1
net_prio        5       1       1
pids    7       37      1

Not-working:

# lsmod | grep tables
nf_tables             217088  90 nft_compat,nft_counter
nfnetlink              20480  2 nft_compat,nf_tables
ip_tables              28672  0
x_tables               32768  3 ip_tables,nft_compat,xt_tcpudp

Working:

# lsmod | grep tables
nf_tables             196608  20 nft_compat,nft_counter
nfnetlink              16384  2 nft_compat,nf_tables
ip_tables              28672  0
x_tables               32768  7 xt_comment,ip_tables,nft_compat,xt_mark,xt_tcpudp,xt_connmark,xt_addrtype

nftables is not installed on the working system. It is installed by default on the non-working system.

Not-working:

# iptables -V
iptables v1.8.7 (nf_tables)

# nft -V
nftables v0.9.8 (E.D.S.)
  cli:          editline
  json:         yes
  minigmp:      no
  libxtables:   yes

# apt-cache policy libxtables12
libxtables12:
  Installed: 1.8.7-1
  Candidate: 1.8.7-1
  Version table:
 *** 1.8.7-1 500
        500 http://raspbian.raspberrypi.org/raspbian bullseye/main armhf Packages
        100 /var/lib/dpkg/status

# apt-cache policy libnftables1
libnftables1:
  Installed: 0.9.8-3.1
  Candidate: 0.9.8-3.1
  Version table:
 *** 0.9.8-3.1 500
        500 http://raspbian.raspberrypi.org/raspbian bullseye/main armhf Packages
        100 /var/lib/dpkg/status

Working (no further upgrade is available):

# iptables -V
iptables v1.8.2 (nf_tables)

# apt-cache policy nftables
nftables:
  Installed: (none)
  Candidate: 0.9.6-1~bpo10+1~0
  Version table:
     0.9.6-1~bpo10+1~0 500
        500 http://archive.raspberrypi.org/debian buster/main armhf Packages
     0.9.0-2 500
        500 http://raspbian.raspberrypi.org/raspbian buster/main armhf Packages

# apt-cache policy libxtables12 
libxtables12:
  Installed: 1.8.2-4
  Candidate: 1.8.2-4
  Version table:
 *** 1.8.2-4 500
        500 http://raspbian.raspberrypi.org/raspbian buster/main armhf Packages
        100 /var/lib/dpkg/status

# apt-cache policy libnftables1
libnftables1:
  Installed: (none)
  Candidate: 0.9.6-1~bpo10+1~0
  Version table:
     0.9.6-1~bpo10+1~0 500
        500 http://archive.raspberrypi.org/debian buster/main armhf Packages

Raspberry Pi OS project provides a 32-bit Desktop version of their OS for x86 systems. Download at the bottom of the page: https://www.raspberrypi.com/software/operating-systems/ https://downloads.raspberrypi.org/rpd_x86/images/rpd_x86-2022-07-04/2022-07-01-raspios-bullseye-i386.iso sha256sum: 5fa906df25e600bf7d7e6a5eb7b0e9b6605e60992ee6c8efe79bc99e7c2452bd

Using our build guide to compile and install ivpn base package, the issue is not present: https://www.ivpn.net/knowledgebase/linux/building-the-ivpn-cli-client-for-linux-on-a-raspberry-pi/

This x86 installation includes the X server, a desktop environment, and GUI applications; there are 1936 installed packages. The non-working ARM installation only has 624 packages. Package lists for each system are included just in case it helps (ARM = not working, x86 = working).

arm-installed.txt x86-installed.txt

jordan-ivpn commented 1 year ago

Tested the latest ARM desktop Rasp Pi OS release and the firewall issue is still present. This suggests that the issue is not related to a missing package.

An issue is also present with the splittun feature.

# ivpn splittun -appadd /usr/bin/chromium-browser
Error: unable to run application in Split Tunneling environment: Split Tunneling is disabled

# ivpn splittun -on
Split Tunnel : Enabled

# ivpn splittun -appadd /usr/bin/chromium-browser
Error: failed to enable Split Tunneling: (exit status 2) Warning: Default IPv6 gateway is not defined.
iptables v1.8.7 (nf_tables): Couldn't load match `cgroup':No such file or directory
Try `iptables -h' or 'iptables --help' for more information.

I had disabled IPv6 on this device. I'll re-enable it to see if the error goes away.

Seems to be the same issue with the iptables cgroup match:

$ ivpn splittun -appadd /usr/bin/chromium-browser
Error: failed to enable Split Tunneling: (exit status 2) iptables v1.8.7 (nf_tables): Couldn't load match `cgroup':No such file or directory
Try `iptables -h' or 'iptables --help' for more information.

Added the -x option to the splittun.sh and firewall.sh scripts for debugging:

# ivpn splittun -appadd /usr/bin/chromium-browser                                                                
Error: failed to enable Split Tunneling: (exit status 2) + _cgroup_name=ivpn-exclude                                                           
+ _cgroup_classid=0x4956504e                                                                                                                   
+ _cgroup_folder=/sys/fs/cgroup/net_cls/ivpn-exclude
jordan-ivpn commented 1 year ago

Pointer to report from Reddit: https://www.reddit.com/r/IVPN/comments/yt2m0q/cant_use_ivpn_with_firewall_on_raspberry_pi_os/

stenya commented 1 year ago

The firewall script is fixed. Now the IVPN firewall will not fail if allowing rules for split-tunnel are unable to apply. The change will be available in the next official release of the IVPN client.

But this change does not fix Split-Tunnel functionality. Split-Tunneling does not work on the latest 'Raspberry Pi OS Lite'. The iptables ... -m cgroup --cgroup ... keep failing. It seems, the cgroup match module for iptables does not work (or is not installed) for some reason on 'Raspberry Pi OS Lite'.

stenya commented 1 year ago

I am going to close the ticket since the IVPN firewall issue is fixed. We can create a separate ticket related to the Split-Tunnel issue on "Raspberry PI OS Lite"

stenya commented 1 year ago

@gorkapernas v3.9.53. The chanbge is ninimal. I do not expect any side effects. Simple regression check required: FW should work as before.

gorkapernas commented 1 year ago

Verified on v3.9.53, run a sanity on a couple of Linux distros (Ubuntu 20.04 and Fedora 37) with especial attention on the FW feature, everything works as expected.

stenya commented 1 year ago

Released in v3.10.0 Beta