search

stangri / repo.openwrt.melmac.net

OpenWrt/LEDE Project Packages Repository
97 stars 6 forks source link

Cannot get pbr working reliably with 3 VPNs #19

Closed usr97629238 closed 1 year ago

usr97629238 commented 1 year ago

Trying to get pbr working using wan and 3 VPNs (1 Wireguard (tun_w) and 2 OpenVpn (tun_o1, tun_o2)). I'm on OpenWrt 22.03.3 x86/64, tried both with and without the dnsmasq-full's nft sets. Used to work fine on previous OpenWrt versions using the vpn-policy-routing. No matter what I try, I'm only able to switch between wan and the tun_w. The strange thing is that occassionaly assigning the iterface to one of the tun_o works for some reason, but then it goes down by itself.

All in all, it goes as follows:

pbr setting:  effective:
tun_o1        tun_w
tun_o2        tun_w (sometimes tun_o1 (not a typo))
tun_w         tun_w
br-wan        br-wan

All the interfaces are up and the host is able to access them just fine: 

curl --interface tun_o1 https://ipecho.net/plain/
169.1...
curl --interface tun_o2 https://ipecho.net/plain/
163.1...
curl --interface tun_w https://ipecho.net/plain/
104.2...
curl --interface br-wan https://ipecho.net/plain/
109.2...

And that's a test fresh install, just installed pbr and configured the interfaces.

The relevant configs are as follows:

/etc/config/pbr:
config pbr 'config'
        option strict_enforcement '1'
        option resolver_set 'none'
        option ipv6_enabled '0'
        option boot_timeout '30'
        option rule_create_option 'add'
        option procd_reload_delay '1'
        list webui_supported_protocol 'all'
        list webui_supported_protocol 'tcp'
        list webui_supported_protocol 'udp'
        list webui_supported_protocol 'tcp udp'
        list webui_supported_protocol 'icmp'
        option verbosity '1'
        list ignored_interface 'vpnserver'
        list ignored_interface 'wgserver'
        list ignored_interface 'eth0 br-lan'
        option webui_show_ignore_target '0'
        list supported_interface 'wan'
        list supported_interface 'tun_o1'
        list supported_interface 'tun_o2'
        list supported_interface 'tun_w'
        option enabled '1'

config policy
        option name 'test'
        option dest_addr 'ipleak.net ipecho.net'
        option interface 'vpn_o1'

/etc/config/network:
config interface 'vpn_o2'
        option proto 'none'
        option device 'tun_o2'

config interface 'vpn_o1'
        option proto 'none'
        option device 'tun_o1'

config interface 'vpn_w'
        option proto 'none'
        option device 'tun_w'

config device
        option name 'tun_o2'
        option ipv6 '0'

config device
        option name 'tun_o1'
        option ipv6 '0'

config device
        option name 'tun_w'
        option ipv6 '0'

/etc/config/firewall:
config zone
        option name 'vpn_w'
        option output 'ACCEPT'
        option masq '1'
        option mtu_fix '1'
        list network 'vpn_w'
        option input 'REJECT'
        option forward 'REJECT'

config forwarding
        option src 'lan'
        option dest 'vpn_w'

config zone
        option name 'vpn_o1'
        option output 'ACCEPT'
        option masq '1'
        option mtu_fix '1'
        list network 'vpn_o1'
        option input 'REJECT'
        option forward 'REJECT'

config zone
        option name 'vpn_o2'
        option output 'ACCEPT'
        option masq '1'
        option mtu_fix '1'
        list network 'vpn_o2'
        option input 'REJECT'
        option forward 'REJECT'

config forwarding
        option src 'lan'
        option dest 'vpn_o1'

config forwarding
        option src 'lan'
        option dest 'vpn_o2'

route:
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         172.1-----      192.0.0.0       UG    0      0        0 tun_w
default         10.0.0.1        0.0.0.0         UG    0      0        0 br-wan
10.0.0.0        *               255.255.252.0   U     0      0        0 br-lan
10.0.0.0        *               255.255.252.0   U     0      0        0 br-wan
10.1-----       *               255.255.0.0     U     0      0        0 tun_o1
10.4-----       *               255.255.248.0   U     0      0        0 tun_o2
64.0.0.0        172.1-----      192.0.0.0       UG    0      0        0 tun_w
128.0.0.0       172.1-----      192.0.0.0       UG    0      0        0 tun_w
162.1-----      10.0.0.1        255.255.255.255 UGH   0      0        0 br-lan
172.1-----      *               255.255.255.0   U     0      0        0 tun_w
192.0.0.0       172.1-----      192.0.0.0       UG    0      0        0 tun_w
224.0.0.0       10.0.0.1        240.0.0.0       UG    0      0        0 br-lan

Not sure what is possibly wrong with my config.

stangri commented 1 year ago

Please refer to the README's Help section and provide all information requested there.

usr97629238 commented 1 year ago

content of /etc/config/dhcp (stock actually)

config dnsmasq
        option domainneeded '1'
        option boguspriv '1'
        option filterwin2k '0'
        option localise_queries '1'
        option rebind_protection '1'
        option rebind_localhost '1'
        option local '/lan/'
        option domain 'lan'
        option expandhosts '1'
        option nonegcache '0'
        option authoritative '1'
        option readethers '1'
        option leasefile '/tmp/dhcp.leases'
        option resolvfile '/tmp/resolv.conf.d/resolv.conf.auto'
        option nonwildcard '1'
        option localservice '1'
        option ednspacket_max '1232'

config dhcp 'lan'
        option interface 'lan'
        option start '100'
        option limit '150'
        option leasetime '12h'
        option dhcpv4 'server'
        option dhcpv6 'server'
        option ra 'server'
        option ra_slaac '1'
        list ra_flags 'managed-config'
        list ra_flags 'other-config'

config dhcp 'wan'
        option interface 'wan'
        option ignore '1'

config odhcpd 'odhcpd'
        option maindhcp '0'
        option leasefile '/tmp/hosts/odhcpd'
        option leasetrigger '/usr/sbin/odhcpd-update'
        option loglevel '4'

content of /etc/config/firewall (nothing except added interfaces)

config defaults
    option synflood_protect '1'
    option input 'ACCEPT'
    option output 'ACCEPT'
    option forward 'REJECT'
    option disable_ipv6 '1'

config zone
    option name 'lan'
    list network 'lan'
    option input 'ACCEPT'
    option output 'ACCEPT'
    option forward 'ACCEPT'

config zone
    option name 'wan'
    list network 'wan'
    list network 'wan6'
    option input 'REJECT'
    option output 'ACCEPT'
    option forward 'REJECT'
    option masq '1'
    option mtu_fix '1'

config forwarding
    option src 'lan'
    option dest 'wan'

config rule
    option name 'Allow-DHCP-Renew'
    option src 'wan'
    option proto 'udp'
    option dest_port '68'
    option target 'ACCEPT'
    option family 'ipv4'

config rule
    option name 'Allow-Ping'
    option src 'wan'
    option proto 'icmp'
    option icmp_type 'echo-request'
    option family 'ipv4'
    option target 'ACCEPT'

config rule
    option name 'Allow-IGMP'
    option src 'wan'
    option proto 'igmp'
    option family 'ipv4'
    option target 'ACCEPT'

config rule
    option name 'Allow-DHCPv6'
    option src 'wan'
    option proto 'udp'
    option dest_port '546'
    option family 'ipv6'
    option target 'ACCEPT'

config rule
    option name 'Allow-MLD'
    option src 'wan'
    option proto 'icmp'
    option src_ip 'fe80::/10'
    list icmp_type '130/0'
    list icmp_type '131/0'
    list icmp_type '132/0'
    list icmp_type '143/0'
    option family 'ipv6'
    option target 'ACCEPT'

config rule
    option name 'Allow-ICMPv6-Input'
    option src 'wan'
    option proto 'icmp'
    list icmp_type 'echo-request'
    list icmp_type 'echo-reply'
    list icmp_type 'destination-unreachable'
    list icmp_type 'packet-too-big'
    list icmp_type 'time-exceeded'
    list icmp_type 'bad-header'
    list icmp_type 'unknown-header-type'
    list icmp_type 'router-solicitation'
    list icmp_type 'neighbour-solicitation'
    list icmp_type 'router-advertisement'
    list icmp_type 'neighbour-advertisement'
    option limit '1000/sec'
    option family 'ipv6'
    option target 'ACCEPT'

config rule
    option name 'Allow-ICMPv6-Forward'
    option src 'wan'
    option dest '*'
    option proto 'icmp'
    list icmp_type 'echo-request'
    list icmp_type 'echo-reply'
    list icmp_type 'destination-unreachable'
    list icmp_type 'packet-too-big'
    list icmp_type 'time-exceeded'
    list icmp_type 'bad-header'
    list icmp_type 'unknown-header-type'
    option limit '1000/sec'
    option family 'ipv6'
    option target 'ACCEPT'

config rule
    option name 'Allow-IPSec-ESP'
    option src 'wan'
    option dest 'lan'
    option proto 'esp'
    option target 'ACCEPT'

config rule
    option name 'Allow-ISAKMP'
    option src 'wan'
    option dest 'lan'
    option dest_port '500'
    option proto 'udp'
    option target 'ACCEPT'

config zone
    option name 'vpn_w'
    option output 'ACCEPT'
    option masq '1'
    option mtu_fix '1'
    list network 'vpn_w'
    option input 'REJECT'
    option forward 'REJECT'

config forwarding
    option src 'lan'
    option dest 'vpn_w'

config zone
    option name 'vpn_o1'
    option output 'ACCEPT'
    option masq '1'
    option mtu_fix '1'
    list network 'vpn_o1'
    option input 'REJECT'
    option forward 'REJECT'

config zone
    option name 'vpn_o2'
    option output 'ACCEPT'
    option masq '1'
    option mtu_fix '1'
    list network 'vpn_o2'
    option input 'REJECT'
    option forward 'REJECT'

config forwarding
    option src 'lan'
    option dest 'vpn_o1'

config forwarding
    option src 'lan'
    option dest 'vpn_o2'

config include 'pbr'
    option fw4_compatible '1'
    option type 'script'
    option path '/usr/share/pbr/pbr.firewall.include'

content of /etc/config/network (same here)

config interface 'loopback'
    option device 'lo'
    option proto 'static'
    option ipaddr '127.0.0.1'
    option netmask '255.0.0.0'

config device
    option name 'br-lan'
    option type 'bridge'
    list ports 'eth0'
    option ipv6 '0'

config device
    option name 'br-wan'
    option type 'bridge'
    list ports 'eth1'
    option ipv6 '0'

config interface 'lan'
    option device 'br-lan'
    option proto 'static'
    option ipaddr '10.0.0.13'
    option netmask '255.255.252.0'
    option gateway '10.0.0.2'
    list dns '1.1.1.1'
    option ip6assign '60'
    option delegate '0'

config interface 'wan'
    option proto 'static'
    option netmask '255.255.252.0'
    option ipaddr '10.0.0.12'
    option gateway '10.0.0.1'
    list dns '1.1.1.1'
    option device 'br-wan'

config route
    option gateway '10.0.0.1'
    option interface 'lan'
    option onlink '1'
    option netmask '240.0.0.0'
    option target '224.0.0.0'

config interface 'vpn_o2'
    option proto 'none'
    option device 'tun_o2'

config interface 'vpn_o1'
    option proto 'none'
    option device 'tun_o1'

config interface 'vpn_w'
    option proto 'none'
    option device 'tun_w'

config device
    option name 'tun_o2'
    option ipv6 '0'

config device
    option name 'tun_o1'
    option ipv6 '0'

config device
    option name 'tun_w'
    option ipv6 '0'

content of /etc/config/pbr

config pbr 'config'
        option strict_enforcement '1'
        option resolver_set 'none'
        option ipv6_enabled '0'
        option boot_timeout '30'
        option rule_create_option 'add'
        option procd_reload_delay '1'
        list webui_supported_protocol 'all'
        list webui_supported_protocol 'tcp'
        list webui_supported_protocol 'udp'
        list webui_supported_protocol 'tcp udp'
        list webui_supported_protocol 'icmp'
        option verbosity '2'
        list ignored_interface 'vpnserver'
        list ignored_interface 'wgserver'
        list ignored_interface 'eth0 br-lan'
        option webui_show_ignore_target '0'
        list supported_interface 'wan'
        list supported_interface 'tun_o1'
        list supported_interface 'tun_o2'
        list supported_interface 'tun_w'
        option enabled '1'

config policy
        option name 'test'
        option dest_addr 'ipleak.net ipecho.net'
        option interface 'vpn_o2'

the output of /etc/init.d/pbr status

============================================================
pbr - environment
pbr 1.0.1-16 running on OpenWrt 22.03.3. WAN (IPv4): lan/br-lan/10.0.0.2.
============================================================
Dnsmasq version 2.86  Copyright (c) 2000-2021 Simon Kelley
Compile time options: IPv6 GNU-getopt no-DBus UBus no-i18n no-IDN DHCP no-DHCPv6 no-Lua TFTP no-conntrack no-ipset no-auth no-cryptohash no-DNSSEC no-ID loop-detect inotify dumpfile
============================================================
pbr chains - policies
        chain pbr_forward {
        }
        chain pbr_input {
        }
        chain pbr_output {
        }
        chain pbr_prerouting {
                ip daddr @pbr_vpn_o2_4_dst_ip_cfg026ff5 goto pbr_mark_0x020000 comment "test"
        }
        chain pbr_postrouting {
        }
============================================================
pbr chains - marking
        chain pbr_mark_0x010000 {
                counter packets 0 bytes 0 meta mark set meta mark & 0xff01ffff | 0x00010000
                return
        }
        chain pbr_mark_0x020000 {
                counter packets 12 bytes 800 meta mark set meta mark & 0xff02ffff | 0x00020000
                return
        }
        chain pbr_mark_0x030000 {
                counter packets 0 bytes 0 meta mark set meta mark & 0xff03ffff | 0x00030000
                return
        }
        chain pbr_mark_0x040000 {
                counter packets 0 bytes 0 meta mark set meta mark & 0xff04ffff | 0x00040000
                return
        }
============================================================
pbr nft sets
        set pbr_vpn_o2_4_dst_ip_cfg026ff5 {
                type ipv4_addr
                flags interval
                auto-merge
                comment "test"
                elements = { 34.160.111.145, 95.85.16.212 }
        }
============================================================
IPv4 table 256 route: default via 10.0.0.1 dev br-wan
IPv4 table 256 rule(s):
30000:  from all fwmark 0x10000/0xff0000 lookup pbr_wan
IPv4 table 257 route: default via 10.4------ dev tun_o2
IPv4 table 257 rule(s):
30001:  from all fwmark 0x20000/0xff0000 lookup pbr_vpn_o2
IPv4 table 258 route: default via 10.1------ dev tun_o1
IPv4 table 258 rule(s):
30002:  from all fwmark 0x30000/0xff0000 lookup pbr_vpn_o1
IPv4 table 259 route: default via 172.1------ dev tun_w
IPv4 table 259 rule(s):
30001:  from all fwmark 0x20000/0xff0000 lookup pbr_vpn_w
30002:  from all fwmark 0x30000/0xff0000 lookup pbr_vpn_w
30003:  from all fwmark 0x40000/0xff0000 lookup pbr_vpn_w

the output of /etc/init.d/pbr reload with verbosity setting set to 2

Activating traffic killswitch [✓]
Setting up routing for 'wan/br-wan/10.0.0.1' [✓]
Setting up routing for 'vpn_o2/tun_o2/10.4-------' [✓]
Setting up routing for 'vpn_o1/tun_o1/10.1-------' [✓]
Setting up routing for 'vpn_w/tun_w/172.1-------' [✓]
Routing 'test' via vpn_o2 [✓]
Deactivating traffic killswitch [✓]
pbr 1.0.1-16 monitoring interfaces: wan vpn_o2 vpn_o1 vpn_w
pbr 1.0.1-16 (nft) started with gateways:
wan/br-wan/10.0.0.1
vpn_o2/tun_o2/10.4-------
vpn_o1/tun_o1/10.1-------
vpn_w/tun_w/172.1------- [✓]
stangri commented 1 year ago

Please elaborate on the issue. ELIM5.

usr97629238 commented 1 year ago

Like I've said:

  1. All the 4 interfaces are up and running fine:

    curl --interface tun_o1 https://ipecho.net/plain/
169.1...
curl --interface tun_o2 https://ipecho.net/plain/
163.1...
curl --interface tun_w https://ipecho.net/plain/
104.2...
curl --interface br-wan https://ipecho.net/plain/
109.2...

  2. However, I'm unable to configure pbr to route access for some other computer B (to which that OpenWrt host acts as a gateway) through either tun_o1 or tun_o2, the only working options are wan or tun_w. I mean this setting: config policy option name 'test' option dest_addr 'ipleak.net ipecho.net' option interface 'vpn_o2'

  3. Trying to set it to either tun_o1 or tun_o2 results in routing through tun_w.

  4. Sometimes though setting it to tun_o2 works for sometime, although through tun_o1

  5. All in all, it goes as follows: pbr setting: effective:

    pbr setting:  effective route:
tun_o1        tun_w
tun_o2        tun_w (sometimes tun_o1 (not a typo))
tun_w         tun_w
br-wan        br-wan

    Hope that helps

Edit: Here is an example:

  1. I set pbr to route through wan interface and the external ip on computer B is 109.2...
  2. pbr to vpn_o1: 104.2....
  3. pbr to vpn_o2: 104.2....
  4. pbr to vpn_w: 104.2....

And nevermind me using the tun and vpn interchangeably.

stangri commented 1 year ago

In the flawed test example posted above (check README for information on how to properly target domains in policies), I see that the packets for the pbr_vpn_o2_4_dst_ip_cfg026ff5 set are leaving via tun_02:

IPv4 table 257 route: default via 10.4------ dev tun_o2
IPv4 table 257 rule(s):
30001:  from all fwmark 0x20000/0xff0000 lookup pbr_vpn_o2

According to the nft set counter, I see 12 packets left that way.

usr97629238 commented 1 year ago

Ok, just went through them all

wan:
============================================================
pbr chains - marking
        chain pbr_mark_0x010000 {
                counter packets 6 bytes 400 meta mark set meta mark & 0xff01ffff | 0x00010000
                return
        }
        chain pbr_mark_0x020000 {
                counter packets 0 bytes 0 meta mark set meta mark & 0xff02ffff | 0x00020000
                return
        }
        chain pbr_mark_0x030000 {
                counter packets 0 bytes 0 meta mark set meta mark & 0xff03ffff | 0x00030000
                return
        }
        chain pbr_mark_0x040000 {
                counter packets 0 bytes 0 meta mark set meta mark & 0xff04ffff | 0x00040000
                return
        }
============================================================
pbr nft sets
        set pbr_wan_4_dst_ip_cfg026ff5 {
                type ipv4_addr
                flags interval
                auto-merge
                comment "test"
                elements = { 34.160.111.145, 95.85.16.212 }
        }
============================================================
IPv4 table 256 route: default via 10.0.0.1 dev br-wan
IPv4 table 256 rule(s):
30000:  from all fwmark 0x10000/0xff0000 lookup pbr_wan
IPv4 table 257 route: default via 10.4------ dev tun_o2
IPv4 table 257 rule(s):
30001:  from all fwmark 0x20000/0xff0000 lookup pbr_vpn_o2
IPv4 table 258 route: default via 10.1------ dev tun_o1
IPv4 table 258 rule(s):
30002:  from all fwmark 0x30000/0xff0000 lookup pbr_vpn_o1
IPv4 table 259 route: default via 172.1------ dev tun_w

vpn_o1:
============================================================
pbr chains - marking
        chain pbr_mark_0x010000 {
                counter packets 0 bytes 0 meta mark set meta mark & 0xff01ffff | 0x00010000
                return
        }
        chain pbr_mark_0x020000 {
                counter packets 0 bytes 0 meta mark set meta mark & 0xff02ffff | 0x00020000
                return
        }
        chain pbr_mark_0x030000 {
                counter packets 6 bytes 400 meta mark set meta mark & 0xff03ffff | 0x00030000
                return
        }
        chain pbr_mark_0x040000 {
                counter packets 0 bytes 0 meta mark set meta mark & 0xff04ffff | 0x00040000
                return
        }
============================================================
pbr nft sets
        set pbr_vpn_o1_4_dst_ip_cfg026ff5 {
                type ipv4_addr
                flags interval
                auto-merge
                comment "test"
                elements = { 34.160.111.145, 95.85.16.212 }
        }
============================================================
IPv4 table 256 route: default via 10.0.0.1 dev br-wan
IPv4 table 256 rule(s):
30000:  from all fwmark 0x10000/0xff0000 lookup pbr_wan
IPv4 table 257 route: default via 10.4------ dev tun_o2
IPv4 table 257 rule(s):
30001:  from all fwmark 0x20000/0xff0000 lookup pbr_vpn_o2
IPv4 table 258 route: default via 10.1------ dev tun_o1
IPv4 table 258 rule(s):
30002:  from all fwmark 0x30000/0xff0000 lookup pbr_vpn_o1
IPv4 table 259 route: default via 172.1------ dev tun_w
IPv4 table 259 rule(s):
30001:  from all fwmark 0x20000/0xff0000 lookup pbr_vpn_w
30002:  from all fwmark 0x30000/0xff0000 lookup pbr_vpn_w
30003:  from all fwmark 0x40000/0xff0000 lookup pbr_vpn_w

vpn_o2:
============================================================
pbr chains - marking
        chain pbr_mark_0x010000 {
                counter packets 0 bytes 0 meta mark set meta mark & 0xff01ffff | 0x00010000
                return
        }
        chain pbr_mark_0x020000 {
                counter packets 6 bytes 400 meta mark set meta mark & 0xff02ffff | 0x00020000
                return
        }
        chain pbr_mark_0x030000 {
                counter packets 0 bytes 0 meta mark set meta mark & 0xff03ffff | 0x00030000
                return
        }
        chain pbr_mark_0x040000 {
                counter packets 0 bytes 0 meta mark set meta mark & 0xff04ffff | 0x00040000
                return
        }
============================================================
pbr nft sets
        set pbr_vpn_o2_4_dst_ip_cfg026ff5 {
                type ipv4_addr
                flags interval
                auto-merge
                comment "test"
                elements = { 34.160.111.145, 95.85.16.212 }
        }
============================================================
IPv4 table 256 route: default via 10.0.0.1 dev br-wan
IPv4 table 256 rule(s):
30000:  from all fwmark 0x10000/0xff0000 lookup pbr_wan
IPv4 table 257 route: default via 10.4------ dev tun_o2
IPv4 table 257 rule(s):
30001:  from all fwmark 0x20000/0xff0000 lookup pbr_vpn_o2
IPv4 table 258 route: default via 10.1------ dev tun_o1
IPv4 table 258 rule(s):
30002:  from all fwmark 0x30000/0xff0000 lookup pbr_vpn_o1
IPv4 table 259 route: default via 172.1------ dev tun_w
IPv4 table 259 rule(s):
30001:  from all fwmark 0x20000/0xff0000 lookup pbr_vpn_w
30002:  from all fwmark 0x30000/0xff0000 lookup pbr_vpn_w
30003:  from all fwmark 0x40000/0xff0000 lookup pbr_vpn_w

vpn_w:
============================================================
pbr chains - marking
        chain pbr_mark_0x010000 {
                counter packets 0 bytes 0 meta mark set meta mark & 0xff01ffff | 0x00010000
                return
        }
        chain pbr_mark_0x020000 {
                counter packets 0 bytes 0 meta mark set meta mark & 0xff02ffff | 0x00020000
                return
        }
        chain pbr_mark_0x030000 {
                counter packets 0 bytes 0 meta mark set meta mark & 0xff03ffff | 0x00030000
                return
        }
        chain pbr_mark_0x040000 {
                counter packets 6 bytes 400 meta mark set meta mark & 0xff04ffff | 0x00040000
                return
        }
============================================================
pbr nft sets
        set pbr_vpn_w_4_dst_ip_cfg026ff5 {
                type ipv4_addr
                flags interval
                auto-merge
                comment "test"
                elements = { 34.160.111.145, 95.85.16.212 }
        }
============================================================
IPv4 table 256 route: default via 10.0.0.1 dev br-wan
IPv4 table 256 rule(s):
30000:  from all fwmark 0x10000/0xff0000 lookup pbr_wan
IPv4 table 257 route: default via 10.4------ dev tun_o2
IPv4 table 257 rule(s):
30001:  from all fwmark 0x20000/0xff0000 lookup pbr_vpn_o2
IPv4 table 258 route: default via 10.1------ dev tun_o1
IPv4 table 258 rule(s):
30002:  from all fwmark 0x30000/0xff0000 lookup pbr_vpn_o1
IPv4 table 259 route: default via 172.1------ dev tun_w
IPv4 table 259 rule(s):
30001:  from all fwmark 0x20000/0xff0000 lookup pbr_vpn_w
30002:  from all fwmark 0x30000/0xff0000 lookup pbr_vpn_w
30003:  from all fwmark 0x40000/0xff0000 lookup pbr_vpn_w

And it says that yes, the packets are going through the right interface, however in reality they went (judging by curl https://ipecho.net/plain/) through wan for the wan setting and through vpn_w for all the 3 other settings.

usr97629238 commented 1 year ago

Did some more tests, and it's actually not necessary to have two OpenVpn VPNs for it to fail. It fails even with one, although less often. And I haven't figured out what causes it, just switching the VPN on and off, sometimes it works, sometimes it fails.

Here is the config for Proton VPN that I've been using for testing (in case something is wrong with it):

client
dev tun_o1
proto udp
remote nl-free-44.protonvpn.net 4569
keepalive 10 60
resolv-retry infinite
nobind
cipher AES-256-CBC
auth SHA512
comp-lzo no
verb 3
tun-mtu 1500
tun-mtu-extra 32
mssfix 1450
persist-key
persist-tun
reneg-sec 3600
remote-cert-tls server
auth-user-pass /etc/openvpn/proton.auth
pull
fast-io
float
pull-filter ignore "redirect-gateway"
script-security 2
key-direction 1
<tls-auth></tls-auth>
<ca></ca>

And here are the pbr outputs: Failing:

/etc/init.d/pbr status
============================================================
pbr - environment
pbr 1.0.1-16 running on OpenWrt 22.03.3. WAN (IPv4): lan/br-lan/10.0.0.2.
============================================================
Dnsmasq version 2.86  Copyright (c) 2000-2021 Simon Kelley
Compile time options: IPv6 GNU-getopt no-DBus UBus no-i18n no-IDN DHCP no-DHCPv6 no-Lua TFTP no-conntrack no-ipset no-auth no-cryptohash no-DNSSEC no-ID loop-detect inotify dumpfile
============================================================
pbr chains - policies
        chain pbr_forward {
        }
        chain pbr_input {
        }
        chain pbr_output {
        }
        chain pbr_prerouting {
                ip daddr @pbr_vpn_o1_4_dst_ip_cfg026ff5 goto pbr_mark_0x020000 comment "test"
        }
        chain pbr_postrouting {
        }
============================================================
pbr chains - marking
        chain pbr_mark_0x010000 {
                counter packets 0 bytes 0 meta mark set meta mark & 0xff01ffff | 0x00010000
                return
        }
        chain pbr_mark_0x020000 {
                counter packets 12 bytes 800 meta mark set meta mark & 0xff02ffff | 0x00020000
                return
        }
        chain pbr_mark_0x030000 {
                counter packets 0 bytes 0 meta mark set meta mark & 0xff03ffff | 0x00030000
                return
        }
============================================================
pbr nft sets
        set pbr_vpn_o1_4_dst_ip_cfg026ff5 {
                type ipv4_addr
                flags interval
                auto-merge
                comment "test"
                elements = { 34.160.111.145, 95.85.16.212 }
        }
============================================================
Error: ipv4: FIB table does not exist.
Dump terminated
IPv4 table 257 route:
IPv4 table 257 rule(s):
IPv4 table 258 route: default via 10.0.0.1 dev br-wan
IPv4 table 258 rule(s):
30000:  from all fwmark 0x10000/0xff0000 lookup pbr_wan
IPv4 table 259 route: default via 10.1------ dev tun_o1
IPv4 table 259 rule(s):
30001:  from all fwmark 0x20000/0xff0000 lookup pbr_vpn_o1
IPv4 table 260 route: default via 172.1----- dev tun_w
IPv4 table 260 rule(s):
30001:  from all fwmark 0x20000/0xff0000 lookup pbr_vpn_w
30002:  from all fwmark 0x30000/0xff0000 lookup pbr_vpn_w

/etc/init.d/pbr reload
Activating traffic killswitch [✓]
Setting up routing for 'wan/br-wan/10.0.0.1' [✓]
Setting up routing for 'vpn_o1/tun_o1/10.1------' [✓]
Setting up routing for 'vpn_w/tun_w/172.1-----' [✓]
Routing 'test' via vpn_o1 [✓]
Deactivating traffic killswitch [✓]
pbr 1.0.1-16 monitoring interfaces: wan vpn_o1 vpn_w
pbr 1.0.1-16 (nft) started with gateways:
wan/br-wan/10.0.0.1
vpn_o1/tun_o1/10.1------
vpn_w/tun_w/172.1----- [✓]

Working:

/etc/init.d/pbr status
============================================================
pbr - environment
pbr 1.0.1-16 running on OpenWrt 22.03.3. WAN (IPv4): lan/br-lan/10.0.0.2.
============================================================
Dnsmasq version 2.86  Copyright (c) 2000-2021 Simon Kelley
Compile time options: IPv6 GNU-getopt no-DBus UBus no-i18n no-IDN DHCP no-DHCPv6 no-Lua TFTP no-conntrack no-ipset no-auth no-cryptohash no-DNSSEC no-ID loop-detect inotify dumpfile
============================================================
pbr chains - policies
        chain pbr_forward {
        }
        chain pbr_input {
        }
        chain pbr_output {
        }
        chain pbr_prerouting {
                ip daddr @pbr_vpn_o1_4_dst_ip_cfg026ff5 goto pbr_mark_0x020000 comment "test"
        }
        chain pbr_postrouting {
        }
============================================================
pbr chains - marking
        chain pbr_mark_0x010000 {
                counter packets 0 bytes 0 meta mark set meta mark & 0xff01ffff | 0x00010000
                return
        }
        chain pbr_mark_0x020000 {
                counter packets 6 bytes 400 meta mark set meta mark & 0xff02ffff | 0x00020000
                return
        }
        chain pbr_mark_0x030000 {
                counter packets 0 bytes 0 meta mark set meta mark & 0xff03ffff | 0x00030000
                return
        }
============================================================
pbr nft sets
        set pbr_vpn_o1_4_dst_ip_cfg026ff5 {
                type ipv4_addr
                flags interval
                auto-merge
                comment "test"
                elements = { 34.160.111.145, 95.85.16.212 }
        }
============================================================
Error: ipv4: FIB table does not exist.
Dump terminated
IPv4 table 257 route:
IPv4 table 257 rule(s):
IPv4 table 258 route: default via 10.0.0.1 dev br-wan
IPv4 table 258 rule(s):
30000:  from all fwmark 0x10000/0xff0000 lookup pbr_wan
IPv4 table 259 route: default via 10.1------ dev tun_o1
IPv4 table 259 rule(s):
30001:  from all fwmark 0x20000/0xff0000 lookup pbr_vpn_o1
IPv4 table 260 route: default via 172.1----- dev tun_w
IPv4 table 260 rule(s):
30002:  from all fwmark 0x30000/0xff0000 lookup pbr_vpn_w

/etc/init.d/pbr reload
Activating traffic killswitch [✓]
Setting up routing for 'wan/br-wan/10.0.0.1' [✓]
Setting up routing for 'vpn_o1/tun_o1/10.1------' [✓]
Setting up routing for 'vpn_w/tun_w/172.1-----' [✓]
Routing 'test' via vpn_o1 [✓]
Deactivating traffic killswitch [✓]
pbr 1.0.1-16 monitoring interfaces: wan vpn_o1 vpn_w
pbr 1.0.1-16 (nft) started with gateways:
wan/br-wan/10.0.0.1
vpn_o1/tun_o1/10.1------
vpn_w/tun_w/172.1----- [✓]

The only difference here is: IPv4 table 259 rule(s): 30001: from all fwmark 0x20000/0xff0000 lookup pbr_vpn_o1 IPv4 table 260 route: default via 172.1----- dev tun_w IPv4 table 260 rule(s): 30001: from all fwmark 0x20000/0xff0000 lookup pbr_vpn_w 30002: from all fwmark 0x30000/0xff0000 lookup pbr_vpn_w

There is that 30001 rule twice in the failing case, which actually looks like a culprit.

What other things should I check for? Cause I'm in no way proficient in those nft rules.

stangri commented 1 year ago

Your setup does not support routing domain name requests reliably. Check readme on how to handle domain names.

usr97629238 commented 1 year ago

Your setup does not support routing domain name requests reliably. Check readme on how to handle domain names.

Could you be a bit more specific? ELIM5

And check my previous reply about the 30001 rule appearing twice.

stangri commented 1 year ago

And check my previous reply about the 30001 rule appearing twice.

Do you have an idea when/how it happens? Obviously the table is not cleared properly, if you can narrow down when/how it happens, would be easier to hunt down the case.

Could you be a bit more specific? ELIM5

config pbr 'config' option resolver_set 'none'

Can't find documentation on dns per network interfaces in the wiki anymore, likely these will prevent domain policies from working too:

config interface 'lan' list dns '1.1.1.1'

config interface 'wan' list dns '1.1.1.1'

stangri commented 1 year ago

Could you please also add contents of /etc/iproute2/rt_tables in these conditions:

  1. pbr service stopped
  2. pbr service is working as intended
  3. pbr service is failing to route properly
stangri commented 1 year ago

Version pbr 1.0.1-17 should clean up the marking rules properly, however it wouldn't completely help in your case, as int he failing state you've provided, one table was missing marking rule/routing completely.

usr97629238 commented 1 year ago

Do you have an idea when/how it happens? Obviously the table is not cleared properly, if you can narrow down when/how it happens, would be easier to hunt down the case.

TBH I've been under the impression that it's pbr that is adding/clearing those rules, no? There is nothing installed and configured on this test host, just a stock fresh 22.03.3 install with only pbr, wireguard and openvpn packages added. And I've been doing nothing except enabling/disabling vpn, switching interfaces in the luci-app-pbr and restarting pbr until it starts working/failing.

option resolver_set 'none'

Like I've said, also thought about that, tried installing dnsmasq-full from the snapshots, made no difference. And actually for the last tests I've added the ipecho.net 34.160.111.145 to the hosts just to be sure, also made no difference. Should I install dnsmasq-full for the next tests?

/etc/iproute2/rt_tables

No difference there:

#
# reserved values
#
128     prelocal
255     local
254     main
253     default
0       unspec
#
# local
#
#1      inr.ruhep
257 pbr_vpn_o2
258 pbr_wan
259 pbr_vpn_o1
260 pbr_vpn_w
#
# reserved values
#
128     prelocal
255     local
254     main
253     default
0       unspec
#
# local
#
#1      inr.ruhep
257 pbr_vpn_o2
258 pbr_wan
259 pbr_vpn_o1
260 pbr_vpn_w
usr97629238 commented 1 year ago

Do you have an idea when/how it happens? Obviously the table is not cleared properly, if you can narrow down when/how it happens, would be easier to hunt down the case.

It looks like if the interface that is set for some policy goes down, the policy resets to the default policy (adds a new 30001 rule), without clearing first the old 30001 rule.

And after some more fiddling here are two 30002 rules:

30001:  from all fwmark 0x20000/0xff0000 lookup pbr_vpn_o1
IPv4 table 260 route: default via 10.0.0.1 dev br-wan
IPv4 table 260 rule(s):
30000:  from all fwmark 0x10000/0xff0000 lookup pbr_wan
30002:  from all fwmark 0x30000/0xff0000 lookup pbr_wan
IPv4 table 261 route: default via 172.1----- dev tun_w
IPv4 table 261 rule(s):
30001:  from all fwmark 0x20000/0xff0000 lookup pbr_vpn_w
30002:  from all fwmark 0x30000/0xff0000 lookup pbr_vpn_w

Disable interface o1 -> double 30001 rules (even without restarting pbr) Enable interface o1 -> double 30002 rules (even without restarting pbr) And a next pbr restart clears those.

Changing interfaces and save&applying in luci-app-pbr does not clear the double rules.

Update: And nope, the pbr restart doesn't always clear the double rules. Actually in this state, no matter what combinations of enable/disable/start/stop/reset vpn and/or pbr I've tried it did not get cleared:

IPv4 table 258 route:
IPv4 table 258 rule(s):
IPv4 table 259 route: default via 10.1----- dev tun_o1
IPv4 table 259 rule(s):
30001:  from all fwmark 0x20000/0xff0000 lookup pbr_vpn_o1
IPv4 table 260 route: default via 10.0.0.1 dev br-wan
IPv4 table 260 rule(s):
30000:  from all fwmark 0x10000/0xff0000 lookup pbr_wan
30002:  from all fwmark 0x30000/0xff0000 lookup pbr_wan
IPv4 table 261 route: default via 172.1----- dev tun_w
IPv4 table 261 rule(s):
30001:  from all fwmark 0x20000/0xff0000 lookup pbr_vpn_w
30002:  from all fwmark 0x30000/0xff0000 lookup pbr_vpn_w
usr97629238 commented 1 year ago

Do you make x86/64 builds? I can test a current snapshot if you'd like.

stangri commented 1 year ago

Version pbr 1.0.1-19 and later always restarts the whole service/all interfaces if the interface_reload is called for the OpenVPN interface.

usr97629238 commented 1 year ago

1.0.1-19

I see. It's not pushed yet though to the opkg https://downloads.openwrt.org/releases/22.03.3/packages/x86_64/packages/

called for the OpenVPN

Are you sure it's only affected by OpenVPN interfaces? I'd say that's happening on any interfaces connecting/disconnecting. Should do some tests.

usr97629238 commented 1 year ago

Not sure why the updates aren't pushed anymore through opkg. So I've manually downloaded pbr 1.0.1-21 and everything seems fine thus far. Thanks!

There is only one minor issue, I'll post the screenshot at the openwrt forum.