search

stangri / source.openwrt.melmac.net

OpenWrt Packages
GNU General Public License v3.0
144 stars 47 forks source link

[pbr] issue: IPv6, missing default route in VPN table if the default route is not present in main table #202

Open egc112 opened 2 weeks ago

egc112 commented 2 weeks ago

[pbr] issue: IPv6, missing default route in VPN table if the default route is not present in main table

I was experimenting with IPv6 and PBR and ran into some problems.

This is a test router with which I am tinkering a lot, so it might just be gremlins on the router.

DL-WRX36, K6.33
pbr 1.1.5-8 running on OpenWrt SNAPSHOT. WAN (IPv4): wan/wan/192.168.21.1. WAN (IPv6): wan6/wan/2a01:cb18:8660:de00::/64.

IPv6 works without PBR with a WireGuard tunnel to Mullvad. I had to do some tweaks to get the default routing via the VPN right:

Network
•   WAN6 > Advanced tab  > Disable Source routing
•   WAN4 set metric to 20
•   WG interface set metric to 10 
Firewall
•   The WG interface needs to be in its one zone
WG firewall zone>  Advanced settings > Enable IPv6 Masquerading

With this ipleak.net and traceroute show the use of IPv6 via the tunnel.

Now on to PBR. When PBR is enabled, tables for WAN and VPN are made with appropriate default routing

The problem arises, for IPv6 only, when I disable default routing by either disabling Route Allowed IPs or disabling default gateway. IPv4 is working, but in the IPv6 VPN table there is no longer a default route via the VPN

I looked at the code and although I do not grasp all the details there seems to be a difference between IPv4 and IPv6

IPv4:

                if [ -n "$gw4" ] || [ "$strict_enforcement" -ne '0' ]; then
                    ipv4_error=0
                    if [ -z "$gw4" ]; then
                        try "$ip_bin" -4 route add unreachable default table "$tid" >/dev/null 2>&1 || ipv4_error=1
                    else
                        try "$ip_bin" -4 route add default via "$gw4" dev "$dev" table "$tid" >/dev/null 2>&1 || ipv4_error=1
                    fi
# shellcheck disable=SC2086
                    while read -r i; do
                        i="$(echo "$i" | sed 's/ linkdown$//')"
                        i="$(echo "$i" | sed 's/ onlink$//')"
                        idev="$(echo "$i" | grep -Eso 'dev [^ ]*' | awk '{print $2}')"
                        if ! is_supported_iface_dev "$idev"; then
                            try "$ip_bin" -4 route add $i table "$tid" >/dev/null 2>&1 || ipv4_error=1
                        fi
                    done << EOF
                    $($ip_bin -4 route list table main)
EOF

As far as I can see it will either add and unreachable or the default route to the routing table and then adds "local" rules to the routing table

IPv6

The IPv6 code seems to do things differently:

                if [ -n "$ipv6_enabled" ]; then
                    ipv6_error=0
                    $ip_bin -6 rule del table "$tid" >/dev/null 2>&1
                    $ip_bin -6 route flush table "$tid" >/dev/null 2>&1
                    if { [ -n "$gw6" ] && [ "$gw6" != "::/0" ]; } || [ "$strict_enforcement" -ne '0' ]; then
                        if [ -z "$gw6" ] || [ "$gw6" = "::/0" ]; then
                            try "$ip_bin" -6 route add unreachable default table "$tid" >/dev/null 2>&1 || ipv6_error=1
                        elif "$ip_bin" -6 route list table main | grep -q " dev $dev6 "; then
                            "$ip_bin" -6 route add default via "$gw6" dev "$dev6" table "$tid" >/dev/null 2>&1 || ipv6_error=1
                            while read -r i; do
                                i="$(echo "$i" | sed 's/ linkdown$//')"
                                i="$(echo "$i" | sed 's/ onlink$//')"
                                # shellcheck disable=SC2086
                                try "$ip_bin" -6 route add $i table "$tid" >/dev/null 2>&1 || ipv6_error=1
                            done << EOF
                            $($ip_bin -6 route list table main | grep " dev $dev6 ")
EOF
                        else
                            try "$ip_bin" -6 route add "$($ip_bin -6 -o a show "$dev6" | awk '{print $4}')" dev "$dev6" table "$tid" >/dev/null 2>&1 || ipv6_error=1
                            try "$ip_bin" -6 route add default dev "$dev6" table "$tid" >/dev/null 2>&1 || ipv6_error=1
                        fi
                    fi

This seems to set only default routes if there is no unreachable route set or if there is no "local" route which in my case results in no default route set as there is a local route (the route of the WG interface). (The fact that there is a default route when the default route is in the main table is that it is copied with the "local" route copying)

This is my VPN table without the default route:

root@DL-WRX36:~# ip -6 route show table pbr_wg_mullv_se
fc00:bbbb:bbbb:bb01::/64 dev wg_mullv_se proto static metric 10 pref medium

I hacked the code a bit and shuffled things around and now it seems to work:

                if [ -n "$ipv6_enabled" ]; then
                    ipv6_error=0
                    $ip_bin -6 rule del table "$tid" >/dev/null 2>&1
                    $ip_bin -6 route flush table "$tid" >/dev/null 2>&1
                    if { [ -n "$gw6" ] && [ "$gw6" != "::/0" ]; } || [ "$strict_enforcement" -ne '0' ]; then
                        if [ -z "$gw6" ] || [ "$gw6" = "::/0" ]; then
                            try "$ip_bin" -6 route add unreachable default table "$tid" >/dev/null 2>&1 || ipv6_error=1
                        else
                            try "$ip_bin" -6 route add "$($ip_bin -6 -o a show "$dev6" | awk '{print $4;exit;}')" dev "$dev6" table "$tid" >/dev/null 2>&1 || ipv6_error=1
                            try "$ip_bin" -6 route add default dev "$dev6" table "$tid" >/dev/null 2>&1 || ipv6_error=1
                        fi
                        if "$ip_bin" -6 route list table main | grep -q " dev $dev6 "; then
                            "$ip_bin" -6 route add default via "$gw6" dev "$dev6" table "$tid" >/dev/null 2>&1 || ipv6_error=1
                            while read -r i; do
                                i="$(echo "$i" | sed 's/ linkdown$//')"
                                i="$(echo "$i" | sed 's/ onlink$//')"
                                # shellcheck disable=SC2086
                                try "$ip_bin" -6 route add $i table "$tid" >/dev/null 2>&1 || ipv6_error=1
                            done << EOF
                            $($ip_bin -6 route list table main | grep " dev $dev6 ")
EOF
                        fi
                    fi

Now that the code with default route runs, I hit another problem the following rule threw an error because the wan interface has two entries so I added an exit; in awk '{print $4;exit;}') Of course not the way to solve this but it got my code running

try "$ip_bin" -6 route add "$($ip_bin -6 -o a show "$dev6" | awk '{print $4;exit;}')" dev "$dev6" table "$tid" >/dev/null 2>&1 || ipv6_error=1

Maybe I am wasting your time so apologizes beforehand and of course my hacking is not a proper solution but only meant to demonstrate what I found

Thanks for all you excellent work

Regards, Erik

Your configs

  1. /etc/config/dhcp
config dnsmasq
        option domainneeded '1'
        option localise_queries '1'
        option rebind_protection '1'
        option local '/home9/'
        option domain 'home9'
        option expandhosts '1'
        option cachesize '1000'
        option authoritative '1'
        option readethers '1'
        option leasefile '/tmp/dhcp.leases'
        option localservice '0'
        option ednspacket_max '1232'
        option confdir '/tmp/dnsmasq.d'
        list rebind_domain 'home'
        list server '/home/192.168.0.1'

config dhcp 'lan'
        option interface 'lan'
        option start '128'
        option limit '64'
        option leasetime '12h'
        option dhcpv4 'server'
        option dhcpv6 'server'
        option ra 'server'
        list ra_flags 'managed-config'
        list ra_flags 'other-config'

config dhcp 'wan'
        option interface 'wan'
        option ignore '1'

config odhcpd 'odhcpd'
        option maindhcp '0'
        option leasefile '/tmp/hosts/odhcpd'
        option leasetrigger '/usr/sbin/odhcpd-update'
        option loglevel '4'

config dhcp 'guest'
        option interface 'guest'
        option start '100'
        option limit '150'
        option leasetime '12h'

config tag 'tag1'
        list dhcp_option '6,8.8.8.8,8.8.4.4'

config host
        option name 'Galaxy-S20-FE-van-E'
        option ip '192.168.9.223'
        list mac 'XX:XX:XX:XX:XX:XX'

config host
        option name 'nitro'
        option ip '192.168.9.23'
        list mac 'XX:XX:XX:XX:XX:XX'
        option dns '1'
  1. /etc/config/network
config interface 'loopback'
        option device 'lo'
        option proto 'static'
        option ipaddr '127.0.0.1'
        option netmask '255.0.0.0'

config globals 'globals'
        option ula_prefix 'fd30:68f6:1257::/48'
        option packet_steering '1'

config device
        option name 'br-lan'
        option type 'bridge'
        list ports 'lan1'
        list ports 'lan2'
        list ports 'lan3'

config interface 'lan'
        option device 'br-lan'
        option proto 'static'
        option ipaddr '192.168.9.1'
        option netmask '255.255.255.0'
        option ip6assign '64'
        option delegate '0'
        option ip6hint '90'

config interface 'wan'
        option device 'wan'
        option proto 'dhcp'
        option peerdns '0'
        list dns '9.9.9.9'
        list dns '1.0.0.1'
        list dns '2620:fe::10'
        list dns '2606:4700:4700::1001'
        option metric '20'

config interface 'wan6'
        option device 'wan'
        option proto 'dhcpv6'
        option reqaddress 'try'
        option reqprefix 'auto'
        option peerdns '0'
        option sourcefilter '0'

config interface 'wgoraclecloud'
        option proto 'wireguard'
        option private_key '+sfs'
        option mtu '1420'
        list addresses '10.168.0.3/24'
        option listen_port '9998'
        list dns '149.112.112.112'
        option dns_metric '0'
        option disabled '1'

config interface 'wgserver'
        option proto 'wireguard'
        option private_key 'eGsTsdfs'
        option listen_port '56666'
        list addresses '172.31.31.1/24'
        option mtu '1200'

config wireguard_wgserver
        option description 'nitro'
        option public_key 'Ih'
        option private_key 'sdfdf'
        option route_allowed_ips '1'
        list allowed_ips '172.31.31.2/32'

config interface 'guest'
        option proto 'static'
        option device 'br-guest'
        option ipaddr '192.168.91.1'
        option netmask '255.255.255.0'
        option defaultroute '0'

config device
        option type 'bridge'
        option name 'br-guest'
        option bridge_empty '1'
        list ports 'lan4'

config interface 'wg_stos_6'
        option proto 'wireguard'
        option private_key 'iPRhiXZCW'
        list addresses '172.21.21.9/24'
        option listen_port '51220'
        list dns '192.168.6.1'
        option defaultroute '0'

config wireguard_wg_stos_6
        option description 'EA8500'
        option public_key 'vypYdM'
        option route_allowed_ips '1'
        option endpoint_host 'efgh.sfgt.com'
        option endpoint_port '51810'
        option persistent_keepalive '25'
        list allowed_ips '172.21.21.0/24'
        list allowed_ips '192.168.6.0/24'
        list allowed_ips '0.0.0.0/0'

config interface 'tun0_ks_ro'
        option proto 'none'
        option device 'tun0'
        option disabled '1'
        option auto '0'

config wireguard_wgoraclecloud
        option description 'Imported peer configuration'
        option public_key 'wJz'
        option preshared_key 'gbIr'
        option endpoint_host '140.238.218.87'
        option endpoint_port '51801'
        option persistent_keepalive '100'
        list allowed_ips '10.168.0.0/24'
        list allowed_ips '10.0.0.0/24'
        list allowed_ips '0.0.0.0/0'
        option route_allowed_ips '1'

config interface 'ovpnserver1'
        option proto 'none'
        option device 'tun2'
        option auto '0'
        option disabled '1'

config interface 'wg_mullv_se'
        option proto 'wireguard'
        option private_key 'iLFBB'
        option delegate '0'
        list dns '10.64.0.1'
        option metric '10'
        list addresses '10.69.78.222/24'
        list addresses 'fc00:bbbb:bbbb:bb01::6:4edd/64'
        option defaultroute '0'

config wireguard_wg_mullv_se
        option description 'mullvad-se-got-wg-004.conf'
        option public_key 'veGD'
        option persistent_keepalive '25'
        option endpoint_host '185.213.154.69'
        option endpoint_port '51820'
        list allowed_ips '::/0'
        list allowed_ips '0.0.0.0/0'
        option route_allowed_ips '1'

config interface 'tun1'
        option proto 'none'
        option device 'tun1'
        option auto '0'
        option disabled '1'

config device
        option name 'phy0-ap0'
        option macaddr 'A4:97:33:DF:97:F4'
  1. /etc/config/firewall

config defaults
        option input 'REJECT'
        option output 'ACCEPT'
        option forward 'REJECT'

config zone
        option name 'lan'
        option input 'ACCEPT'
        option output 'ACCEPT'
        option forward 'ACCEPT'
        list device 'tun2'
        list network 'guest'
        list network 'lan'
        list network 'wg_stos_6'

config zone
        option name 'wan'
        option input 'REJECT'
        option output 'ACCEPT'
        option forward 'REJECT'
        option mtu_fix '1'
        option masq '1'
        list network 'wan'
        list network 'wan6'
        list device 'tun0'

config rule
        option name 'Allow-DHCP-Renew'
        option src 'wan'
        option proto 'udp'
        option dest_port '68'
        option target 'ACCEPT'
        option family 'ipv4'

config rule
        option name 'Allow-Ping'
        option src 'wan'
        option proto 'icmp'
        option icmp_type 'echo-request'
        option family 'ipv4'
        option target 'ACCEPT'

config rule
        option name 'Allow-IGMP'
        option src 'wan'
        option proto 'igmp'
        option family 'ipv4'
        option target 'ACCEPT'

config rule
        option name 'Allow-DHCPv6'
        option src 'wan'
        option proto 'udp'
        option dest_port '546'
        option family 'ipv6'
        option target 'ACCEPT'

config rule
        option name 'Allow-MLD'
        option src 'wan'
        option proto 'icmp'
        option src_ip 'fe80::/10'
        list icmp_type '130/0'
        list icmp_type '131/0'
        list icmp_type '132/0'
        list icmp_type '143/0'
        option family 'ipv6'
        option target 'ACCEPT'

config rule
        option name 'Allow-ICMPv6-Input'
        option src 'wan'
        option proto 'icmp'
        list icmp_type 'echo-request'
        list icmp_type 'echo-reply'
        list icmp_type 'destination-unreachable'
        list icmp_type 'packet-too-big'
        list icmp_type 'time-exceeded'
        list icmp_type 'bad-header'
        list icmp_type 'unknown-header-type'
        list icmp_type 'router-solicitation'
        list icmp_type 'neighbour-solicitation'
        list icmp_type 'router-advertisement'
        list icmp_type 'neighbour-advertisement'
        option limit '1000/sec'
        option family 'ipv6'
        option target 'ACCEPT'

config rule
        option name 'Allow-ICMPv6-Forward'
        option src 'wan'
        option dest '*'
        option proto 'icmp'
        list icmp_type 'echo-request'
        list icmp_type 'echo-reply'
        list icmp_type 'destination-unreachable'
        list icmp_type 'packet-too-big'
        list icmp_type 'time-exceeded'
        list icmp_type 'bad-header'
        list icmp_type 'unknown-header-type'
        option limit '1000/sec'
        option family 'ipv6'
        option target 'ACCEPT'

config rule
        option name 'Allow-IPSec-ESP'
        option src 'wan'
        option dest 'lan'
        option proto 'esp'
        option target 'ACCEPT'

config rule
        option name 'Allow-ISAKMP'
        option src 'wan'
        option dest 'lan'
        option dest_port '500'
        option proto 'udp'
        option target 'ACCEPT'

config redirect
        option dest 'lan'
        option target 'DNAT'
        option name 'allow-http'
        option src 'wan'
        option src_dport '8080'
        option dest_ip '192.168.9.1'
        option dest_port '80'

config redirect
        option target 'DNAT'
        option name 'allow-ssh'
        option src 'wan'
        option src_dport '22'
        option dest_port '22'
        option dest 'lan'
        option dest_ip '192.168.9.1'

config zone
        option name 'ovpn_client'
        option input 'REJECT'
        option output 'ACCEPT'
        option forward 'REJECT'
        option masq '1'
        option mtu_fix '1'
        option masq6 '1'
        list network 'tun1_ks_ro'
        list network 'wg_mullv_se'
        list network 'tun1'

config forwarding
        option src 'lan'
        option dest 'ovpn_client'

config rule
        option name 'wgserver'
        list proto 'udp'
        option src 'wan'
        option dest_port '52199'
        option target 'ACCEPT'

config zone
        option name 'wgserver'
        option input 'ACCEPT'
        option output 'ACCEPT'
        option forward 'ACCEPT'
        option mtu_fix '1'
        list network 'wgoraclecloud'
        list network 'wgserver'

config forwarding
        option src 'wgserver'
        option dest 'ovpn_client'

config forwarding
        option src 'wgserver'
        option dest 'wan'

config forwarding
        option src 'lan'
        option dest 'wgserver'

config include 'miniupnpd'
        option type 'script'
        option path '/usr/share/miniupnpd/firewall.include'

config rule
        option name 'ovpnserver1'
        option target 'ACCEPT'
        option src 'wan'
        option dest_port '1196'

config forwarding
        option src 'wgserver'
        option dest 'lan'

config include 'bcp38'
        option type 'script'
        option path '/usr/lib/bcp38/run.sh'

config rule
        option name 'allow_forward'
        option src 'wan'
        option dest '*'
        option target 'ACCEPT'
        list src_ip '10.0.0.0/8'
        list src_ip '192.168.0.0/16'

config rule
        option name 'allow_input'
        option src 'wan'
        option target 'ACCEPT'
        list src_ip '192.168.0.0/16'
        list src_ip '10.0.0.0/8'

config forwarding
        option src 'lan'
        option dest 'wan'

config include 'pbr'
        option fw4_compatible '1'
        option type 'script'
        option path '/usr/share/pbr/firewall.include'
  1. /etc/config/pbr
config pbr 'config'
        option enabled '1'
        option verbosity '2'
        option strict_enforcement '0'
        option resolver_set 'dnsmasq.nftset'
        option ipv6_enabled '1'
        option boot_timeout '45'
        option rule_create_option 'add'
        option procd_reload_delay '1'
        option webui_show_ignore_target '1'
        list webui_supported_protocol 'all'
        list webui_supported_protocol 'tcp'
        list webui_supported_protocol 'udp'
        list webui_supported_protocol 'tcp udp'
        list webui_supported_protocol 'icmp'
        list ignored_interface 'ovpnserver1'
        list ignored_interface 'wgserver'

config include
        option path '/usr/share/pbr/pbr.user.aws'
        option enabled '0'

config include
        option path '/usr/share/pbr/pbr.user.netflix'
        option enabled '0'

config policy
        option name 'wgserver'
        option interface 'ignore'
        option dest_addr '172.31.31.0/24'

config policy
        option name 'Plex/Emby Local Server'
        option interface 'wan'
        option src_port '8096 8920 32400'
        option enabled '0'

config policy
        option name 'Plex/Emby Remote Servers'
        option interface 'wan'
        option dest_addr 'plex.tv my.plexapp.com emby.media app.emby.media tv.emby.media'
        option enabled '0'

config include
        option enabled '0'
        option path '/usr/share/pbr/pbr.user.wg_server_and_client'

config include
        option enabled '0'
        option path '/usr/share/pbr/pbr.user.aws.egc'

config policy
        option name 'fe20'
        option src_addr '98:B8:BC:8B:3F:9E'
        option interface 'wg_mullv_se'
stangri commented 2 weeks ago

Thank you for the report and suggested patch Erik. I'm in the process of preparing to remove the iptables-related code and reworking the nft code to improve the nft_file_mode support (by adding all elements to the nft set in one command and thus hopefully avoiding the nft set collision errors), so I'll try to review this with the attention it deserves at some point next week.

egc112 commented 2 weeks ago

Thanks, good idea to remove the iptables-related code as stable and main builds are now nftables