[pbr] issue: Unexpected exit code -9 on firewall reload

[schuettecarsten]

When firewall is reloaded, pbr is executed to update routes. That's fine so far. But the script exits with exit code -9 before pbr finished to update the rules, causing the firewall reload script to end before pbr finished to update. That might cause race conditions if firewall is reloaded multiple times.

pbr restart work without any errors, except that setting up the routing rules is somewhat slow - about 10 seconds per route, is that normal?

root@GatewayDummi:/etc/init.d# ./firewall reload
Automatically including '/usr/share/nftables.d/table-post/20-miniupnpd.nft'
Automatically including '/usr/share/nftables.d/table-post/30-pbr.nft'
Automatically including '/usr/share/nftables.d/chain-post/dstnat/20-miniupnpd.nft'
Automatically including '/usr/share/nftables.d/chain-post/forward/20-miniupnpd.nft'
Automatically including '/usr/share/nftables.d/chain-post/mangle_forward/30-pbr.nft'
Automatically including '/usr/share/nftables.d/chain-post/mangle_input/30-pbr.nft'
Automatically including '/usr/share/nftables.d/chain-post/mangle_output/30-pbr.nft'
Automatically including '/usr/share/nftables.d/chain-post/mangle_postrouting/30-pbr.nft'
Automatically including '/usr/share/nftables.d/chain-post/mangle_prerouting/30-pbr.nft'
Automatically including '/usr/share/nftables.d/chain-post/srcnat/20-miniupnpd.nft'
Setting up routing for 'wan/pppoe-wan/***.194/***:d082/64' [✓]
Setting up routing for 'vpn_wg0/***.1/***::1/64' [✓]
Setting up routing for 'vpn_wg1/***.1/***::1/124' [✓]
Routing 'ssh' via vpn_wg1 [✓]
Routing 'ssh_v6' via vpn_wg1 [✓]
Routing 'rdp' via vpn_wg1 Include '/usr/share/pbr/firewall.include' failed with exit code -9
root@GatewayDummi:/etc/init.d# [✓]
Routing 'rdp_v6' via vpn_wg1 [✓]
pbr 1.1.4-16 monitoring interfaces: wan vpn_wg0 vpn_wg1

root@GatewayDummi:/etc/init.d# ./pbr restart
Removing routing for 'wan/pppoe-wan/***.194/***:d082/64' [✓]
Removing routing for 'vpn_wg0/***.1/***::1/64' [✓]
Removing routing for 'vpn_wg1/***.1/***::1/124' [✓]
pbr 1.1.4-16 (nft mode) stopped [✓]
Using wan interface (on_start): wan
Found wan gateway (on_start): ***.194
Using wan6 interface (on_start): wan6
Found wan6 gateway (on_start): ***:3c3a
Setting up routing for 'wan/pppoe-wan/***.194/***:d082/64' [✓]
Setting up routing for 'vpn_wg0/***.1/***::1/64' [✓]
Setting up routing for 'vpn_wg1/***.1/***::1/124' [✓]
Routing 'ssh' via vpn_wg1 [✓]
Routing 'ssh_v6' via vpn_wg1 [✓]
Routing 'rdp' via vpn_wg1 [✓]
Routing 'rdp_v6' via vpn_wg1 [✓]
pbr 1.1.4-16 monitoring interfaces: wan vpn_wg0 vpn_wg1
pbr 1.1.4-16 (nft mode) started with gateways:
wan/pppoe-wan/***.194/***:d082/64 [✓]
vpn_wg0/***.1/***::1/64
vpn_wg1/***.1/***::1/124
root@GatewayDummi:/etc/init.d#

[stangri]

With fw4/nft, could you please try using the nft file mode option? This would significantly speed up firewall reloads.

[schuettecarsten]

Oh yes, that looks better.

[schuettecarsten]

IÄve just reopened this issue. The nft file is created at '/usr/share/nftables.d/, but these files are not included in the list of files that should be persisted on firmware upgrade. No on the first boot after a firmware flash, the files are missing and it looks like they're not recreated automatically?

[stangri]

Seems to me like different issues tho.

Could you please elaborate which files are missing after first boot and/or firmware upgrade?

[schuettecarsten]

The missing file is /usr/share/nftables.d/ruleset-post/30-pbr.nft.

[stangri]

That file is re-created on pbr service:

• boot
• start
• reload
• restart

[schuettecarsten]

Ok, I looked into the logs while firewall loads and expected that the file is recreated in this case also.

[stangri]

Ok, I looked into the logs while firewall loads and expected that the file is recreated in this case also.

Ah, you're right, It should be re-created on the firewall reload if that file doesn't exist. In that case the on_firewall_reload command to procd script turns into start command.

[schuettecarsten]

I also run into the issue that the file is NOT recreated on the boot after a firmware flash, pbr ends in invalid state and needs to be restarted manually to work properly after flash.

[stangri]

I also run into the issue that the file is NOT recreated on the boot after a firmware flash, pbr ends in invalid state and needs to be restarted manually to work properly after flash.

What do you mean "also"?

Please elaborate on exact steps of firmware flash.

[schuettecarsten]

I upload the file to /tmp/firmware.bin and run sysupgrade -v -u /tmp/firmware.bin on an SSH console. On the next boot, PBR does not load and shows some "Invalid" errors on its Luci status page. "Reload" fixes this.

[stangri]

Thanks for your prompt reply. Going from which version of OpenWrt to which version of OpenWrt? What's in the log for pbr after first boot?

[schuettecarsten]

I just set up a rsyslog server and let you know on my next flash - tomorrow. Then I will also add some screenshots to clarify the issue and my settings.