mwan3: 23.05 and ipset functionality broken

[jamesmacwhite]

Maintainer: @feckert Environment: OpenWrt 23.05

Description:

With the release of 23.05, a critical change to the dnsmasq-full compile options was made where ipset support was removed. This has had a major impact to the mwan3 package and its ipset functionality, given mwan3 does not currently support nfset and now ipset chains will no longer work on 23.05 regardless of any compatibility/translation packages.

Further discussion: https://forum.openwrt.org/t/23-05-dnsmasq-ipsets-and-mwan3-incompatibility/174926

As an initial/short term action, I'll look at updating the documentation to make this clear, I'd imagine various users of mwan3 will be leveraging the ipset functionality.

This however does highlight the need for mwan3 to look at transitioning to nftables without compatibility layers. I appreciate however this is easier said than done, given the complexity and iptables specific code contained within.

[axxelh]

Summarizing what I said in this post slightly, as a workaround for 23.05 I believe compiling dnsmasq-full with -DHAVE_IPSET will at least give users a path forward till mwan3 is updated. This would not change the decision to have /etc/config/dhcp generate nftsets, it would just allow users willing to edit /etc/dnsmasq.conf to retain some functionality.

There would still need to be an mwan3 documentation update, but given that the /etc/dnsmasq.conf path is already documented in mwan3, it seems like it would be an acceptable workaround for now, till the mwan3 transition to nftables is resolved.

[axxelh]

After some back-and-forth in the forum thread, I can confirm that compiling with -DHAVE_IPSET by changing the PACKAGE_dnsmasq_full_ipset default to "y" restores functionality to the same level as existed in 22.03. ipset creation from uci/Luci will still not interoperate with mwan3, but that is not new with 22.05, and workarounds exist (creating sets in rc.local, etc.).

This seems harmless, has no apparent negative side effects, so I am hopeful it will be considered for the next service release until mwan3 is rewritten for nftables.

[NephewBoost]

@axxelh Can you walk me through the steps to get it working??

[axxelh]

@axxelh Can you walk me through the steps to get it working??

I assume you mean how to build it yourself? I don't think its really amenable to a walkthrough. The instructions to do a build are here, and the dnsmasq_full_ipset change is available as part of the menuconfig.

I don't consider it practical to manage my own build of openwrt long term, so I've frozen on 22.03 till this is addressed, either in a service release or with the mwan3 rewrite to nftables.

[jamesmacwhite]

After some back-and-forth in the forum thread, I can confirm that compiling with -DHAVE_IPSET by changing the PACKAGE_dnsmasq_full_ipset default to "y" restores functionality to the same level as existed in 22.03. ipset creation from uci/Luci will still not interoperate with mwan3, but that is not new with 22.05, and workarounds exist (creating sets in rc.local, etc.).

This seems harmless, has no apparent negative side effects, so I am hopeful it will be considered for the next service release until mwan3 is rewritten for nftables.

I am looking to move to 23.xx soon as I've stayed on 21.02 for a long time, partly because of mwan3. I would potentially create your own init script that can be loaded earlier to initialise creating sets in managed sequence compared to rc.local. This file is right at the end of the boot sequence. You may have other applications which boot earlier referencing sets which will cause problems, if they are not created in the right timing order. You should potentially look at the dnsmasq START value and put an extra init script following after to ensure these are available as soon as you can, otherwise you'll get race conditions with other applications potentially.

It is a while I last looked at this, but it unfortunately seems no progress has been made for mwan3 to support nftables natively (I fully acknowledge I do not have enough networking knowledge to assist with porting it). There is the PBR package which provides some of the functionality of mwan3, PBR cannot however do load balancing/failover, but can conditionally route traffic, depending on needs.

[Kishi85]

I've re-written a bash script (from the forums) that now runs as an init-service which sync's ipsets for mwan3 from nft sets in (almost) real-time (thanks to nft monitor). Removing the need to recompile dnsmasq-full with ipset and enabling mwan3 to work with dnsmasq's set functionality again as simple as possible.

The script (and it's predecessor) can be found on the OpenWRT forums: https://forum.openwrt.org/t/23-05-dnsmasq-ipsets-and-mwan3-incompatibility/174926/36

[jamesmacwhite]

Nice work! Happy to add it to the wiki as a workaround solution, given it lowers the barrier of recompiling dnsmasq-full if you are happy with that?

[Kishi85]

Nice work! Happy to add it to the wiki as a workaround solution, given it lowers the barrier of recompiling dnsmasq-full if you are happy with that?

Please feel free to do so.

[bernikr]

@Kishi85 thanks for the script. posting here, because I dont have a login for the forum.

If anyone runs into the following issue (which is not the most helpful error message):

root@OpenWrt:/etc/init.d# service nft2ipset start
/sbin/service: line 10: /etc/init.d/nft2ipset: not found

The solution for me was to replace the two #!/bin/bash in the script with #!/bin/sh, as bash is not installed on my router.

[Kishi85]

The script functionality itself will require bash to properly and fully function, so you'll have to install it or otherwise the base script inside the init.d-service wrapper might not work properly on some commands as it uses specific syntax that might not be present in normal openwrt /bin/sh (especially the extended test syntax from bash [[ ... ]] for wildcard matching). If you use the script with anything else than bash I've not tested it and I'm unsure if it will work properly for all cases.

[jamesmacwhite]

I'll prefix the bash requirement when added to the Wiki as this is not default on OpenWrt in many cases.

[Kishi85]

I've taken the time to make sure this works with OpenWRTs default /bin/sh, no more bash required. See https://forum.openwrt.org/t/23-05-dnsmasq-ipsets-and-mwan3-incompatibility/174926/40 for the new version.

[jamesmacwhite]

Added to the Wiki. It might be good if the script itself was maybe placed as gist, so it has some versioning/reliable endpoint for wget, to make it easier to deploy?

I've added it as is on the wiki currently, but let me know.

[Kishi85]

Gist is actually a good suggestion. Here it is: https://gist.github.com/Kishi85/b7f379f9aa19f4878af28b8e1a8887ab

This makes the installation as easy as:

wget -O/etc/init.d/nft2ipset https://gist.github.com/Kishi85/b7f379f9aa19f4878af28b8e1a8887ab/raw/
chmod +x /etc/init.d/nft2ipset
service nft2ipset enable
service nft2ipset start

[jamesmacwhite]

Awesome! I will update the Wiki later on this evening or if anyone else has access, feel to amend also.

Edit: Sneaked in the update while at work. Should be all good now.

[AT-StephenDetomasi]

Is there a proper fix for MWAN in the works? These hacks are overly complex and it shouldn't be accepted as a solution. I've been struggling for a good hour or so and still can't get failover to work as expected on OpenWrt 23.05.5.

[200433zc]

just wait or revert to a usable version> Is there a proper fix for MWAN in the works? These hacks are overly complex and it shouldn't be accepted as a solution. I've been struggling for a good hour or so and still can't get failover to work as expected on OpenWrt 23.05.5.

[feckert]

Due to other projects, the implementation of mwan3 with nftables has been stalled. Let's see when I get back to it. So far I only have a proof of concept implementation for pinging the targets. The whole nftables handling is still missing.

[200433zc]

Due to other projects, the implementation of mwan3 with nftables has been stalled. Let's see when I get back to it. So far I only have a proof of concept implementation for pinging the targets. The whole nftables handling is still missing.

Looking forward to your return

[AT-StephenDetomasi]

Thanks for the update, looking forward to seeing an update. It's a pretty important feature for a lot of folks. Happy to be involved with testing when the time is right.

[marrocksd]

+1