wulfy23 / rpi4

OpenWrt full for rpi4
214 stars 24 forks source link

fw4/nft migration process of build related iptables functions and related user uses #26

Open LionHeartP opened 1 year ago

LionHeartP commented 1 year ago

Hi all. As per the title, something must have changed in the newest snapshot's firewall configuration. My log is full of errors;

Sat Nov 12 11:55:11 2022 daemon.err dnsmasq[24046]: nftset inet fw4 usrcdn Error: No such file or directory Sat Nov 12 11:55:11 2022 daemon.err dnsmasq[24046]: nftset inet fw4 usrcdn Error: No such file or directory Sat Nov 12 11:55:11 2022 daemon.err dnsmasq[24046]: nftset inet fw4 usrcdn6 Error: No such file or directory Sat Nov 12 11:55:17 2022 daemon.err dnsmasq[24046]: nftset inet fw4 usrcdn Error: No such file or directory Sat Nov 12 11:55:17 2022 daemon.err dnsmasq[24046]: nftset inet fw4 usrcdn Error: No such file or directory Sat Nov 12 11:55:17 2022 daemon.err dnsmasq[24046]: nftset inet fw4 usrcdn Error: No such file or directory Sat Nov 12 11:55:17 2022 daemon.err dnsmasq[24046]: nftset inet fw4 usrcdn Error: No such file or directory Sat Nov 12 11:55:17 2022 daemon.err dnsmasq[24046]: nftset inet fw4 usrcdn Error: No such file or directory Sat Nov 12 11:55:17 2022 daemon.err dnsmasq[24046]: nftset inet fw4 usrcdn Error: No such file or directory Sat Nov 12 11:55:17 2022 daemon.err dnsmasq[24046]: nftset inet fw4 usrcdn Error: No such file or directory Sat Nov 12 11:55:17 2022 daemon.err dnsmasq[24046]: nftset inet fw4 usrcdn Error: No such file or directory Sat Nov 12 11:55:17 2022 daemon.err dnsmasq[24046]: nftset inet fw4 usrcdn Error: No such file or directory Sat Nov 12 11:55:17 2022 daemon.err dnsmasq[24046]: nftset inet fw4 usrcdn Error: No such file or directory Sat Nov 12 11:55:17 2022 daemon.err dnsmasq[24046]: nftset inet fw4 usrcdn Error: No such file or directory Sat Nov 12 11:55:17 2022 daemon.err dnsmasq[24046]: nftset inet fw4 usrcdn Error: No such file or directory Sat Nov 12 11:55:17 2022 daemon.err dnsmasq[24046]: nftset inet fw4 usrcdn Error: No such file or directory Sat Nov 12 11:55:17 2022 daemon.err dnsmasq[24046]: nftset inet fw4 usrcdn Error: No such file or directory Sat Nov 12 11:55:20 2022 daemon.err dnsmasq[24046]: nftset inet fw4 usrcdn Error: No such file or directory Sat Nov 12 11:55:20 2022 daemon.err dnsmasq[24046]: nftset inet fw4 usrcdn Error: No such file or directory Sat Nov 12 11:55:20 2022 daemon.err dnsmasq[24046]: nftset inet fw4 usrcdn6 Error: No such file or directory

Any solutions? PEBKAC perhaps? xD

wulfy23 commented 1 year ago

Thanks alot for the info and not PEBKAC.

We didn;tdont run fw4 so will need to look into the underlying change causing the above behavior (first guess is the dnsmasq init.d logic). I'm only using ipsets for QoS on my system so its semi non-critical but any peeps using 'sets' for security rules should probably downgrade or use 'release/stable'...

As a (guess at a quick) workaround removing nft style packages MAY correct the issues above but not sure how it detects firewall (ipset method) in use...

(I removed the 'auto' offering of this 'current' build in the interim ~ also due to luci breaking if argon is selected ~ and if i'm unable to resolve by 5 hours time I'll re-upload the previous 'current' build and re-instate that but it is getting borderline long in the tooth) OR thinking aloud we may finally have to make the jump to fw4 which is pretty ironed out by now barring some auxillary package foibles and whatnot... hmmmmm....

EDIT1: ok so without checking source level changes /solutions/workrounds/comments yet... master seems to not add in ipset support any more...

Sun Nov 13 12:24:02 2022 daemon.err dnsmasq[11080]: nftset inet fw4 usrcdn Error: No such file or directory
Sun Nov 13 12:24:02 2022 daemon.err dnsmasq[11080]: nftset inet fw4 usrcdn6 Error: No such file or directory
Sun Nov 13 12:25:24 2022 daemon.err dnsmasq[11080]: nftset inet fw4 bulk Error: No such file or directory
Sun Nov 13 12:25:24 2022 daemon.err dnsmasq[11080]: nftset inet fw4 bulk6 Error: No such file or directory
[root@rpi-dca6325631 ../r212fw4usrcdnTHING 53°]# ipset -L | grep usrcdn
Name: usrcdn
Name: usrcdn6
^C

[root@rpi-dca6325631 ../r212fw4usrcdnTHING 54°]# ipset -L | grep bulk
Name: bulk
Name: bulk6
^C

[root@rpi-dca6325631 ../r212fw4usrcdnTHING 55°]# dnsmasq -v
Dnsmasq version 2.87  Copyright (c) 2000-2022 Simon Kelley
Compile time options: IPv6 GNU-getopt no-DBus UBus no-i18n no-IDN DHCP DHCPv6 no-Lua TFTP conntrack >>> no-ipset<<< nftset auth cryptohash DNSSEC no-ID loop-detect inotify dumpfile

need to check if they have an ipset variant or whether we run parrallel nftsets or just switch to nftables/fw4 totally...

(sidenote: the lineage of those entries are from sqm and custom classification so for alot of people they could just be remnants of trying my sqm script and can be removed from the dnsmasq config entirely... albeit we still need to address issue at hand with high priority)

EDIT2:

snapshot_7.1.75-11_r21243 ( as 'current' ) is a manually 'operated on' build selection ditching all iptables stuff as previously harmonious dual install behavior is broken pretty badly. will leave this up for a bit to see if/what comes up before going much further... anyone using vpn policies / custom iptables.user / banip etc. etc. should steer clear of master/current maybe, while this transition / method is investigated further...

for me, i'm kind of ok to jump back to 22 or 21 for iptables if I need that stuff for the time being... ( but I pretty much run master most of the time... the QoS stuff being first on my list of stuff to verify / optimise / update / whatever )... as for the masq errors... they are mostly 'informational' so i'm just gonna leave that stuff in my config for a while as it's good for debugging purposes...

EDIT3:

lol, those errors are still present... they (set creation) were instantiated via QoS script... so may need to update that but surprised OS will not instantiate non existing set before trying to add... anyway, as mentioned most people can just remove those ipsets from their dnsmasq config...

( apart from non-existing set use by masq it's mostly stuff I have to update / remove on my end -> aka all previous build related firewall stuff is pretty void for now - unless obviously it breaks something important / general normal operation of OS )

LionHeartP commented 1 year ago

Can confirm that after updating to the firewall4 snapshot, your custom SQP script obviously doesn't work anymore, but switching to another one works. Also, after removing the custom ipsets (damn those looked hugely useful while accompanied by the custom sqm script ofc), no more errors!

wulfy23 commented 1 year ago

Mega sloppy play around with smudging some function across... you are welcome to try if you like (no rush or requirement... just an FYI as expressed interest);

run whole block below on oneline

cp /etc/init.d/firewall /etc/custom/initd_firewall.backup; curl -sSL "https://raw.github.com/wulfy23/rpi4/master/utilities/nft.user.tar.gz" > /nft.user.tar.gz; (cd / && tar -xvzf nft.user.tar.gz)

you'll need to restart probably firewall first, then dnsmasq then sqm if you choose to test the qos script again not that I expect it to function well... just for seeing if sets are populated I suppose... It will probably mess up your dnsmasq config again by adding sets to it so maybe back that up for easy revert if you decide to try it...

and to undo (if needed) just

cp /etc/custom/initd_firewall.backup /etc/init.d/firewall; sync; reboot

LionHeartP commented 1 year ago

Mega sloppy play around with smudging some function across... you are welcome to try if you like (no rush or requirement... just an FYI as expressed interest);

run whole block below on oneline

cp /etc/init.d/firewall /etc/custom/initd_firewall.backup; curl -sSL "https://raw.github.com/wulfy23/rpi4/master/utilities/nft.user.tar.gz" > /nft.user.tar.gz; (cd / && tar -xvzf nft.user.tar.gz)

you'll need to restart probably firewall first, then dnsmasq then sqm if you choose to test the qos script again not that I expect it to function well... just for seeing if sets are populated I suppose... It will probably mess up your dnsmasq config again by adding sets to it so maybe back that up for easy revert if you decide to try it...

and to undo (if needed) just

cp /etc/custom/initd_firewall.backup /etc/init.d/firewall; sync; reboot

Finally gotten around to trying this out today. After running the command and rebooting I got: Sat Nov 19 17:14:32 2022 user.notice SQM: ctinfo_4layercake_rpi4.qos was started on pppoe-wan successfully

Ipsets were not populated, but traffic shaping IS working because I don't get any bufferbloat. Not sure if working as intended though.

Cheers!

edit: To clarify, I have IP Sets under the General settings tab in DHCP and DNS, but none under the IP Sets tab.

wulfy23 commented 1 year ago

thanks!, dnsmasq is adding the right addresses to nftsets... so that's good, the underlying QoS/SQM is functional... but all the rules to leverage those addresses and dscp afaict may not work, or very limited bits of them got translated...

also has integrated 'blocklist' which from my use of the internet this week is surprisingly speedy/snappy... not sure whether this is due to dropping all the dnsmasq lists (adblock) or nft... probably 65% plus is from the dnsmasq simplification if I had to guess... (well could be other stuff in the masq update too I suppose)

so what i'd heard about nft being slower in many regards seems to be non-starter... at least on these devices... good stuff...

yeah the ipsets tab is/was the 'official' way to get around that error we were getting with the nftset's not existing when dnsmasq started up... I cheated for now and just created them from the SQM script... which is possibly saner/more modular if that's the only place we use them... depends on startup order and stuff I suppose...

wulfy23 commented 9 months ago

I've renamed this topic to be a more ongoing place for anything weird or build specific regarding to 23.0x or master regarding fw4/nft.

Massive thanks once again to @LionHeartP for your earlier testing, without this, moving over / providing 23.0x.x builds would have been not possible or pretty rough experience...

Just a few points to summarize on the current state of alot of related stuff to the above...

a) rpi4.qos is now just a copy of 'cake.qos' I think... (i.e. I pretty much removed / disabled it but needed a 'friendly' way for everyone not to have issues so those that have it 'chosen' in sqm settings just get 'cake.qos' or whatever it is...

(edit: the old logic is pretty much compatible / still runs semi smoothly except ipset creation needs to be updated to nftset to play with new dnsmasq... ) also ctinfo / nft / bpf underlying core operation was not investigated) afaik xt CONNMARK is operational under nft...

b) I had to add a 'dhcp' (dnsmasq) config scavenger on firstboot to trash some/all ipset entries the rpi4.qos script created, always should have put those in /tmp/dnsmasq.d not in config, but i'd written it and was tired and it was working... lesson learned (and thanks to the forum dude who pointed that out too)

can't remember if that is enabled on firstboot or everyboot, but if have custom ipset entries under the new system, and they get removed, let me know...

I will disable that crawler in a month or two or something, (give enough people time to upgrade)

c) any services / scripts etc. etc. that made use of iptables on 23.x and master(nft/fw4) are now unknown entities...

in principle, iptables commands get translated to nft commands... so they 'can/should!' work... but the heirarchy/buckets/tables which they were destined to go into in essence make all but the simplest / isolated of uses either not work or not as functional as intended...


while standard official provides /etc/nftables.d/ (usr/share?) for custom user functionality... most people doing smaller stuff that in the past could/were advised to be done via any method (rc.local, uci-firewall include, uci-firewall_rules, other) are now being heavily advised to use just uci-firewall_rules (and ipsets))


as with fw3/iptables builds I requisitioned /etc/firewall.user > /etc/custom/firewall.user for some build related checks and options (that not many used) except for leveraging for OS state awareness during other related step/debugging processes...

this time, for now... I've added a small HOOK to /etc/init.d/firewall that loads /etc/custom/nft.user for anything 'build specific' (setup stuff, or stuff that needs to be enabled by a user) that might be needed for general housekeeping...

I DONT/WOULD NOT USE THIS FOR ADDING 'non general streamling/optimisation' or modifying at all... (without explicit user enablement via /root/wrt.ini VARIABLE)

and at the moment it does nothing... just there in case needed... but I call via init.d not include this time... (dunno why I did that, may change it)


Service wise anything that is not hugely popular i'd say, unless it was using really bespoke (not integrated with OS naming conventions or timing etc.) rules chances are you'll see that they don't work/like they did on 22/fw3/iptables

i.e. upnp (was alot of work but dunno about it), transmission, docker, nft-qos?, etc. etc. etc.

most popular services from what i've seen from some superficial scans seem to be working really well (banip, pbr, etc. etc.)


right now I'm not planning on doing much, but a routers FW will inevitably, enter the realm in terms of operation and needs... (but customisation / efficiency / bugfixing are always of interest)

anyway, don't really expect anyone to read all that jibberish but was kinda necessary to put down a 'state of affairs' and 'call to action' regarding issues/needs moving forward with the iptables > nft migration


i.e. just a lil example of running firewall restart manually on my loooooong held / munged fw config results in carps of (benign output)... related to the change... (i should really start fresh ;)

-disabled sections ok -old includes no longer running [didnt need/use them but you may have one / some] (requires 'option fw4_compatible 1' ~ after you hand test!) -new nft includes ok-ish need to peruse -reload not supported

[root@rpi-dca6325631 ../_argon_2.3.1 59°] /etc/init.d/firewall restart
Section @rule[1] (BLOCKDoT4) is disabled, ignoring section
Section @rule[2] (BLOCKDoT6) is disabled, ignoring section
Section @rule[7] (mqqtoggle-test1) is disabled, ignoring section
Section @rule[8] (mqqtoggle-fish-extra) is disabled, ignoring section
Section @rule[14] (Support-UDP-Traceroute) is disabled, ignoring section
Section @include[0] option 'reload' is not supported by fw4
Section @include[0] is not marked as compatible with fw4, ignoring section
Section @include[0] requires 'option fw4_compatible 1' to be considered compatible
Section agh option 'reload' is not supported by fw4
Section agh is disabled, ignoring section
Automatically including '/usr/share/nftables.d/table-post/30-pbr.nft'
Automatically including '/usr/share/nftables.d/chain-post/mangle_forward/30-pbr.nft'
Automatically including '/usr/share/nftables.d/chain-post/mangle_input/30-pbr.nft'
Automatically including '/usr/share/nftables.d/chain-post/mangle_output/30-pbr.nft'
Automatically including '/usr/share/nftables.d/chain-post/mangle_postrouting/30-pbr.nft'
Automatically including '/usr/share/nftables.d/chain-post/mangle_prerouting/30-pbr.nft'

FIN

(well some few words on ipset entries masq/fw maybe in future)

LionHeartP commented 9 months ago

Thank you for the update! While the update finished successfully (?) there seems to be no DNS resolving happening? Not sure what's wrong, but I can ping IPs successfully while no pages open on any device in the network.

Can't see anything suspicious in the logs. What could have triggered this behavior?

edit: Gave up and started clean. All issues fixed :)

wulfy23 commented 9 months ago

Thanks for the report... and bummer functional client network did not come up.

Most likely thing would be a service I don't use (https-dns?), or maybe something stale in the dnsmasq config...

from memory when my masq didn't come up last time (similar underlying system), that was when I added some code to tear down stuff other services put in there (but https dns is the only one that has prevented functioning clients)... on upgrade and firstboot... chance those things I added messed it up too...

(note: a dmesg and a logread from the effected system might have shown some obvious signs but yeah, given timing / readership etc. starting fresh is probably better anyways)

anyway, glad you started fresh and are back up and running... were I not the maintainer, I should be doing the same!