Add --save-ipset

[SpareSimian]

I want to replace all entries in an existing ipset with a new list. It seems the easiest way is to flush the current set and load the new list with --add-entries-from-file. But there doesn't seem to be an explicit way to invoke the underlying ipset flush function. Note that I don't want to delete the set because it's in use by a direct rule.

[SpareSimian]

What's really needed is a way to snapshot the current ipset in the kernel back to the XML file so that it can be reloaded on boot or firewall restart. Any external manipulations can be done directly through the ipset command/API, and then firewalld can be tasked to save the result. Something like --runtime-to-permanent but for one ipset.

[pgnd]

Just getting started exploring FWD, and have bumped into its ipset handling.

I've a not-unrelated goal -- to be able to swap ipsets. It falls under the 'integrate better' and 'do more stuff' with ipset's full cmd set wishlist ...

e.g., use case:

I've a pre-existing ipset It's large It's externally managed It's asynchronously updated, independently of any FW start/stop, typically on a crontab &/or push-from-src schedule. It's maintained & snapshotted/persisted, externally under ipset command control, in .ipset form -- not xml.

In order to "--new-ipset-from-file" in FWD, it's necessary to create/convert-to xml, then load that.

That's slow. Particularly if there's a large change in the dataset.

Ideally, it'd be useful to be able to ipset-swap (e.g., http://ipset.netfilter.org/tips.html) within firewall-cmd ...

The entire need to identify & manage external ipsets with addition of fixed, direct.xml rules is clumsy, at best, imo. Particularly if, e.g., I want to use multiple chains and do some conditional branching -- as in the case of wanting to "-j DROP" only if "in THIS ipset, but not THAT ipset". I.e., an effective

--match-set THIS_IPSET!THAT_IPSET

exclusion (Shorewall does this very nicely, fwiw)

[erig0]

@pgnd, your request sounds different than @SpareSimian's. If you agree, then maybe you should open a new issue.

[pgnd]

@erig0

Hi. fwiw, I don't 'agree' -- I think it's very much related, and depending on how 'my' issue is addressed, it can well directly affect how @SpareSimian's is addressed. i.e., the common thread here is the moving in-and-out of ipset data-- whether it's originated 'from' internal, FWD xml-> external ipset, or 'from' external ipset to internal xml format.

But NOT thinking thru the whole ipset mgmt structure holistically, including some real-world use cases (currently, it's a slog to get anywhere near the capabilities of Shorewall, e.g.), is IMO a mistake, and a lost opportunity.

That said, not my project -- so I'll do what folks want here.

[SpareSimian]

I agree with pgnd. The issue is how to update an ipset externally and keep the kernel and XML in sync. The update may be large and involve both inserts and deletes, so the swap feature should be used for the kernel part of the update function.

I think the best approach is for the external app to create a new temporary ipset in the kernel and then use a firewalld command to swap it into place over the desired kernel ipset while updating the XML file.

I'm continuing to use firewalld to create the ipset and hold its initial contents in XML, because it's used by a direct rule. But my external (cron-driven) process will bulk-update the ipset once firewalld is up.

[pgnd]

@SpareSimian

I'm continuing to use firewalld to create the ipset and hold its initial contents in XML, because it's used by a direct rule

I'm of mixed-mind on this :-/ Likely due to some intertia on my part ...

With Shorewall, to date, I've been doing all of my big-&-frequently-changing ipset mgmt (creation/update/swaps) externally. It's clean, and, IMO, correctly separated from a FW app. Shorewall then has simply used those sets in its rules/compiler referenced by ipset name. This allows for convenient/simply rule manipulation of ipset data (setting NOTRACK, set intersections, etc etc)

I've been trying to get to the equivalent here with FWD ...

So far, without the swap, I've managed nothing elegant or pereformant. I have managed to get quite annoyed with having to write/manage noisy xml in direct.xml ...

EDIT: If the whole nft(ables)/(e)BFP biz was clearer to mere mortals, and further along (?)

https://developers.redhat.com/blog/2018/08/10/firewalld-the-future-is-nftables/ https://firewalld.org/2018/07/nftables-backend https://www.networkworld.com/article/3299457/linux/linux-kernel-418-better-security-leaner-code.html

I might guess the right answer here is to switch to eBFP. Which, iiuc, includes much/all of ipset's capabilities ... Unless I'm misreading it, it's still far too soon, and iptables is gonna necessarily be around a good while longer

[SpareSimian]

What would you use external to firewalld to initialize your ipsets? I found this 3rd party systemd unit:

https://github.com/BroHui/systemd-ipset-service

[pgnd]

Hadn't seen that one ...

I've done effectively similar DIY.

A bash script lists my ipsets&configs, creates or updates, etc.

It's invoked by a systemd script for on-boot mgmt.

My ipset-data cronjobs then either swap their "tmp.ipset" with the existing ones after a successful update, or call a function within the bash script.