2018/07/27/ipset-save-with-ufw-and-iptables-persistent-and

[selivan]

This is the comment thread for Persistent ipset for Ubuntu/Debian compatible with ufw and iptables-persistent.

[dalek-sec-pest-control]

Thank you for this thread and sharing your idea. Using UFW and wanted to use a large ipset for filtering. Very helpful to have this.

PS: you credited the wrong user on your update. It should have credited: https://www.reddit.com/user/Derhomp

[dalek-sec-pest-control]

(small edit: on your install, it should "ipset-persistent.service")

[selivan]

@dalek-sec-pest-control I'm glad it was useful to you :) And thanks for the hints, fixed reddit user link and install command.

[karavan750]

ExecStop=/bin/ipset flush ExecStopPost=/bin/ipset destroy

ExecStop=/sbin/ipset flush ExecStopPost=/sbin/ipset destroy

[selivan]

@karavan750 Thanks! Fixed that.

[kelly7552]

My post startup is blowing up on an emplty file ConditionFileNotEmpty=/etc/iptables/ipset

What's supposed to be in /etc/iptables/ipset?

● ipset-persistent.service - ipset persistancy service Loaded: loaded (/etc/systemd/system/ipset-persistent.service; enabled; vendor preset: enabled) Active: inactive (dead) Condition: start condition failed at Wed 2019-06-05 13:40:16 PDT; 17min ago └─ ConditionFileNotEmpty=/etc/ipsets.conf was not met

[selivan]

@kelly7552

What's supposed to be in /etc/iptables/ipset?

Your ipset rules, which you can save with ipset save > /etc/iptables/ipset.

No rules - no reason to start service.

[mgardi91]

Hihi, first of all, thank you very much for this! But now, I have a question, I wanted, that ipset saves the Rules before the shutdown. I uncommented the line, but it doesn't save the new lists. I think the problem is, that ipset needs sudo priv., to save the file. This is my first Linux system (Ubuntu Server 18.04), so please have compassion. :'D

[selivan]

Hi @mgardi91

systemd unit runs from root user, unless it has another option User. So it has the required privileges.

You can use journalctl to see unit logs: journalctl -u ipset-persistent.service.

Let's check if it shows something interesting.

[mgardi91]

Hi @selivan , thank you very much for ur respond! Like I said, I am very new, so I hope, this is the log you need:

admin@vmd42824:~$ sudo journalctl -u ipset-persistent.service -- Logs begin at Thu 2019-09-05 12:35:53 CEST, end at Tue 2019-09-10 00:01:42 CEST. -- Sep 05 14:49:36 vmd42824.contaboserver.net systemd[1]: Started ipset persistent configuration. -- Reboot -- Sep 05 14:59:44 vmd42824.contaboserver.net systemd[1]: Started ipset persistent configuration. -- Reboot -- Sep 05 22:07:20 vmd42824.contaboserver.net systemd[1]: Started ipset persistent configuration. -- Reboot -- Sep 05 22:42:22 vmd42824.contaboserver.net systemd[1]: Started ipset persistent configuration. -- Reboot -- Sep 06 13:32:38 vmd42824.contaboserver.net systemd[1]: Started ipset persistent configuration. -- Reboot -- Sep 06 13:38:09 vmd42824.contaboserver.net systemd[1]: Started ipset persistent configuration. -- Reboot -- Sep 06 14:34:37 vmd42824.contaboserver.net systemd[1]: Started ipset persistent configuration. -- Reboot -- Sep 06 15:54:07 vmd42824.contaboserver.net systemd[1]: Started ipset persistent configuration. -- Reboot -- Sep 07 12:40:25 vmd42824.contaboserver.net systemd[1]: Started ipset persistent configuration. -- Reboot -- Sep 07 14:18:47 vmd42824.contaboserver.net systemd[1]: Started ipset persistent configuration. -- Reboot -- Sep 08 00:04:08 vmd42824.contaboserver.net systemd[1]: Started ipset persistent configuration. -- Reboot -- Sep 09 01:50:09 vmd42824.contaboserver.net systemd[1]: Started ipset persistent configuration. -- Reboot -- Sep 09 03:21:53 vmd42824.contaboserver.net systemd[1]: Started ipset persistent configuration.

May the following information will help too, I have three Users:

1. root (of course)
2. admin (with sudo) and
3. teamspeak (as a system user)

And a little other question, I read, that if I change my Server-Name, that I have to change a few other files too?

[selivan]

@mgardi91 If you change server name, you may also want to add it to /etc/hosts to be resolved to 127.0.0.1, in addition to localhost. glibc needs that sometimes.

Logs show that unit isn't stopped properly on system shutdown, I will look into it when I have time.

[pepoluan]

Awesome idea! From your script, I derived another script that automagically creates the prerequisite file so the service can be "installed" and run without any prerequisite procedures.

Here it is: https://gitlab.com/snippets/1966389

Full credits are given, of course.

[selivan]

@pepoluan Glad to see that my work is useful. Merged one of your changes back into the article, about DefaultDependencies. No idea how I got that line into the script, probably copy/paste gone wrong. Mentioned you in the article.

@mgardi91 Without DefaultDependencies line the unit should be able to save ipset rules on shutdown.

[bluebrown]

This makes systemd-networkd-wait-online.service fail on ubuntu 20.

It is working in itself, ipsets and iptables are restored properly but the mentioned service fails on boot.

Could it be becaue ufw is disabled?

[selivan]

@bluebrown

Looks like a bug :beetle: . Let's try debug it:

1. If ufw if disabled, are you using iptables-persistent?
2. Check that disabling ipset-persistent.service fixes the problem
3. Disable ipset-persistent.service, change iptables rules so that ipset is not used - replace them with simple IP address rules - and see if the problem stays. If it does, then your iptables rules are affecting systemd-networkd-wait-online, not ipset.
4. Show you iptables rules and ipset rules

I do not have much time now, but I will try to look at that soon.

P.S. If you decide to look for help at stackoverflow or serverfault - please post the link to the question here.

[selivan]

@bluebrown Quickly tried this setup on virtual machine with Ubuntu 20.04 LTS:

• ipset-persistent.service set like in the article
• iptables-persistent is installed

/etc/iptables/rules.v4:

# Generated by iptables-save v1.8.4 on Sun Apr  4 17:53:08 2021
*filter
:INPUT DROP [0:0]
-A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -p tcp -m tcp --dport 22 -m set --match-set myset-ip src -j REJECT
-A INPUT -p tcp -m tcp --dport 22 -j ACCEPT
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [0:0]
COMMIT
# Completed on Sun Apr  4 17:53:08 2021

/etc/iptables/ipset :

create myset-ip hash:ip family inet hashsize 1024 maxelem 65536
add myset-ip 2.2.2.2
add myset-ip 3.3.3.3

systemd-networkd-wait-online.service looks fine:

● systemd-networkd-wait-online.service - Wait for Network to be Configured
     Loaded: loaded (/lib/systemd/system/systemd-networkd-wait-online.service; enabled; vendor preset: enabled)
     Active: active (exited) since Sun 2021-04-04 18:04:07 UTC; 2min 29s ago
       Docs: man:systemd-networkd-wait-online.service(8)
    Process: 759 ExecStart=/lib/systemd/systemd-networkd-wait-online (code=exited, status=0/SUCCESS)
   Main PID: 759 (code=exited, status=0/SUCCESS)

Apr 04 18:04:06 ubuntu2004.localdomain systemd[1]: Starting Wait for Network to be Configured...
Apr 04 18:04:07 ubuntu2004.localdomain systemd[1]: Finished Wait for Network to be Configured.

[bluebrown]

Yes I am using persistent iptables alongside. ufwis installed but disabled. I just the raw iptables.

1. if I remove ufwfrom the script it still hangs
2. if I disable ipset-persistentt it still hangs, (I have also removed violating rules before).
3. If I disable both it boots up with no issue

So it appears to be related to netfilter-persistent and not to ipset-persistent

[selivan]

@bluebrown Well, now you can debug this using binary search: disable half the iptables rules, see if the problem stays, see which half has the problem, disable half of that half, and so on.

Please be kind to write the result of the research here. I'd like to know a little more about systemd-networkd-wait-online, which probably works in dark mysterious ways, like systemd itself :)

[bluebrown]

Well, I figured it out. It just had this realization and yes it turned out to be true it also makes sense.

Those were my rules.v6, as I thought I don't need ipv6, so I want to block it completely.

*filter
:INPUT DROP [0:0]
:FORWARD DROP [0:0]
:OUTPUT DROP [0:0]
COMMIT

Of course some internal things like network-manager and networkd are still trying to check things out. So I have dropped packets from networkd which caused the event loop to time out.

I should allow at least local interface for this I guess.

I can also disableipv6 via the below in /etc/sysctl.d/disableipv6.conf

net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1

[stefanlasiewski]

Thank you so much for this. This has really been bugging me. What's worse is that netfilter-persistent will actually fail to load any rules if /etc/iptables/rules.v4 contains any ipset rules. That seems like a serious bug and Ubuntu has ignored it.

After I installed this, I ran into one additional problem. If I run the restart command, ipset will complain about pre-existing rules:

# systemctl restart ipset-persistent.service
Error: Systemd start for ipset-persistent.service failed!
journalctl log for ipset-persistent.service:
-- Logs begin at Tue 2021-02-09 18:49:35 PST, end at Thu 2021-05-13 16:24:07 PDT. --
May 13 16:22:54 docker01 systemd[1]: Starting ipset persistent configuration...
May 13 16:22:54 docker01 ipset[27045]: ipset v6.34: Error in line 1: Set cannot be created: set with the same name already exists
May 13 16:22:54 docker01 systemd[1]: ipset-persistent.service: Main process exited, code=exited, status=1/FAILURE

To fix this, I added -exist to the following command:

ExecStart=/sbin/ipset restore -file /etc/iptables/ipset

So it now reads like this:

ExecStart=/sbin/ipset restore -exist -file /etc/iptables/ipset

The manpage says this about -exist:

If the -exist option is specified, ipset ignores the error otherwise raised when the  same set (setname and create parameters are identical) already exists.

[selivan]

Hey @stefanlasiewski

I poked -exist option, it is a bit dangerous because ipset restore -exist -file <file> will not load the sets from <file>, but will merge sets from file with existing sets. However, in this case we can use it, because on restart first we flush existing ip sets.

Updated the article, thanks for the contribution!

[markdesilva]

This is probably a stupid question but what happens if I have 2 or more lists? Do I just ignore the ConditionFileNotEmpty and just string together multiple ipset restore commands into ExecStart?

Thank you.

[selivan]

@markdesilva

ipset save > /etc/iptables/ipset command will save all your lists to the file, you don't have to use separate file for each one.

[markdesilva]

Ah ok! I was under the misguided notion that each list had to be loaded separately!

Thank you!

[threehappypenguins]

Duuuuuuuuuude! You just saved the day! I have asterisk running on a free tier GCP instance, and the egress to Australia and China was costing me money because Google charges for egress to those countries. So I figured out how to use ipset (I use a script) to run periodically and update CIDR ip addresses to drop OUTPUT for iptables. But upon reboot, iptables-persistent wouldn't run because ipset wasn't persistent. So I was scratching my head trying to figure out some way to make ipset persistent. The only thing I would add is I did sudo chmod +x /etc/systemd/system/ipset-persistent.service before reloading the daemon, and I also needed to save the ipset rules to a file first (or I would get Active: inactive (dead) when checking the status of the new service); note, I had to run this as root user:

sudo su
ipset save > /etc/iptables/ipset
exit

Thanks so much for this!!!

[selivan]

@threehappypenguins You are welcome! Updated the post a little, adding note about saving ipsets to /etc/iptables/ipset.

chmod +x /etc/systemd/system/ipset-persistent.service - that's weird, systemd does not require unit config files to be executable.

Thank you for the feedback, it is nice to know your work helped somebody )

[threehappypenguins]

@selivan you're probably right about not needing the chmod permissions. I was getting the inactive (dead) error and thought that I needed to give it more permissions. I suppose I can always remove the permissions and see if it still works, just to confirm.

[threehappypenguins]

@selivan I tried to change the file permissions back to -rw-r--r-- with sudo chmod 644 but then it didn't run right. So I nuked the whole thing by disabling it and starting over with a new file, but skipping the chmod +x step and everything worked fine!