Crowdsec ipset rules do not survive a firewalld reload

[stephdl]

Steps to reproduce

• Install crowdsec
• once installed check if the ipv4/ipv6 sets of ipset are in iptables

  
  [root@R1-pve ~]# ip6tables -L
  Chain INPUT (policy ACCEPT)
  target     prot opt source               destination         
  DROP       all      anywhere             anywhere             match-set crowdsec6-blacklists src

Chain FORWARD (policy ACCEPT) target prot opt source destination

Chain OUTPUT (policy ACCEPT) target prot opt source destination

[root@R1-pve ~]# iptables -L Chain INPUT (policy ACCEPT) target prot opt source destination
DROP all -- anywhere anywhere match-set crowdsec-blacklists src

Chain FORWARD (policy ACCEPT) target prot opt source destination

Chain OUTPUT (policy ACCEPT) target prot opt source destination

reload the firewall like agent.add_public_service after it has added opened services 

firewall-cmd --reload

**Expected behavior**

I expect that my sets are loaded

**Actual behavior**

The sets are not loaded anymore and obviously crwdsec can ban, the drop is not honored 

[root@R1-pve ~]# iptables -L Chain INPUT (policy ACCEPT) target prot opt source destination

Chain FORWARD (policy ACCEPT) target prot opt source destination

Chain OUTPUT (policy ACCEPT) target prot opt source destination

[root@R1-pve ~]# ip6tables -L Chain INPUT (policy ACCEPT) target prot opt source destination

Chain FORWARD (policy ACCEPT) target prot opt source destination

Chain OUTPUT (policy ACCEPT) target prot opt source destination

however if I restart  crowdsec-firewall-bouncer.service the sets are back

`systemctl restart  crowdsec-firewall-bouncer.service`

I tried to make permanent the set to the drop zone of firewalld or with also a rich-rule of the public zone but we face to a strange behavior with two kinds of sets, crowdset wants to use the set that you can see with `ipset -L -n` and since I want to make `permanent` the set to firewalld, it looks like there is a conflict

**Components**

crowdsec 1.0.6

[stephdl]

QA

Install crowdsec ghcr.io/nethserver/crowdsec:1.0.7-dev.3

Now the test is really fun, we run the bouncer inside a container, however the test must be done on debian and rocky9

As a side note we do not remove permanent rules of firewalld nor remove deb and rpm of crowdsec-firewall-bouncer-iptables, we just stop and disable it

test case 1

• direct install add-module ghcr.io/nethserver/crowdsec:1.0.7-dev.3
• Verify the service are up systemctl status crowdsec1 crowdsec1-firewall-bouncer
• verify that you can see the nftables sets (possibly you have ips inside, better is yes)

  nft list set ip crowdsec crowdsec-blacklists
  nft list set ip6 crowdsec6 crowdsec6-blacklists

• remove the whitelist and try to be banned on your LAN by SSH

  cscli parsers remove  crowdsecurity/whitelists
  systemctl restart crowdsec1

• you can see the BAN by cscli decisions list
• you can remove the ban by : cscli decisions delete -i xxx.xxx.xxx.xxx
• verify the nft rules are not removed by systemctl restart firewalld && firewall-cmd --reload
• stop the crowdsec1-firewall-bouncer, the rules in nft must be removed

test case 2

• install add-module ghcr.io/nethserver/crowdsec:1.0.6
• now upgrade api-cli run update-module --data '{"module_url":"ghcr.io/nethserver/crowdsec:1.0.7-dev.3","instances":["crowdsec1"],"force":true}'
• Verify the service are up systemctl status crowdsec1 crowdsec1-firewall-bouncer
• verify the former service of crowdsec-firewall-bouncer is down systemctl status crowdsec-firewall-bouncer
• verify the old set of ipsets are empty, it should but not an issue if they have a timeout (automatically removed once the timeout is ended) ipset -L iptables -L
• verify that you can see the nftables sets (possibly you have ips inside, better is yes)

  nft list set ip crowdsec crowdsec-blacklists
  nft list set ip6 crowdsec6 crowdsec6-blacklists

• remove the whitelist and try to be banned on your LAN by SSH

  cscli parsers remove  crowdsecurity/whitelists
  systemctl restart crowdsec1

• you can see the BAN by cscli decisions list
• you can remove the ban by : cscli decisions delete -i xxx.xxx.xxx.xxx
• verify the nft rules are not removed by systemctl restart firewalld && firewall-cmd --reload
• stop the crowdsec1-firewall-bouncer, the rules in nft must be removed

[mrmarkuz]

Test case 1: OK

Test case 2: NOK

On a single Rocky node it works. On a Debian cluster node after a firewall-cmd --reload the rules aren't there anymore:

# iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

[DavidePrincipi]

QA note

In some Debian installations (i.e. Digital Ocean hosting) the nft command is not installed.

apt install nftables

[DavidePrincipi]

In testing crowdsec 1.0.7-dev.4

[DavidePrincipi]

Test case

Repeat test cases 1 and 2. Notice that

1. new testing version 1.0.7-dev.4

2. the cscli command was moved into the module environment. Invocation becomes for instance:

   runagent -m crowdsec1 cscli ...

3. the bouncer service name is now crowdsec1-firewall-bouncer.service (notice the MODULE_ID is used as service name prefix)

   

[nrauso]

on Rocky Linux:

• test case 1: VERIFIED
• test case 2: VERIFIED

On Debian:

• test case 1: VERIFIED
• test case 2: VERIFIED

Note for Debian: used DO VPS so I needed to install nftables tool.

[DavidePrincipi]

VERIFIED

Verified additional fix, and test case https://community.nethserver.org/t/ns8-crowdsec-limited-domain-levels-in-allow-list/23301/7

[DavidePrincipi]

Released in https://github.com/NethServer/ns8-crowdsec/releases/tag/1.0.7