search

crowdsecurity / cs-firewall-bouncer

Crowdsec bouncer written in golang for firewalls
MIT License
119 stars 43 forks source link

Can't start iptables bouncer with ipv6 setting turned on and DOCKER-USER chain enabled #346

Open mendozal opened 10 months ago

mendozal commented 10 months ago

I'm having trouble trying to start the iptables firewall bouncer with ipv6 enabled and these settings:

Ubuntu 22.04 UFW enabled and needed for the specific docker services Docker IPV6 available and configured

This is my crowdsec-firewall-bouncer.yaml config

mode: iptables
update_frequency: 10s
log_mode: file
log_dir: /var/log/
log_level: info
log_compression: true
log_max_size: 100
log_max_backups: 3
log_max_age: 30
api_url: http://127.0.0.1:8080/
api_key: ---REDACTED ---
insecure_skip_verify: false
disable_ipv6: false
deny_action: DROP
deny_log: false
supported_decisions_types:
  - ban
#to change log prefix
#deny_log_prefix: "crowdsec: "
#to change the blacklists name
blacklists_ipv4: crowdsec-blacklists
blacklists_ipv6: crowdsec6-blacklists
#type of ipset to use
ipset_type: nethash
#if present, insert rule in those chains
iptables_chains:
  - INPUT
#  - FORWARD
  - DOCKER-USER

## nftables
nftables:
  ipv4:
    enabled: true
    set-only: false
    table: crowdsec
    chain: crowdsec-chain
    priority: -10
  ipv6:
    enabled: true
    set-only: false
    table: crowdsec6
    chain: crowdsec6-chain
    priority: -10

nftables_hooks:
  - input
  - forward

# packet filter
pf:
  # an empty string disables the anchor
  anchor_name: ""

prometheus:
  enabled: false
  listen_addr: 127.0.0.1
  listen_port: 60601

If I leave the disable_ipv6 to false, the service fails to start

These are the bouncer's logs

time="27-12-2023 18:24:19" level=info msg="Starting crowdsec-firewall-bouncer v0.0.28-debian-pragmatic-af6e7e25822c2b1a02168b99ebbf8458bc6728e5"
time="27-12-2023 18:24:19" level=info msg="backend type : iptables"
time="27-12-2023 18:24:19" level=info msg="iptables for ipv4 initiated"
time="27-12-2023 18:24:19" level=info msg="iptables clean-up : /usr/sbin/iptables -D INPUT -m set --match-set crowdsec-blacklists src -j DROP"
time="27-12-2023 18:24:19" level=info msg="iptables clean-up : /usr/sbin/iptables -D DOCKER-USER -m set --match-set crowdsec-blacklists src -j DROP"
time="27-12-2023 18:24:19" level=info msg="ipset clean-up : /usr/sbin/ipset -exist destroy crowdsec-blacklists"
time="27-12-2023 18:24:19" level=info msg="Checking existing set"
time="27-12-2023 18:24:19" level=info msg="ipset set-up : /usr/sbin/ipset -exist create crowdsec-blacklists nethash timeout 300 maxelem 131072"
time="27-12-2023 18:24:20" level=info msg="Rule doesn't exist (/usr/sbin/iptables -C INPUT -m set --match-set crowdsec-blacklists src -j DROP)"
time="27-12-2023 18:24:20" level=info msg="Rule doesn't exist (/usr/sbin/iptables -C DOCKER-USER -m set --match-set crowdsec-blacklists src -j DROP)"
time="27-12-2023 18:24:20" level=info msg="iptables set-up : /usr/sbin/iptables -I INPUT -m set --match-set crowdsec-blacklists src -j DROP"
time="27-12-2023 18:24:20" level=info msg="iptables set-up : /usr/sbin/iptables -I DOCKER-USER -m set --match-set crowdsec-blacklists src -j DROP"
time="27-12-2023 18:24:20" level=info msg="iptables for ipv6 initiated"
time="27-12-2023 18:24:20" level=info msg="iptables clean-up : /usr/sbin/ip6tables -D INPUT -m set --match-set crowdsec6-blacklists src -j DROP"
time="27-12-2023 18:24:20" level=info msg="iptables clean-up : /usr/sbin/ip6tables -D DOCKER-USER -m set --match-set crowdsec6-blacklists src -j DROP"
time="27-12-2023 18:24:20" level=error msg="error while removing set entry in iptables : exit status 1 --> ip6tables: Bad rule (does a matching rule exist in that chain?).\n"
time="27-12-2023 18:24:20" level=info msg="ipset clean-up : /usr/sbin/ipset -exist destroy crowdsec6-blacklists"
time="27-12-2023 18:24:20" level=error msg="set destroy error : exit status 1 - ipset v7.15: Set cannot be destroyed: it is in use by a kernel component\n"
time="27-12-2023 18:24:20" level=info msg="Checking existing set"
time="27-12-2023 18:24:21" level=warning msg="iptables check command (/usr/sbin/ip6tables -C INPUT -m set --match-set crowdsec6-blacklists src -j DROP) failed : exit status 1"
time="27-12-2023 18:24:21" level=warning msg="iptables check command (/usr/sbin/ip6tables -C DOCKER-USER -m set --match-set crowdsec6-blacklists src -j DROP) failed : exit status 1"
time="27-12-2023 18:24:21" level=info msg="iptables set-up : /usr/sbin/ip6tables -I INPUT -m set --match-set crowdsec6-blacklists src -j DROP"
time="27-12-2023 18:24:21" level=info msg="iptables set-up : /usr/sbin/ip6tables -I DOCKER-USER -m set --match-set crowdsec6-blacklists src -j DROP"
time="27-12-2023 18:24:21" level=warning msg="Error inserting set in iptables (/usr/sbin/ip6tables -I DOCKER-USER -m set --match-set crowdsec6-blacklists src -j DROP): exit status 1 : ip6tables: No chain/target/match by that name.\n"
time="27-12-2023 18:24:21" level=fatal msg="iptables init failed: while inserting set in iptables: exit status 1"

If I set disable_ipv6 to true, the service starts fine, no mention of crowdsec6-blacklists in the logs.

If I disable the DOCKER-USER chain but leave disable_ipv6 to false in the bouncer's config I still get a few errors, but the bouncer starts.

time="27-12-2023 18:32:33" level=info msg="Starting crowdsec-firewall-bouncer v0.0.28-debian-pragmatic-af6e7e25822c2b1a02168b99ebbf8458bc6728e5"
time="27-12-2023 18:32:33" level=info msg="backend type : iptables"
time="27-12-2023 18:32:33" level=info msg="iptables for ipv4 initiated"
time="27-12-2023 18:32:33" level=info msg="iptables clean-up : /usr/sbin/iptables -D INPUT -m set --match-set crowdsec-blacklists src -j DROP"
time="27-12-2023 18:32:33" level=error msg="error while removing set entry in iptables : exit status 1 --> iptables: Bad rule (does a matching rule exist in that chain?).\n"
time="27-12-2023 18:32:33" level=info msg="ipset clean-up : /usr/sbin/ipset -exist destroy crowdsec-blacklists"
time="27-12-2023 18:32:33" level=info msg="Checking existing set"
time="27-12-2023 18:32:33" level=info msg="ipset set-up : /usr/sbin/ipset -exist create crowdsec-blacklists nethash timeout 300 maxelem 131072"
time="27-12-2023 18:32:34" level=info msg="Rule doesn't exist (/usr/sbin/iptables -C INPUT -m set --match-set crowdsec-blacklists src -j DROP)"
time="27-12-2023 18:32:34" level=info msg="iptables set-up : /usr/sbin/iptables -I INPUT -m set --match-set crowdsec-blacklists src -j DROP"
time="27-12-2023 18:32:34" level=info msg="iptables for ipv6 initiated"
time="27-12-2023 18:32:34" level=info msg="iptables clean-up : /usr/sbin/ip6tables -D INPUT -m set --match-set crowdsec6-blacklists src -j DROP"
time="27-12-2023 18:32:34" level=error msg="error while removing set entry in iptables : exit status 1 --> ip6tables: Bad rule (does a matching rule exist in that chain?).\n"
time="27-12-2023 18:32:34" level=info msg="ipset clean-up : /usr/sbin/ipset -exist destroy crowdsec6-blacklists"
time="27-12-2023 18:32:34" level=info msg="Checking existing set"
time="27-12-2023 18:32:34" level=info msg="ipset set-up : /usr/sbin/ipset -exist create crowdsec6-blacklists nethash timeout 300 family inet6 maxelem 131072"
time="27-12-2023 18:32:35" level=warning msg="iptables check command (/usr/sbin/ip6tables -C INPUT -m set --match-set crowdsec6-blacklists src -j DROP) failed : exit status 1"
time="27-12-2023 18:32:35" level=info msg="iptables set-up : /usr/sbin/ip6tables -I INPUT -m set --match-set crowdsec6-blacklists src -j DROP"
time="27-12-2023 18:32:35" level=info msg="Using API key auth"
time="27-12-2023 18:32:35" level=info msg="config is valid"
time="27-12-2023 18:32:35" level=info msg="Shutting down backend"
time="27-12-2023 18:32:35" level=info msg="iptables clean-up : /usr/sbin/iptables -D INPUT -m set --match-set crowdsec-blacklists src -j DROP"
time="27-12-2023 18:32:35" level=info msg="ipset clean-up : /usr/sbin/ipset -exist destroy crowdsec-blacklists"
time="27-12-2023 18:32:35" level=error msg="set destroy error : exit status 1 - ipset v7.15: Set cannot be destroyed: it is in use by a kernel component\n"
time="27-12-2023 18:32:35" level=info msg="iptables clean-up : /usr/sbin/ip6tables -D INPUT -m set --match-set crowdsec6-blacklists src -j DROP"
time="27-12-2023 18:32:35" level=info msg="ipset clean-up : /usr/sbin/ipset -exist destroy crowdsec6-blacklists"
time="27-12-2023 18:32:35" level=error msg="set destroy error : exit status 1 - ipset v7.15: Set cannot be destroyed: it is in use by a kernel component\n"
time="27-12-2023 18:32:35" level=info msg="Starting crowdsec-firewall-bouncer v0.0.28-debian-pragmatic-af6e7e25822c2b1a02168b99ebbf8458bc6728e5"
time="27-12-2023 18:32:35" level=info msg="backend type : iptables"
time="27-12-2023 18:32:35" level=info msg="iptables for ipv4 initiated"
time="27-12-2023 18:32:35" level=info msg="iptables clean-up : /usr/sbin/iptables -D INPUT -m set --match-set crowdsec-blacklists src -j DROP"
time="27-12-2023 18:32:35" level=error msg="error while removing set entry in iptables : exit status 1 --> iptables: Bad rule (does a matching rule exist in that chain?).\n"
time="27-12-2023 18:32:35" level=info msg="ipset clean-up : /usr/sbin/ipset -exist destroy crowdsec-blacklists"
time="27-12-2023 18:32:35" level=info msg="Checking existing set"
time="27-12-2023 18:32:35" level=info msg="ipset set-up : /usr/sbin/ipset -exist create crowdsec-blacklists nethash timeout 300 maxelem 131072"
time="27-12-2023 18:32:36" level=info msg="Rule doesn't exist (/usr/sbin/iptables -C INPUT -m set --match-set crowdsec-blacklists src -j DROP)"
time="27-12-2023 18:32:36" level=info msg="iptables set-up : /usr/sbin/iptables -I INPUT -m set --match-set crowdsec-blacklists src -j DROP"
time="27-12-2023 18:32:36" level=info msg="iptables for ipv6 initiated"
time="27-12-2023 18:32:36" level=info msg="iptables clean-up : /usr/sbin/ip6tables -D INPUT -m set --match-set crowdsec6-blacklists src -j DROP"
time="27-12-2023 18:32:36" level=error msg="error while removing set entry in iptables : exit status 1 --> ip6tables: Bad rule (does a matching rule exist in that chain?).\n"
time="27-12-2023 18:32:36" level=info msg="ipset clean-up : /usr/sbin/ipset -exist destroy crowdsec6-blacklists"
time="27-12-2023 18:32:36" level=info msg="Checking existing set"
time="27-12-2023 18:32:36" level=info msg="ipset set-up : /usr/sbin/ipset -exist create crowdsec6-blacklists nethash timeout 300 family inet6 maxelem 131072"
time="27-12-2023 18:32:37" level=warning msg="iptables check command (/usr/sbin/ip6tables -C INPUT -m set --match-set crowdsec6-blacklists src -j DROP) failed : exit status 1"
time="27-12-2023 18:32:37" level=info msg="iptables set-up : /usr/sbin/ip6tables -I INPUT -m set --match-set crowdsec6-blacklists src -j DROP"
time="27-12-2023 18:32:37" level=info msg="Using API key auth"
time="27-12-2023 18:32:37" level=info msg="Processing new and deleted decisions . . ."
time="27-12-2023 18:32:47" level=info msg="15002 decisions added"
LaurenceJJones commented 10 months ago

You need to enable ipv6 support for docker, because DOCKER-USER by default only applies to ipv4 and since the chain doesn't exist we cannot bind to the table on ipv6. The chains yaml configuration is both ipv4 and ipv6, currently we dont have a way to configure either unless you change to ipset only mode and write the rules yourself.

Duplicate of #227

mendozal commented 10 months ago

I see it now.

I was confused because IPv6 was enabled and working in general, but the internal docker network has it disabled.

I requested this to que software provider and they will look into enabling IPv6 support for the internal docker network(s). I'll update this issue if that solves it.

Bluematrix2 commented 10 months ago

I did run into the same issue. Maybe worth adding a note on the installation page of the firewall-bounce.