Open Icosa-Consulting opened 8 months ago
Also,
My bad for not using the template. I just wanted to post this more for awareness.
Hi @mmetc I can confirm the problem with the current 0.0.29-rc3 on OpenWrt.
If I use the set-only mode where I create the nftables rules with the scripts of OpenWrt package the log file states:
time="2024-07-16T11:54:47Z" level=error msg="can't collect dropped packets for ipv4 from nft: while running /usr/sbin/nft -j list chain ip crowdsec crowdsec-chain: exit status 1"
The names of the tables (crowdsec
, crowdsec6
) and chains are identical to the ones created by the bouncer itself if not running in set-only mode.
The names of the chains are indeed crowdsec-chain-input
, crowdsec-chain-forward
, crowdsec6-chain-input
and crowdsec6-chain-forward
whereas the configured chain names in the config are crowdsec-chain
and crowdsec6-chain
.
I'm not sure what the problem is but it seems as if the -input and -forward postfixes are not used when trying to build the chain names for collecting the data via using ntf in set-only mode.
For OpenWrt I need to create the rules myself. Even though the current version is near to what the OpenWrt package already used, in OpenWrt the filtering also needs to act on defined interfaces and conntrack state (to reduce usage of queries to the set).
This is my setup on OpenWrt with bouncer 0.0.29-rc3 for comparison:
With the OpenWrt scripts creating the rules and the bouncer running set-only:
table ip crowdsec {
set crowdsec-blacklists {
type ipv4_addr
flags timeout
elements = { ... }
}
chain crowdsec-chain-input {
type filter hook input priority filter + 4; policy accept;
iifname { "wg1", "eth1" } ct state new ip saddr @crowdsec-blacklists counter packets 0 bytes 0 drop
}
chain crowdsec-chain-forward {
type filter hook forward priority filter + 4; policy accept;
iifname { "wg1", "eth1" } ct state new ip daddr != 224.0.0.0/4 ip saddr @crowdsec-blacklists counter packets 17 bytes 928 drop
}
}
table ip6 crowdsec6 {
set crowdsec6-blacklists {
type ipv6_addr
flags timeout
elements = { ... }
}
chain crowdsec6-chain-input {
type filter hook input priority filter + 4; policy accept;
iifname { "wg1", "eth1" } ct state new ip6 saddr @crowdsec6-blacklists counter packets 0 bytes 0 drop
}
chain crowdsec6-chain-forward {
type filter hook forward priority filter + 4; policy accept;
iifname { "wg1", "eth1" } ct state new ip6 saddr @crowdsec6-blacklists counter packets 84 bytes 6720 drop
}
}
This is how the bouncer creates the rules if I don't use set-only:
table ip crowdsec {
set crowdsec-blacklists {
type ipv4_addr
flags timeout
elements = { ... }
}
chain crowdsec-chain-input {
type filter hook input priority filter + 4; policy accept;
ip saddr @crowdsec-blacklists counter packets 0 bytes 0 drop
}
chain crowdsec-chain-forward {
type filter hook forward priority filter + 4; policy accept;
ip saddr @crowdsec-blacklists counter packets 0 bytes 0 drop
}
}
table ip6 crowdsec6 {
set crowdsec6-blacklists {
type ipv6_addr
flags timeout
elements = { ... }
}
chain crowdsec6-chain-input {
type filter hook input priority filter + 4; policy accept;
ip6 saddr @crowdsec6-blacklists counter packets 0 bytes 0 drop
}
chain crowdsec6-chain-forward {
type filter hook forward priority filter + 4; policy accept;
ip6 saddr @crowdsec6-blacklists counter packets 0 bytes 0 drop
}
}
In the second version (rules created by the bouncer) metrics on blocked ips are working. In the first version (rules created by the OpenWrt script) the metrics on blocked ips are not working and the logfile is flooded with the error messages as shown above.
Hello,
I have a configuration that only sets the decisions in element lists using more customized rules for processing. when I enable the prometheus collector on the bouncer I get no stats and see errors in the logs (listed below)
Config: (Im using the .local extension to override the .yml file in /etc/crowdsec/bouncers)
NFT Chain Rule
Override Config
blacklists_ipv4: crowdsec-blacklists4 blacklists_ipv6: crowdsec-blacklists6
nftables: ipv4: enabled: true set-only: true table: crowdsec4 chain: block_ingress4 ipv6: enabled: true set-only: true table: crowdsec6 chain: block_ingress6 # nftables_hooks:
LOGS
The crowdsec-firewall-bouncer.log shows
time="29-12-2023 16:25:35" level=info msg="1 decision added" time="29-12-2023 16:25:42" level=error msg="can't collect dropped packets for ipv4 from nft: exit status 1" time="29-12-2023 16:25:42" level=error msg="can't collect total banned IPs for ipv4 from nft:exit status 1" time="29-12-2023 16:25:42" level=error msg="can't collect dropped packets for ipv6 from nft: exit status 1" time="29-12-2023 16:25:42" level=error msg="can't collect total banned IPs for ipv6 from nft:exit status 1" time="29-12-2023 16:25:52" level=error msg="can't collect dropped packets for ipv4 from nft: exit status 1" time="29-12-2023 16:25:52" level=error msg="can't collect total banned IPs for ipv4 from nft:exit status 1" time="29-12-2023 16:25:52" level=error msg="can't collect dropped packets for ipv6 from nft: exit status 1" time="29-12-2023 16:25:52" level=error msg="can't collect total banned IPs for ipv6 from nft:exit status 1" time="29-12-2023 16:26:02" level=error msg="can't collect dropped packets for ipv4 from nft: exit status 1" time="29-12-2023 16:26:02" level=error msg="can't collect total banned IPs for ipv4 from nft:exit status 1" time="29-12-2023 16:26:02" level=error msg="can't collect dropped packets for ipv6 from nft: exit status 1" time="29-12-2023 16:26:02" level=error msg="can't collect total banned IPs for ipv6 from nft:exit status 1" time="29-12-2023 16:26:12" level=error msg="can't collect dropped packets for ipv4 from nft: exit status 1" time="29-12-2023 16:26:12" level=error msg="can't collect total banned IPs for ipv4 from nft:exit status 1" time="29-12-2023 16:26:12" level=error msg="can't collect dropped packets for ipv6 from nft: exit status 1" time="29-12-2023 16:26:12" level=error msg="can't collect total banned IPs for ipv6 from nft:exit status 1" time="29-12-2023 16:26:22" level=error msg="can't collect dropped packets for ipv4 from nft: exit status 1" time="29-12-2023 16:26:22" level=error msg="can't collect total banned IPs for ipv4 from nft:exit status 1" time="29-12-2023 16:26:22" level=error msg="can't collect dropped packets for ipv6 from nft: exit status 1" time="29-12-2023 16:26:22" level=error msg="can't collect total banned IPs for ipv6 from nft:exit status 1"
Sourced from /pkg/nftables/metrics.go
metrics.go
It appears in the code the collector is looking for a specific chain name, which I have specified in the config. Which translates into:
nft -j list chain inet crowdsec4 block_ingress4
The error "exit status 1" must be coming from the NFT binary itself. If I run that command I get the following:
{ "nftables": [ { "metainfo": { "version": "1.0.6", "release_name": "Lester Gooch #5", "json_schema_version": 1 } }, { "chain": { "family": "inet", "table": "crowdsec4", "name": "block_ingress4", "handle": 1 } }, { "rule": { "family": "inet", "table": "crowdsec4", "chain": "block_ingress4", "handle": 19, "expr": [ { "match": { "op": "==", "left": { "payload": { "protocol": "ip", "field": "saddr" } }, "right": "@bad-actors4" } }, { "log": { "prefix": "DROP-IN-actor", "group": 0 } }, { "counter": { "packets": 47, "bytes": 2162 } }, { "drop": null } ] } }, { "rule": { "family": "inet", "table": "crowdsec4", "chain": "block_ingress4", "handle": 20, "expr": [ { "match": { "op": "==", "left": { "payload": { "protocol": "ip", "field": "saddr" } }, "right": "@blackhole4" } }, { "log": { "prefix": "DROP-IN-blackhole", "group": 0 } }, { "counter": { "packets": 0, "bytes": 0 } }, { "drop": null } ] } }, { "rule": { "family": "inet", "table": "crowdsec4", "chain": "block_ingress4", "handle": 21, "expr": [ { "match": { "op": "==", "left": { "payload": { "protocol": "ip", "field": "saddr" } }, "right": "@crowdsec-blacklists4" } }, { "log": { "prefix": "DROP-IN4", "group": 0 } }, { "counter": { "packets": 249, "bytes": 12882 } }, { "drop": null } ] } }, { "rule": { "family": "inet", "table": "crowdsec4", "chain": "block_ingress4", "handle": 22, "expr": [ { "return": null } ] } } ] }
Mitigation
I've currently disabled the prometheus collector in the config, which stops the error in the logs, however I'd still like to collect some metrics with it. I know you guys are busy over there so I'm not stressed about it.