crowdsec-firewall-bouncer.service v0.0.23-debian-pragmatic : hang netlink server when starting

[lelanta]

Hello, Servers are proxmox v7.2 with severals lxc containers as web and mail servers on debian v11. On containers crowdsec seems to run well. I also test your wordpress plugins as we mainly use wordpress and joomla.

Your solution seems to be really a big progress in battle against virtual criminals but...

We use only nftables for all servers with specifics tables, chains and rules and not the integrated iptables firewall from proxmox.

PROBLEM IS :

I installed crowdsec with crowdsec-firewall-bouncer on a proxmox host : no problems. I change configuration of the bouncer to work with our netfilter rules regarding your last release and instructions. Big problem : netlink of the server become down and we have only access to it with kvm and not ssh. As soon as i stop the bouncer from kvm, netlink of server become again on line. I attach to this couriel logs and configuration files for expertise. Now i purged the bouncer and reinstall it with default parameters : it's not hang and service run. In waiting, Best regards.

########################### I/ : crowdsec config file : ########################### cscli config show Global:

• Configuration Folder : /etc/crowdsec
• Data Folder : /var/lib/crowdsec/data
• Hub Folder : /etc/crowdsec/hub
• Simulation File : /etc/crowdsec/simulation.yaml
• Log Folder : /var/log/
• Log level : info
• Log Media : file Crowdsec:
  • Acquisition File : /etc/crowdsec/acquis.yaml
  • Parsers routines : 1 cscli:
  • Output : human
  • Hub Branch :
  • Hub Folder : /etc/crowdsec/hub Local API Server:
  • Listen URL : 127.0.0.1:8080
  • Profile File : /etc/crowdsec/profiles.yaml
  • Trusted IPs:
    • 127.0.0.1
    • ::1
  • Database:
    • Type : sqlite
    • Path : /var/lib/crowdsec/data/crowdsec.db
    • Flush age : 7d
    • Flush size : 5000

########################### II/ : bouncer config file : ########################### cat crowdsec-firewall-bouncer.yaml mode: nftables pid_dir: /var/run/ update_frequency: 10s daemonize: true log_mode: file log_dir: /var/log/ log_level: info log_compression: true log_max_size: 100 log_max_backups: 3 log_max_age: 30 api_url: http://127.0.0.1:8080/ api_key: 30eba9a169bdd4e854db8e50f3840162 insecure_skip_verify: false disable_ipv6: true deny_action: DROP deny_log: false supported_decisions_types:

• ban

  to change log prefix

  deny_log_prefix: "CS: "

  to change the blacklists name

  blacklists_ipv4: ip_banned blacklists_ipv6: crowdsec6-blacklists

  if present, insert rule in those chains

  iptables_chains:

• INPUT

  - FORWARD

  - DOCKER-USER

nftables

nftables: ipv4: enabled: true set-only: true table: blacklist chain: input ipv6: enabled: false set-only: false table: crowdsec6 chain: crowdsec6-chain

packet filter

pf:

an empty string disables the anchor

anchor_name: ""

########################### III/ : crowdsec bouncer log ########################### time="16-05-2022 17:14:59" level=info msg="backend type : nftables" time="16-05-2022 17:14:59" level=info msg="IPV6 is disabled" time="16-05-2022 17:14:59" level=info msg="nftables initiated" time="16-05-2022 17:14:59" level=info msg="Processing new and deleted decisions . . ." time="16-05-2022 17:14:59" level=info msg="416 decisions deleted" time="16-05-2022 17:14:59" level=error msg="unable to commit add decisions Receive: netlink receive: invalid argument"

NETLINK RECEIVE PROBLEM. - CONSOLE SERVER AND SSH CONNECTION OUT

time="16-05-2022 17:14:59" level=info msg="14313 decisions added" time="16-05-2022 17:15:39" level=error msg="auth-api: auth with api key failed return nil response, error: dial tcp 127.0.0.1:8080: i/o timeout" time="16-05-2022 17:15:39" level=error msg="Get ["http://127.0.0.1:8080/v1/decisions/stream?startup=false\"](http://127.0.0.1:8080/v1/decisions/stream?startup=false): dial tcp 127.0.0.1:8080: i/o timeout" time="16-05-2022 17:16:09" level=error msg="auth-api: auth with api key failed return nil response, error: dial tcp 127.0.0.1:8080: i/o timeout" time="16-05-2022 17:20:18" level=info msg="flushing 'ip_banned' set in 'blacklist' table" time="16-05-2022 17:20:18" level=info msg="Shutting down firewall-bouncer service"

########################### III/ : crowdsec.log ########################### time="16-05-2022 18:16:33" level=info msg="Crowdsec v1.3.4-debian-pragmatic-linux-ddfe95e45d98d1e7a6496d2499e2e44a023135be" time="16-05-2022 18:16:33" level=info msg="Loading prometheus collectors" time="16-05-2022 18:16:33" level=info msg="Loading CAPI pusher" time="16-05-2022 18:16:33" level=info msg="Loading grok library /etc/crowdsec/patterns" time="16-05-2022 18:16:33" level=info msg="Loading enrich plugins" time="16-05-2022 18:16:33" level=info msg="Successfully registered enricher 'GeoIpCity'" time="16-05-2022 18:16:33" level=info msg="Successfully registered enricher 'GeoIpASN'" time="16-05-2022 18:16:33" level=info msg="Successfully registered enricher 'IpToRange'" time="16-05-2022 18:16:33" level=info msg="Successfully registered enricher 'reverse_dns'" time="16-05-2022 18:16:33" level=info msg="Successfully registered enricher 'ParseDate'" time="16-05-2022 18:16:33" level=info msg="Loading parsers 6 stages" time="16-05-2022 18:16:33" level=info msg="Loaded 2 parser nodes" file=/etc/crowdsec/parsers/s00-raw/syslog-logs.yaml time="16-05-2022 18:16:33" level=info msg="Loaded 1 parser nodes" file=/etc/crowdsec/parsers/s01-parse/proxmox-logs.yaml time="16-05-2022 18:16:33" level=info msg="Loaded 1 parser nodes" file=/etc/crowdsec/parsers/s01-parse/sshd-logs.yaml time="16-05-2022 18:16:33" level=info msg="Loaded 1 parser nodes" file=/etc/crowdsec/parsers/s02-enrich/dateparse-enrich.yaml time="16-05-2022 18:16:33" level=info msg="Loaded 1 parser nodes" file=/etc/crowdsec/parsers/s02-enrich/geoip-enrich.yaml time="16-05-2022 18:16:33" level=info msg="Loaded 1 parser nodes" file=/etc/crowdsec/parsers/s02-enrich/whitelists.yaml time="16-05-2022 18:16:33" level=info msg="Loaded 7 nodes, 3 stages" time="16-05-2022 18:16:33" level=info msg="Loading postoverflow Parsers" time="16-05-2022 18:16:33" level=info msg="Loaded 0 nodes, 0 stages" time="16-05-2022 18:16:33" level=info msg="Loading 26 scenario files" time="16-05-2022 18:16:33" level=info msg="Adding trigger bucket" cfg=wispy-frog file=/etc/crowdsec/scenarios/vmware-vcenter-vmsa-2021-0027.yaml name=crowdsecurity/vmware-vcenter-vmsa-2021-0027 time="16-05-2022 18:16:33" level=info msg="Adding trigger bucket" cfg=quiet-violet file=/etc/crowdsec/scenarios/apache_log4j2_cve-2021-44228.yaml name=crowdsecurity/apache_log4j2_cve-2021-44228 time="16-05-2022 18:16:33" level=info msg="Adding leaky bucket" cfg=billowing-sound file=/etc/crowdsec/scenarios/http-backdoors-attempts.yaml name=crowdsecurity/http-backdoors-attempts time="16-05-2022 18:16:33" level=info msg="Adding trigger bucket" cfg=small-shape file=/etc/crowdsec/scenarios/jira_cve-2021-26086.yaml name=crowdsecurity/jira_cve-2021-26086 time="16-05-2022 18:16:33" level=info msg="Adding trigger bucket" cfg=red-rain file=/etc/crowdsec/scenarios/grafana-cve-2021-43798.yaml name=crowdsecurity/grafana-cve-2021-43798 time="16-05-2022 18:16:33" level=info msg="Adding leaky bucket" cfg=cold-haze file=/etc/crowdsec/scenarios/http-xss-probing.yaml name=crowdsecurity/http-xss-probbing time="16-05-2022 18:16:33" level=info msg="Adding trigger bucket" cfg=hidden-waterfall file=/etc/crowdsec/scenarios/fortinet-cve-2018-13379.yaml name=crowdsecurity/fortinet-cve-2018-13379 time="16-05-2022 18:16:33" level=info msg="Adding leaky bucket" cfg=young-night file=/etc/crowdsec/scenarios/http-generic-bf.yaml name=crowdsecurity/http-generic-bf time="16-05-2022 18:16:33" level=info msg="Adding leaky bucket" cfg=winter-wave file=/etc/crowdsec/scenarios/http-generic-bf.yaml name=LePresidente/http-generic-401-bf time="16-05-2022 18:16:33" level=info msg="Adding leaky bucket" cfg=rough-meadow file=/etc/crowdsec/scenarios/http-probing.yaml name=crowdsecurity/http-probing time="16-05-2022 18:16:33" level=info msg="Adding leaky bucket" cfg=dawn-meadow file=/etc/crowdsec/scenarios/ssh-slow-bf.yaml name=crowdsecurity/ssh-slow-bf time="16-05-2022 18:16:33" level=info msg="Adding leaky bucket" cfg=blue-sky file=/etc/crowdsec/scenarios/ssh-slow-bf.yaml name=crowdsecurity/ssh-slow-bf_user-enum time="16-05-2022 18:16:33" level=info msg="Adding leaky bucket" cfg=purple-morning file=/etc/crowdsec/scenarios/http-crawl-non_statics.yaml name=crowdsecurity/http-crawl-non_statics time="16-05-2022 18:16:33" level=info msg="Adding leaky bucket" cfg=shy-silence file=/etc/crowdsec/scenarios/http-path-traversal-probing.yaml name=crowdsecurity/http-path-traversal-probing time="16-05-2022 18:16:33" level=info msg="Adding trigger bucket" cfg=dawn-meadow file=/etc/crowdsec/scenarios/pulse-secure-sslvpn-cve-2019-11510.yaml name=crowdsecurity/pulse-secure-sslvpn-cve-2019-11510 time="16-05-2022 18:16:33" level=info msg="Adding leaky bucket" cfg=floral-wood file=/etc/crowdsec/scenarios/proxmox-bf.yaml name=fulljackz/proxmox-bf time="16-05-2022 18:16:33" level=info msg="Adding leaky bucket" cfg=ancient-flower file=/etc/crowdsec/scenarios/proxmox-bf.yaml name=fulljackz/proxmox-bf-user-enum time="16-05-2022 18:16:33" level=info msg="Adding trigger bucket" cfg=red-rain file=/etc/crowdsec/scenarios/f5-big-ip-cve-2020-5902.yaml name=crowdsecurity/f5-big-ip-cve-2020-5902 time="16-05-2022 18:16:33" level=info msg="Adding trigger bucket" cfg=morning-fire file=/etc/crowdsec/scenarios/http-cve-2021-42013.yaml name=crowdsecurity/http-cve-2021-42013 time="16-05-2022 18:16:33" level=info msg="Adding trigger bucket" cfg=white-dawn file=/etc/crowdsec/scenarios/vmware-cve-2022-22954.yaml name=crowdsecurity/vmware-cve-2022-22954 time="16-05-2022 18:16:33" level=info msg="Adding leaky bucket" cfg=crimson-feather file=/etc/crowdsec/scenarios/http-sqli-probing.yaml name=crowdsecurity/http-sqli-probbing-detection time="16-05-2022 18:16:33" level=info msg="Adding leaky bucket" cfg=still-field file=/etc/crowdsec/scenarios/http-sensitive-files.yaml name=crowdsecurity/http-sensitive-files time="16-05-2022 18:16:33" level=info msg="Adding leaky bucket" cfg=small-silence file=/etc/crowdsec/scenarios/http-bad-user-agent.yaml name=crowdsecurity/http-bad-user-agent time="16-05-2022 18:16:33" level=info msg="Adding trigger bucket" cfg=floral-voice file=/etc/crowdsec/scenarios/thinkphp-cve-2018-20062.yaml name=crowdsecurity/thinkphp-cve-2018-20062 time="16-05-2022 18:16:33" level=info msg="Adding trigger bucket" cfg=silent-rain file=/etc/crowdsec/scenarios/http-open-proxy.yaml name=crowdsecurity/http-open-proxy time="16-05-2022 18:16:33" level=info msg="Adding trigger bucket" cfg=late-cherry file=/etc/crowdsec/scenarios/spring4shell_cve-2022-22965.yaml name=crowdsecurity/spring4shell_cve-2022-22965 time="16-05-2022 18:16:33" level=info msg="Adding trigger bucket" cfg=delicate-snow file=/etc/crowdsec/scenarios/http-cve-2021-41773.yaml name=crowdsecurity/http-cve-2021-41773 time="16-05-2022 18:16:33" level=info msg="Adding leaky bucket" cfg=aged-violet file=/etc/crowdsec/scenarios/ssh-bf.yaml name=crowdsecurity/ssh-bf time="16-05-2022 18:16:33" level=info msg="Adding leaky bucket" cfg=still-waterfall file=/etc/crowdsec/scenarios/ssh-bf.yaml name=crowdsecurity/ssh-bf_user-enum time="16-05-2022 18:16:33" level=info msg="Adding trigger bucket" cfg=divine-shape file=/etc/crowdsec/scenarios/http-w00tw00t.yaml name=ltsich/http-w00tw00t time="16-05-2022 18:16:33" level=warning msg="Loaded 30 scenarios" time="16-05-2022 18:16:33" level=info msg="loading acquisition file : /etc/crowdsec/acquis.yaml" time="16-05-2022 18:16:33" level=warning msg="No matching files for pattern /var/log/nginx/.log" type=file time="16-05-2022 18:16:33" level=warning msg="No matching files for pattern ./tests/nginx/nginx.log" type=file time="16-05-2022 18:16:33" level=info msg="Adding file /var/log/auth.log to datasources" type=file time="16-05-2022 18:16:33" level=info msg="Adding file /var/log/syslog to datasources" type=file time="16-05-2022 18:16:33" level=warning msg="No matching files for pattern /var/log/apache2/.log" type=file time="16-05-2022 18:16:33" level=info msg="Adding file /var/log/syslog to datasources" type=file time="16-05-2022 18:16:33" level=info msg="Adding file /var/log/kern.log to datasources" type=file time="16-05-2022 18:16:33" level=info msg="Adding file /var/log/messages to datasources" type=file time="16-05-2022 18:16:33" level=info msg="test done"

END of LOGS

[buixor]

Hello,

Sorry for the delay. To be sure to understand your issue :

• you had a previous version of the bouncer, but after changing its configuration, it was blocking traffic ?
• you then purged and reinstall bouncer and it worked again ?

Please :

• Join your configuration files and log files using markdown code blocks otherwise it's impossible to read because of formatting issues.
• Can you tell me more about how it the firewall configuration without the bouncer ? It might be a rule priority issue

[lelanta]

Hello, For firewall that cannot be a rule priority issue. nftables manage with several tables and chain the host and all VMs. host with filter table chain input and VMs with filter table chain forward.

The advantage of this configuration is your work only with a centralized firewall on host for all. severals tables and chains have specifics rules (ipv6 disabled on all hosts) tables: filter, table ip filter, ip blacklist, ip ip_france, ip fail2ban, ip crowd, etc... chains: host.rules, vms.rules, geo-block.rules, blacklisted.rules, etc...

This structure is made to not flush or fully delete tables and rules with a nft flush ruleset but only specific table and rules when restart a specific service like fail2ban or crowdsec.

We use proxmox since v3 on all servers since 9 years with previously iptables firewall. since proxmox v6 we use nftables which is a real advance in firewall managment and let us have only half of lines for a better service with advantage of netdev use.

To solve problem, i reinstall only bouncer with standart parameters and all is ok. problem come if you change configuration and particulary for parameter set-only defined to true. I tried to only change name of table and chain and it's ok

for me bouncer entered in a infinite loop ?? because possibility to unloading it with KVM.

I will do some other tests on a local testing proxmox host ASAP.

Now i saw that jarppiko tried to make some modifications on bouncer concerning the use of non standart configuration. i think his modifications have not been merged on last release and is for me incomplete to use it on a proxmox or similar system.

I began to fork bouncer code to analyse structure and code. I think it's necessary to fully implement netfilter api inside to have fully possible configuration. possibility to use severals different type, hooks, priority, policy, etc... at same time to control host and VMs As soon as i'll have finish to analyse and well understand philosophy of code for crowdsec and bouncer. i'll submit to the team a proposal of modifications and if OK, i'll begin to code a new bouncer as soon as finished to explore capabilities of golang. (i developped netware NLMs systems in C during 10 years.)

On a other side , what is this error ? level=error msg="unable to commit add decisions Receive: netlink receive: invalid argument" i think that is when a local decision occur and is not apply because only CAPIs decisions are applied.

To conclude, as i told in last msg, philosophy of crowd is well thinked to share security updates in real-time and as it's a french team ... like me.

Best Regards