search

crowdsecurity / crowdsec

CrowdSec - the open-source and participative security solution offering crowdsourced protection against malicious IPs and access to the most advanced real-world CTI.
https://crowdsec.net
MIT License
8.64k stars 446 forks source link

journal uses an unsupported feature #2217

Closed stephdl closed 1 year ago

stephdl commented 1 year ago

What happened?

We use crowdsec with containers on debian 11 and rocky linux 9.2 with journald acquisition

with debian I have no issue, I see the log parsed when I do ||cscli metrics || https://gist.github.com/stephdl/c4ecb7b9830208a0b66a227e67ea402a

with rocky Linux I noticed that the logs are not parsed when I do cscli metrics (we got the same configuration except the path to journald that is not the same between debian and rockyLinux) https://gist.github.com/stephdl/56568e5e41131055a93481702364aa0f

When I start crowdsec on rockyLinux 9.2 I found this error relevant to logs : https://gist.github.com/stephdl/d322f77ef258d3e23d98afd7b2f556b1

journalctl inside the container cannot read the journal output we have an error : journal uses an unsupported feature. ignoring file.

So I went to google and I found a bug report from redhat relevant to this error, or probably close journalctl "Journal file uses an unsupported feature" https://bugzilla.redhat.com/show_bug.cgi?id=1413388

in short we do not run the same version of systemd on rockyLinux it is higher and we do not probably use the same encryption for what I understand

I use crowdsec 1.5.1 1b8b1d84bcfa docker.io/crowdsecurity/crowdsec:v1.5.1-debian 7 minutes ago Up 7 minutes crowdsec1

on rocky linux 9.2

[root@R2-pve ~]# journalctl --version
systemd 252 (252-13.el9_2)
+PAM +AUDIT +SELINUX -APPARMOR +IMA +SMACK +SECCOMP +GCRYPT +GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS -FIDO2 +IDN2 -IDN -IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 -PWQUALITY +P11KIT -QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD -BPF_FRAMEWORK +XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified

inside the container

[root@R2-pve ~]# podman exec -ti crowdsec1 bash
root@R2-pve:/# journalctl --version
systemd 247 (247.3-7+deb11u2)
+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +ZSTD +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=unified

on debian 11

root@D1:~# journalctl --version
systemd 247 (247.3-7+deb11u2)
+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +ZSTD +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=unified

inside the container 

root@D1:~# podman exec -ti crowdsec1 bash 
root@D1:/# journalctl --version
systemd 247 (247.3-7+deb11u2)
+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +ZSTD +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=unified

For what I understand we should have the same encryption and version on the system inside the host and inside the container

What did you expect to happen?

I expect that we could start acquisition

How can we reproduce it (as minimally and precisely as possible)?

user rocky-linux 9.2 or any system with systemd version 252

Anything else we need to know?

No response

Crowdsec version

```console $ cscli version [root@R2-pve ~]# cscli version 2023/05/22 15:05:34 version: v1.5.1-eddb994c0b48d77b34a3f22b719dc5716670d2ae 2023/05/22 15:05:34 Codename: alphaga 2023/05/22 15:05:34 BuildDate: 2023-05-17_10:59:02 2023/05/22 15:05:34 GoVersion: 1.20.4 2023/05/22 15:05:34 Platform: docker 2023/05/22 15:05:34 Constraint_parser: >= 1.0, <= 2.0 2023/05/22 15:05:34 Constraint_scenario: >= 1.0, < 3.0 2023/05/22 15:05:34 Constraint_api: v1 2023/05/22 15:05:34 Constraint_acquis: >= 1.0, < 2.0 ```

OS version

```console [root@R1-pve ~]# cat /etc/redhat-release Rocky Linux release 9.2 (Blue Onyx) ```

Enabled collections and parsers

```console [root@R2-pve ~]# cscli hub list -o raw crowdsecurity/apache2,enabled,0.1,apache2 support : parser and generic http scenarios ,collections crowdsecurity/base-http-scenarios,enabled,0.6,http common : scanners detection,collections crowdsecurity/dovecot,enabled,0.1,dovecot support : parser and spammer detection,collections crowdsecurity/http-cve,enabled,2.0,,collections crowdsecurity/linux,enabled,0.2,core linux support : syslog+geoip+ssh,collections crowdsecurity/mariadb,enabled,0.1,mariadb support : logs and brute-force scenarios,collections crowdsecurity/nextcloud,enabled,0.3,Nextcloud support : parser and brute-force detection,collections crowdsecurity/nginx,enabled,0.2,nginx support : parser and generic http scenarios,collections crowdsecurity/nginx-proxy-manager,enabled,0.1,Nginx Proxy Manager support : parser and generic http scenarios,collections crowdsecurity/pgsql,enabled,0.1,postgres support : logs and brute-force scenarios,collections crowdsecurity/postfix,enabled,0.2,postfix support : parser and spammer detection,collections crowdsecurity/proftpd,enabled,0.1,proftpd support : parser and brute-force/user enumeration detection,collections crowdsecurity/sshd,enabled,0.2,sshd support : parser and brute-force detection,collections crowdsecurity/traefik,enabled,0.1,traefik support: parser and generic http scenarios,collections crowdsecurity/vsftpd,enabled,0.1,VSFTPD support : logs and brute-force scenarios,collections crowdsecurity/whitelist-good-actors,enabled,0.1,Good actors whitelists,collections crowdsecurity/wordpress,enabled,0.4,wordpress: Bruteforce protection and config probing,collections crowdsecurity/apache2-logs,enabled,1.3,Parse Apache2 access and error logs,parsers crowdsecurity/cri-logs,enabled,0.1,CRI logging format parser,parsers crowdsecurity/dateparse-enrich,enabled,0.2,,parsers crowdsecurity/docker-logs,enabled,0.1,docker json logs parser,parsers crowdsecurity/dovecot-logs,enabled,0.7,Parse dovecot logs,parsers crowdsecurity/geoip-enrich,enabled,0.2,"Populate event with geoloc info : as, country, coords, source range.",parsers crowdsecurity/http-logs,enabled,1.1,"Parse more Specifically HTTP logs, such as HTTP Code, HTTP path, HTTP args and if its a static ressource",parsers crowdsecurity/mariadb-logs,enabled,0.4,Parse MariaDB logs,parsers crowdsecurity/nextcloud-logs,enabled,0.2,Parse nextcloud logs,parsers crowdsecurity/nextcloud-whitelist,enabled,0.7,Whitelist events from nextcloud,parsers crowdsecurity/nginx-logs,enabled,1.3,Parse nginx access and error logs,parsers crowdsecurity/nginx-proxy-manager-logs,enabled,0.2,Parse Nginx Proxy Manager access and error logs,parsers crowdsecurity/pgsql-logs,enabled,0.7,Parse PgSQL logs,parsers crowdsecurity/postfix-logs,enabled,0.4,Parse postfix logs,parsers crowdsecurity/postscreen-logs,enabled,0.2,Parse postscreen logs,parsers crowdsecurity/proftpd-logs,enabled,0.2,Parse proftpd logs,parsers crowdsecurity/sshd-logs,enabled,2.0,Parse openSSH logs,parsers crowdsecurity/syslog-logs,enabled,0.8,,parsers crowdsecurity/traefik-logs,enabled,0.6,Parse Traefik access logs,parsers crowdsecurity/vsftpd-logs,enabled,0.3,Parse VSFTPD logs,parsers crowdsecurity/whitelists,enabled,0.2,Whitelist events from private ipv4 addresses,parsers crowdsecurity/CVE-2019-18935,enabled,0.1,Detect Telerik CVE-2019-18935 exploitation attempts,scenarios crowdsecurity/CVE-2022-26134,enabled,0.1,Detect CVE-2022-26134 exploits,scenarios crowdsecurity/CVE-2022-35914,enabled,0.1,Detect CVE-2022-35914 exploits,scenarios crowdsecurity/CVE-2022-37042,enabled,0.1,Detect CVE-2022-37042 exploits,scenarios crowdsecurity/CVE-2022-40684,enabled,0.2,Detect cve-2022-40684 exploitation attempts,scenarios crowdsecurity/CVE-2022-41082,enabled,0.3,Detect CVE-2022-41082 exploits,scenarios crowdsecurity/CVE-2022-41697,enabled,0.1,Detect CVE-2022-41697 enumeration,scenarios crowdsecurity/CVE-2022-42889,enabled,0.2,Detect CVE-2022-42889 exploits (Text4Shell),scenarios crowdsecurity/CVE-2022-44877,enabled,0.2,Detect CVE-2022-44877 exploits,scenarios crowdsecurity/CVE-2022-46169,enabled,0.1,Detect CVE-2022-46169 brute forcing,scenarios crowdsecurity/apache_log4j2_cve-2021-44228,enabled,0.4,Detect cve-2021-44228 exploitation attemps,scenarios crowdsecurity/dovecot-spam,enabled,0.3,detect errors on dovecot,scenarios crowdsecurity/f5-big-ip-cve-2020-5902,enabled,0.1,Detect cve-2020-5902 exploitation attemps,scenarios crowdsecurity/fortinet-cve-2018-13379,enabled,0.2,Detect cve-2018-13379 exploitation attemps,scenarios crowdsecurity/grafana-cve-2021-43798,enabled,0.1,Detect cve-2021-43798 exploitation attemps,scenarios crowdsecurity/http-backdoors-attempts,enabled,0.3,Detect attempt to common backdoors,scenarios crowdsecurity/http-bad-user-agent,enabled,0.7,Detect bad user-agents,scenarios crowdsecurity/http-bf-wordpress_bf,enabled,0.4,detect wordpress bruteforce,scenarios crowdsecurity/http-crawl-non_statics,enabled,0.3,Detect aggressive crawl from single ip,scenarios crowdsecurity/http-cve-2021-41773,enabled,0.1,cve-2021-41773,scenarios crowdsecurity/http-cve-2021-42013,enabled,0.1,cve-2021-42013,scenarios crowdsecurity/http-generic-bf,enabled,0.4,Detect generic http brute force,scenarios crowdsecurity/http-open-proxy,enabled,0.3,Detect scan for open proxy,scenarios crowdsecurity/http-path-traversal-probing,enabled,0.2,Detect path traversal attempt,scenarios crowdsecurity/http-probing,enabled,0.2,Detect site scanning/probing from a single ip,scenarios crowdsecurity/http-sensitive-files,enabled,0.2,"Detect attempt to access to sensitive files (.log, .db ..) or folders (.git)",scenarios crowdsecurity/http-sqli-probing,enabled,0.2,A scenario that detects SQL injection probing with minimal false positives,scenarios crowdsecurity/http-wordpress_user-enum,enabled,0.1,detect wordpress probing : authors enumeration,scenarios crowdsecurity/http-wordpress_wpconfig,enabled,0.1,detect wordpress probing : variations around wp-config.php by wpscan,scenarios crowdsecurity/http-xss-probing,enabled,0.2,A scenario that detects XSS probing with minimal false positives,scenarios crowdsecurity/jira_cve-2021-26086,enabled,0.1,Detect Atlassian Jira CVE-2021-26086 exploitation attemps,scenarios crowdsecurity/mariadb-bf,enabled,0.1,Detect mariadb bruteforce,scenarios crowdsecurity/nextcloud-bf,enabled,0.2,Detect Nextcloud bruteforce,scenarios crowdsecurity/nginx-req-limit-exceeded,enabled,0.1,Detects IPs which violate nginx's user set request limit.,scenarios crowdsecurity/pgsql-bf,enabled,0.1,Detect PgSQL bruteforce,scenarios crowdsecurity/postfix-spam,enabled,0.2,Detect spammers,scenarios crowdsecurity/proftpd-bf,enabled,0.1,Detect proftpd bruteforce,scenarios crowdsecurity/proftpd-bf_user-enum,enabled,0.1,Detect proftpd user enum bruteforce,scenarios crowdsecurity/pulse-secure-sslvpn-cve-2019-11510,enabled,0.2,Detect cve-2019-11510 exploitation attemps,scenarios crowdsecurity/spring4shell_cve-2022-22965,enabled,0.2,Detect cve-2022-22965 probing,scenarios crowdsecurity/ssh-bf,enabled,0.1,Detect ssh bruteforce,scenarios crowdsecurity/ssh-slow-bf,enabled,0.2,Detect slow ssh bruteforce,scenarios crowdsecurity/thinkphp-cve-2018-20062,enabled,0.3,Detect ThinkPHP CVE-2018-20062 exploitation attemps,scenarios crowdsecurity/vmware-cve-2022-22954,enabled,0.2,Detect Vmware CVE-2022-22954 exploitation attempts,scenarios crowdsecurity/vmware-vcenter-vmsa-2021-0027,enabled,0.1,Detect VMSA-2021-0027 exploitation attemps,scenarios crowdsecurity/vsftpd-bf,enabled,0.1,Detect FTP bruteforce (vsftpd),scenarios ltsich/http-w00tw00t,enabled,0.1,detect w00tw00t,scenarios crowdsecurity/cdn-whitelist,enabled,0.4,Whitelist CDN providers,postoverflows crowdsecurity/rdns,enabled,0.2,Lookup the DNS associated to the source IP only for overflows,postoverflows crowdsecurity/seo-bots-whitelist,enabled,0.4,Whitelist good search engine crawlers,postoverflows ```

Acquisition config

```console root@R2-pve:/# cat /etc/crowdsec/acquis.yaml /etc/crowdsec/acquis.d/* --- source: journalctl journalctl_filter: - "_SYSTEMD_UNIT=sshd.service" labels: type: syslog --- source: journalctl journalctl_filter: - "SYSLOG_IDENTIFIER=traefik1" labels: type: syslog --- source: journalctl journalctl_filter: - "SYSLOG_IDENTIFIER=ldapproxy1" labels: type: syslog --- source: journalctl journalctl_filter: - "SYSLOG_IDENTIFIER=loki1" labels: type: syslog --- source: journalctl journalctl_filter: - "SYSLOG_IDENTIFIER=crowdsec1" labels: type: syslog --- source: journalctl journalctl_filter: - "SYSLOG_IDENTIFIER=promtail1" labels: type: syslog --- cat: '/etc/crowdsec/acquis.d/*': No such file or directory

Config show

```console root@R2-pve:/# cscli config show Global: - Configuration Folder : /etc/crowdsec - Configuration Folder : /etc/crowdsec - Data Folder : /var/lib/crowdsec/data - Hub Folder : /etc/crowdsec/hub - Simulation File : /etc/crowdsec/simulation.yaml - Log Folder : /var/log/ - Log level : info - Log Media : stdout Crowdsec: - Acquisition File : /etc/crowdsec/acquis.yaml - Parsers routines : 1 cscli: - Output : human - Hub Branch : - Hub Folder : /etc/crowdsec/hub API Client: - URL : http://0.0.0.0:8080/ - Login : localhost - Credentials File : /etc/crowdsec/local_api_credentials.yaml Local API Server: - Listen URL : 0.0.0.0:8080 - Profile File : /etc/crowdsec/profiles.yaml - Trusted IPs: - 127.0.0.1 - ::1 - Database: - Type : sqlite - Path : /var/lib/crowdsec/data/crowdsec.db - Flush age : 7d - Flush size : 5000 ```

Prometheus metrics

```console root@R2-pve:/# cscli metrics Local Api Metrics: ╭──────────────────────┬────────┬──────╮ │ Route │ Method │ Hits │ ├──────────────────────┼────────┼──────┤ │ /v1/decisions/stream │ GET │ 94 │ │ /v1/heartbeat │ GET │ 15 │ │ /v1/watchers/login │ POST │ 1 │ ╰──────────────────────┴────────┴──────╯ Local Api Machines Metrics: ╭───────────┬───────────────┬────────┬──────╮ │ Machine │ Route │ Method │ Hits │ ├───────────┼───────────────┼────────┼──────┤ │ localhost │ /v1/heartbeat │ GET │ 15 │ ╰───────────┴───────────────┴────────┴──────╯ Local Api Bouncers Metrics: ╭───────────┬──────────────────────┬────────┬──────╮ │ Bouncer │ Route │ Method │ Hits │ ├───────────┼──────────────────────┼────────┼──────┤ │ localhost │ /v1/decisions/stream │ GET │ 94 │ ╰───────────┴──────────────────────┴────────┴──────╯ Local Api Decisions: ╭────────────────────────────────────────────┬────────┬────────┬───────╮ │ Reason │ Origin │ Action │ Count │ ├────────────────────────────────────────────┼────────┼────────┼───────┤ │ crowdsecurity/CVE-2022-26134 │ CAPI │ ban │ 121 │ │ crowdsecurity/CVE-2022-42889 │ CAPI │ ban │ 4 │ │ crowdsecurity/http-bad-user-agent │ CAPI │ ban │ 3371 │ │ crowdsecurity/CVE-2022-41082 │ CAPI │ ban │ 600 │ │ crowdsecurity/apache_log4j2_cve-2021-44228 │ CAPI │ ban │ 268 │ │ crowdsecurity/http-path-traversal-probing │ CAPI │ ban │ 54 │ │ crowdsecurity/mariadb-bf │ CAPI │ ban │ 7 │ │ crowdsecurity/CVE-2019-18935 │ CAPI │ ban │ 12 │ │ crowdsecurity/fortinet-cve-2018-13379 │ CAPI │ ban │ 81 │ │ crowdsecurity/http-backdoors-attempts │ CAPI │ ban │ 178 │ │ crowdsecurity/http-generic-bf │ CAPI │ ban │ 11 │ │ crowdsecurity/grafana-cve-2021-43798 │ CAPI │ ban │ 12 │ │ crowdsecurity/http-crawl-non_statics │ CAPI │ ban │ 261 │ │ crowdsecurity/http-cve-2021-41773 │ CAPI │ ban │ 3 │ │ crowdsecurity/http-open-proxy │ CAPI │ ban │ 202 │ │ crowdsecurity/CVE-2022-35914 │ CAPI │ ban │ 37 │ │ crowdsecurity/CVE-2022-37042 │ CAPI │ ban │ 11 │ │ crowdsecurity/dovecot-spam │ CAPI │ ban │ 1234 │ │ crowdsecurity/f5-big-ip-cve-2020-5902 │ CAPI │ ban │ 17 │ │ crowdsecurity/thinkphp-cve-2018-20062 │ CAPI │ ban │ 8 │ │ crowdsecurity/http-probing │ CAPI │ ban │ 1351 │ │ crowdsecurity/http-sensitive-files │ CAPI │ ban │ 2 │ │ crowdsecurity/ssh-bf │ CAPI │ ban │ 7114 │ │ crowdsecurity/ssh-slow-bf │ CAPI │ ban │ 4 │ ╰────────────────────────────────────────────┴────────┴────────┴───────╯ ```

Related custom configs versions (if applicable) : notification plugins, custom scenarios, parsers etc.

root@R2-pve:/# cat /etc/crowdsec/profiles.yaml.local name: default_ip_remediation #debug: true filters: - Alert.Remediation == true && Alert.GetScope() == "Ip" decisions: - type: ban duration: 1m # we math a (number of ban + 1) * 4 (units of `duration_expr` comes from `duration`) duration_expr: Sprintf('%dm', (GetDecisionsCount(Alert.GetValue()) + 1) * 4) on_success: break
github-actions[bot] commented 1 year ago

@stephdl: Thanks for opening an issue, it is currently awaiting triage.

In the meantime, you can:

  1. Check Crowdsec Documentation to see if your issue can be self resolved.
  2. You can also join our Discord.
  3. Check Releases to make sure your agent is on the latest version.
Details I am a bot created to help the [crowdsecurity](https://github.com/crowdsecurity) developers manage community feedback and contributions. You can check out my [manifest file](https://github.com/crowdsecurity/crowdsec/blob/master/.github/governance.yml) to understand my behavior and what I can do. If you want to use this for your project, you can check out the [BirthdayResearch/oss-governance-bot](https://github.com/BirthdayResearch/oss-governance-bot) repository.
buixor commented 1 year ago

Hello,

The issue is that systemd 247 and systemd 252 are not compatible. Our current images are using debian-stable, but systemd 252 is not available there. We will provide as well debian-testing images that support systemd 252.

We'll keep you posted,

cf. https://github.com/crowdsecurity/home-assistant-addons/issues/38

stephdl commented 1 year ago

hey @buixor thank for it, cannot wait to test it

stephdl commented 1 year ago

@he2ss how to test your new container ?

Altycoder commented 1 year ago

@buixor I don't want to hijack this issue, but I'm using Arch as a host and crowdsec isn't parsing my system.journal file at all, it doesn't even appear in the metrics.

Arch currently uses systemd 253 (253.5-2-arch) and I'm aware that it's more bleeding edge than debian etc so just wondering if it would be possible to have a non-debian / rolling distro based image to pull that's more up to date e.g. Alpine?

Debian is very conservative and IIRC won't receive new systemd versions over each 2-3 year stable release cycle? That would mean that libsystemd would become seriously out of date in the crowdsec image over time.