search

crowdsecurity / crowdsec

CrowdSec - the open-source and participative security solution offering crowdsourced protection against malicious IPs and access to the most advanced real-world CTI.
https://crowdsec.net
MIT License
8.53k stars 430 forks source link

CrowdSec fails when monitored Caddy logfiles are rotated #2902

Closed pmhausen closed 5 months ago

pmhausen commented 5 months ago

What happened?

The CrowdSec service terminates occasionally when Caddy rotates its logfiles.

What did you expect to happen?

The CrowdSec service should continue to run.

How can we reproduce it (as minimally and precisely as possible)?

I use this Caddy plugin on OPNsense - soon to be integrated into the OPNsense main distribution: https://github.com/Monviech/os-caddy-plugin

This is the Caddyfile definition for the reverse proxy considered in this particular issue:

# Reverse Proxy Domain: "353e05b6-1f79-4e60-9d1b-d702ce7a660e"
gitea.hausen.com {
    log {
        output file /var/log/caddy/access/353e05b6-1f79-4e60-9d1b-d702ce7a660e.log {
            roll_keep_for 10d
        }
    }

    handle {
        reverse_proxy 192.168.2.51:3000 {
        }
    }

    abort
}

On Mar 13th around 4:00 am a logfile rotation took place:

-rwxr-x---  1 root  wheel   3723791 Mar 13 04:15 353e05b6-1f79-4e60-9d1b-d702ce7a660e-2024-03-13T03-15-51.642.log.gz*
-rwxr-x---  1 root  wheel   9614786 Mar 15 11:28 353e05b6-1f79-4e60-9d1b-d702ce7a660e.log*

Consequently CrowdSec shut down with an error about a missing logfile - followed by a restart by Monit, which I implemented as a first band aid:

time="2024-03-13T04:18:05+01:00" level=warning msg="tail for /var/log/caddy/access/353e05b6-1f79-4e60-9d1b-d702ce7a660e-2024-03-13T03-15-51.642.log is empty" tail=/var/log/caddy/access/353e05b6-1f79-4e60-9d1b-d702ce7a660e-2024-03-13T03-15-51.642.log type=file
time="2024-03-13T04:18:05+01:00" level=warning msg="tail for /var/log/caddy/access/353e05b6-1f79-4e60-9d1b-d702ce7a660e-2024-03-13T03-15-51.642.log is empty" tail=/var/log/caddy/access/353e05b6-1f79-4e60-9d1b-d702ce7a660e-2024-03-13T03-15-51.642.log type=file
time="2024-03-13T04:18:05+01:00" level=warning msg="file reader of /var/log/caddy/access/353e05b6-1f79-4e60-9d1b-d702ce7a660e-2024-03-13T03-15-51.642.log died : stat /var/log/caddy/access/353e05b6-1f79-4e60-9d1b-d702ce7a660e-2024-03-13T03-15-51.642.log: no such file or directory" tail=/var/log/caddy/access/353e05b6-1f79-4e60-9d1b-d702ce7a660e-2024-03-13T03-15-51.642.log type=file
time="2024-03-13T04:18:05+01:00" level=info msg="File datasource /var/log/lighttpd/latest.log stopping" tail=/var/log/lighttpd/latest.log type=file
time="2024-03-13T04:18:05+01:00" level=info msg="File datasource /var/log/caddy/access/7242359b-6e6f-43c2-909e-bc76da30596b.log stopping" tail=/var/log/caddy/access/7242359b-6e6f-43c2-909e-bc76da30596b.log type=file
time="2024-03-13T04:18:05+01:00" level=info msg="File datasource /var/log/caddy/access/f20aaa1e-99da-4934-b550-99bf3bd5e055.log stopping" tail=/var/log/caddy/access/f20aaa1e-99da-4934-b550-99bf3bd5e055.log type=file
time="2024-03-13T04:18:05+01:00" level=info msg="File datasource /var/log/filter/latest.log stopping" tail=/var/log/filter/latest.log type=file
time="2024-03-13T04:18:05+01:00" level=info msg="File datasource /var/log/caddy/access/1311de0d-20ce-46da-909c-bb2265cc138b.log stopping" tail=/var/log/caddy/access/1311de0d-20ce-46da-909c-bb2265cc138b.log type=file
time="2024-03-13T04:18:05+01:00" level=info msg="File datasource /var/log/audit/latest.log stopping" tail=/var/log/audit/latest.log type=file
time="2024-03-13T04:18:05+01:00" level=info msg="File datasource /var/log/caddy/access/c48dacc8-47a3-4e3e-b616-7b75b237f90c.log stopping" tail=/var/log/caddy/access/c48dacc8-47a3-4e3e-b616-7b75b237f90c.log type=file
time="2024-03-13T04:18:05+01:00" level=info msg="File datasource /var/log/caddy/access/4062d92e-0a79-4dfb-8ac1-a985f6cd794a.log stopping" tail=/var/log/caddy/access/4062d92e-0a79-4dfb-8ac1-a985f6cd794a.log type=file
time="2024-03-13T04:18:05+01:00" level=info msg="File datasource /var/log/caddy/access/112cf350-363f-4cf9-b705-befcc5ce2199.log stopping" tail=/var/log/caddy/access/112cf350-363f-4cf9-b705-befcc5ce2199.log type=file
time="2024-03-13T04:18:05+01:00" level=info msg="File datasource /var/log/caddy/access/146b4766-c6cc-4b2c-81de-64c19a394a89.log stopping" tail=/var/log/caddy/access/146b4766-c6cc-4b2c-81de-64c19a394a89.log type=file
time="2024-03-13T04:18:05+01:00" level=info msg="File datasource /var/log/caddy/access/896bd5d6-04c2-443b-9b1a-dcc13bcec81d.log stopping" tail=/var/log/caddy/access/896bd5d6-04c2-443b-9b1a-dcc13bcec81d.log type=file
time="2024-03-13T04:18:05+01:00" level=info msg="File datasource /var/log/caddy/access/353e05b6-1f79-4e60-9d1b-d702ce7a660e.log stopping" tail=/var/log/caddy/access/353e05b6-1f79-4e60-9d1b-d702ce7a660e.log type=file
time="2024-03-13T04:18:05+01:00" level=info msg="File datasource /var/log/caddy/access/b2a1852c-39be-46e7-b7bc-bee915e45108.log stopping" tail=/var/log/caddy/access/b2a1852c-39be-46e7-b7bc-bee915e45108.log type=file
time="2024-03-13T04:18:05+01:00" level=info msg="File datasource /var/log/caddy/access/7b8c1671-5011-4842-9f5d-b2c462bf673e.log stopping" tail=/var/log/caddy/access/7b8c1671-5011-4842-9f5d-b2c462bf673e.log type=file
time="2024-03-13T04:18:05+01:00" level=info msg="File datasource /var/log/caddy/access/658a38ef-f651-4f56-b81e-32017e6d9e4c.log stopping" tail=/var/log/caddy/access/658a38ef-f651-4f56-b81e-32017e6d9e4c.log type=file
time="2024-03-13T04:18:05+01:00" level=warning msg="Acquisition is finished, shutting down"
time="2024-03-13T04:18:05+01:00" level=fatal msg="starting acquisition error : file reader of /var/log/caddy/access/353e05b6-1f79-4e60-9d1b-d702ce7a660e-2024-03-13T03-15-51.642.log died : stat /var/log/caddy/access/353e05b6-1f79-4e60-9d1b-d702ce7a660e-2024-03-13T03-15-51.642.log: no such file or directory"
time="2024-03-13T04:20:06+01:00" level=info msg="Enabled feature flags: <none>"
time="2024-03-13T04:20:06+01:00" level=info msg="Crowdsec v1.6.0-freebsd-4b8e6cd7"
time="2024-03-13T04:20:06+01:00" level=info msg="Loading prometheus collectors"
time="2024-03-13T04:20:07+01:00" level=info msg="Loading CAPI manager"
time="2024-03-13T04:20:08+01:00" level=info msg="CAPI manager configured successfully"
time="2024-03-13T04:20:08+01:00" level=warning msg="Machine is not allowed to synchronize decisions, you can enable it with `cscli console enable console_management`"
time="2024-03-13T04:20:08+01:00" level=info msg="CrowdSec Local API listening on 127.0.0.1:8080"
time="2024-03-13T04:20:08+01:00" level=info msg="Start push to CrowdSec Central API (interval: 17s once, then 10s)"
time="2024-03-13T04:20:08+01:00" level=info msg="Start sending metrics to CrowdSec Central API (interval: 36m51s once, then 30m0s)"
time="2024-03-13T04:20:08+01:00" level=info msg="capi metrics: sending"
time="2024-03-13T04:20:08+01:00" level=info msg="Loading grok library /usr/local/etc/crowdsec/patterns"
time="2024-03-13T04:20:08+01:00" level=info msg="last CAPI pull is newer than 1h30, skip."
time="2024-03-13T04:20:08+01:00" level=info msg="Start pull from CrowdSec Central API (interval: 1h59m16s once, then 2h0m0s)"
time="2024-03-13T04:20:10+01:00" level=info msg="Loading enrich plugins"
time="2024-03-13T04:20:10+01:00" level=info msg="Successfully registered enricher 'GeoIpCity'"
time="2024-03-13T04:20:10+01:00" level=info msg="Successfully registered enricher 'GeoIpASN'"
time="2024-03-13T04:20:10+01:00" level=info msg="Successfully registered enricher 'IpToRange'"
time="2024-03-13T04:20:10+01:00" level=info msg="Successfully registered enricher 'reverse_dns'"
time="2024-03-13T04:20:10+01:00" level=info msg="Successfully registered enricher 'ParseDate'"
time="2024-03-13T04:20:10+01:00" level=info msg="Successfully registered enricher 'UnmarshalJSON'"
time="2024-03-13T04:20:10+01:00" level=info msg="Loading parsers from 9 files"
time="2024-03-13T04:20:10+01:00" level=info msg="Loaded 2 parser nodes" file=/usr/local/etc/crowdsec/parsers/s00-raw/syslog-logs.yaml stage=s00-raw
time="2024-03-13T04:20:10+01:00" level=info msg="Loaded 1 parser nodes" file=/usr/local/etc/crowdsec/parsers/s01-parse/caddy-logs.yaml stage=s01-parse
time="2024-03-13T04:20:10+01:00" level=info msg="Loaded 1 parser nodes" file=/usr/local/etc/crowdsec/parsers/s01-parse/opnsense-gui-logs.yaml stage=s01-parse
time="2024-03-13T04:20:10+01:00" level=info msg="Loaded 2 parser nodes" file=/usr/local/etc/crowdsec/parsers/s01-parse/pf-logs.yaml stage=s01-parse
time="2024-03-13T04:20:10+01:00" level=info msg="Loaded 1 parser nodes" file=/usr/local/etc/crowdsec/parsers/s01-parse/sshd-logs.yaml stage=s01-parse
time="2024-03-13T04:20:10+01:00" level=info msg="Loaded 1 parser nodes" file=/usr/local/etc/crowdsec/parsers/s02-enrich/dateparse-enrich.yaml stage=s02-enrich
time="2024-03-13T04:20:10+01:00" level=info msg="Loaded 1 parser nodes" file=/usr/local/etc/crowdsec/parsers/s02-enrich/geoip-enrich.yaml stage=s02-enrich
time="2024-03-13T04:20:10+01:00" level=info msg="Loaded 1 parser nodes" file=/usr/local/etc/crowdsec/parsers/s02-enrich/http-logs.yaml stage=s02-enrich
time="2024-03-13T04:20:10+01:00" level=info msg="Loaded 1 parser nodes" file=/usr/local/etc/crowdsec/parsers/s02-enrich/whitelists.yaml stage=s02-enrich
time="2024-03-13T04:20:10+01:00" level=info msg="Loaded 11 nodes from 3 stages"
time="2024-03-13T04:20:10+01:00" level=info msg="No postoverflow parsers to load"
time="2024-03-13T04:20:10+01:00" level=info msg="Loading 43 scenario files"
time="2024-03-13T04:20:10+01:00" level=info msg="Adding trigger bucket" cfg=blue-pine name=crowdsecurity/grafana-cve-2021-43798
time="2024-03-13T04:20:10+01:00" level=info msg="Adding trigger bucket" cfg=quiet-breeze name=crowdsecurity/http-cve-2021-42013
time="2024-03-13T04:20:10+01:00" level=info msg="Adding trigger bucket" cfg=white-dawn name=crowdsecurity/fortinet-cve-2018-13379
time="2024-03-13T04:20:10+01:00" level=info msg="Adding leaky bucket" cfg=young-moon name=crowdsecurity/http-xss-probbing
time="2024-03-13T04:20:10+01:00" level=info msg="Adding leaky bucket" cfg=proud-resonance name=crowdsecurity/ssh-bf
time="2024-03-13T04:20:10+01:00" level=info msg="Adding leaky bucket" cfg=quiet-paper name=crowdsecurity/ssh-bf_user-enum
time="2024-03-13T04:20:10+01:00" level=info msg="Adding leaky bucket" cfg=aged-wildflower name=crowdsecurity/http-bad-user-agent
time="2024-03-13T04:20:10+01:00" level=info msg="Adding leaky bucket" cfg=frosty-fog name=crowdsecurity/ssh-slow-bf
time="2024-03-13T04:20:10+01:00" level=info msg="Adding leaky bucket" cfg=delicate-frost name=crowdsecurity/ssh-slow-bf_user-enum
time="2024-03-13T04:20:10+01:00" level=info msg="Adding leaky bucket" cfg=red-voice name=firewallservices/pf-scan-multi_ports
time="2024-03-13T04:20:10+01:00" level=info msg="Adding trigger bucket" cfg=silent-glade name=crowdsecurity/http-open-proxy
time="2024-03-13T04:20:10+01:00" level=info msg="Adding trigger bucket" cfg=white-sun name=crowdsecurity/CVE-2023-22515
time="2024-03-13T04:20:10+01:00" level=info msg="Adding trigger bucket" cfg=patient-snowflake name=crowdsecurity/CVE-2022-44877
time="2024-03-13T04:20:10+01:00" level=info msg="Adding trigger bucket" cfg=old-field name=crowdsecurity/CVE-2023-22518
time="2024-03-13T04:20:10+01:00" level=info msg="Adding trigger bucket" cfg=dry-shadow name=crowdsecurity/jira_cve-2021-26086
time="2024-03-13T04:20:10+01:00" level=info msg="Adding leaky bucket" cfg=sparkling-frost name=crowdsecurity/http-backdoors-attempts
time="2024-03-13T04:20:10+01:00" level=info msg="Adding leaky bucket" cfg=little-butterfly name=crowdsecurity/CVE-2022-41697
time="2024-03-13T04:20:10+01:00" level=info msg="Adding trigger bucket" cfg=bitter-pine name=crowdsecurity/CVE-2022-42889
time="2024-03-13T04:20:10+01:00" level=info msg="Adding trigger bucket" cfg=snowy-sun name=crowdsecurity/f5-big-ip-cve-2020-5902
time="2024-03-13T04:20:10+01:00" level=info msg="Adding leaky bucket" cfg=ancient-thunder name=crowdsecurity/http-probing
time="2024-03-13T04:20:10+01:00" level=info msg="Adding leaky bucket" cfg=fragrant-thunder name=crowdsecurity/CVE-2022-46169-bf
time="2024-03-13T04:20:10+01:00" level=info msg="Adding trigger bucket" cfg=small-dust name=crowdsecurity/CVE-2022-46169-cmd
time="2024-03-13T04:20:10+01:00" level=info msg="Adding trigger bucket" cfg=solitary-dawn name=crowdsecurity/http-cve-2021-41773
time="2024-03-13T04:20:10+01:00" level=info msg="Adding trigger bucket" cfg=throbbing-fog name=crowdsecurity/CVE-2022-37042
time="2024-03-13T04:20:10+01:00" level=info msg="Adding trigger bucket" cfg=silent-river name=crowdsecurity/CVE-2022-26134
time="2024-03-13T04:20:10+01:00" level=info msg="Adding trigger bucket" cfg=weathered-breeze name=ltsich/http-w00tw00t
time="2024-03-13T04:20:10+01:00" level=info msg="Adding trigger bucket" cfg=small-snow name=crowdsecurity/vmware-vcenter-vmsa-2021-0027
time="2024-03-13T04:20:10+01:00" level=info msg="Adding trigger bucket" cfg=proud-fog name=crowdsecurity/CVE-2023-49103
time="2024-03-13T04:20:10+01:00" level=info msg="Adding leaky bucket" cfg=shy-darkness name=crowdsecurity/http-sensitive-files
time="2024-03-13T04:20:10+01:00" level=info msg="Adding trigger bucket" cfg=weathered-grass name=crowdsecurity/thinkphp-cve-2018-20062
time="2024-03-13T04:20:10+01:00" level=info msg="Adding trigger bucket" cfg=quiet-shadow name=crowdsecurity/CVE-2022-41082
time="2024-03-13T04:20:10+01:00" level=info msg="Adding trigger bucket" cfg=lingering-butterfly name=crowdsecurity/apache_log4j2_cve-2021-44228
time="2024-03-13T04:20:10+01:00" level=info msg="Adding trigger bucket" cfg=fragrant-sun name=crowdsecurity/CVE-2017-9841
time="2024-03-13T04:20:10+01:00" level=info msg="Adding trigger bucket" cfg=old-moon name=crowdsecurity/netgear_rce
time="2024-03-13T04:20:10+01:00" level=info msg="Adding trigger bucket" cfg=bold-waterfall name=crowdsecurity/pulse-secure-sslvpn-cve-2019-11510
time="2024-03-13T04:20:10+01:00" level=info msg="Adding trigger bucket" cfg=autumn-moon name=crowdsecurity/CVE-2019-18935
time="2024-03-13T04:20:10+01:00" level=info msg="Adding leaky bucket" cfg=small-smoke name=crowdsecurity/http-crawl-non_statics
time="2024-03-13T04:20:10+01:00" level=info msg="Adding trigger bucket" cfg=purple-leaf name=crowdsecurity/CVE-2022-35914
time="2024-03-13T04:20:10+01:00" level=info msg="Adding trigger bucket" cfg=crimson-mountain name=crowdsecurity/spring4shell_cve-2022-22965
time="2024-03-13T04:20:10+01:00" level=info msg="Adding leaky bucket" cfg=summer-sea name=crowdsecurity/http-admin-interface-probing
time="2024-03-13T04:20:10+01:00" level=info msg="Adding leaky bucket" cfg=rough-darkness name=crowdsecurity/http-path-traversal-probing
time="2024-03-13T04:20:10+01:00" level=info msg="Adding leaky bucket" cfg=purple-silence name=crowdsecurity/http-generic-bf
time="2024-03-13T04:20:10+01:00" level=info msg="Adding leaky bucket" cfg=nameless-pond name=LePresidente/http-generic-401-bf
time="2024-03-13T04:20:10+01:00" level=info msg="Adding leaky bucket" cfg=rough-dust name=LePresidente/http-generic-403-bf
time="2024-03-13T04:20:10+01:00" level=info msg="Adding leaky bucket" cfg=frosty-lake name=crowdsecurity/http-sqli-probbing-detection
time="2024-03-13T04:20:10+01:00" level=info msg="Adding trigger bucket" cfg=bitter-resonance name=crowdsecurity/vmware-cve-2022-22954
time="2024-03-13T04:20:10+01:00" level=info msg="Adding trigger bucket" cfg=little-snow name=crowdsecurity/fortinet-cve-2022-40684
time="2024-03-13T04:20:10+01:00" level=info msg="Adding leaky bucket" cfg=fragrant-moon name=crowdsecurity/opnsense-gui-bf
time="2024-03-13T04:20:10+01:00" level=info msg="Loaded 48 scenarios"
time="2024-03-13T04:20:10+01:00" level=info msg="loading acquisition file : /usr/local/etc/crowdsec/acquis.yaml"
time="2024-03-13T04:20:10+01:00" level=warning msg="No matching files for pattern /var/log/nginx/*.log" type=file
time="2024-03-13T04:20:10+01:00" level=warning msg="No matching files for pattern ./tests/nginx/nginx.log" type=file
time="2024-03-13T04:20:10+01:00" level=warning msg="No matching files for pattern /var/log/auth.log" type=file
time="2024-03-13T04:20:10+01:00" level=warning msg="No matching files for pattern /var/log/syslog" type=file
time="2024-03-13T04:20:10+01:00" level=warning msg="No matching files for pattern /var/log/httpd-access.log" type=file
time="2024-03-13T04:20:10+01:00" level=warning msg="No matching files for pattern /var/log/httpd-error.log" type=file
time="2024-03-13T04:20:10+01:00" level=info msg="loading acquisition file : /usr/local/etc/crowdsec/acquis.d/caddy.yaml"
time="2024-03-13T04:20:10+01:00" level=info msg="Force add watch on /var/log/caddy/access" type=file
time="2024-03-13T04:20:10+01:00" level=info msg="Adding file /var/log/caddy/access/112cf350-363f-4cf9-b705-befcc5ce2199.log to datasources" type=file
time="2024-03-13T04:20:10+01:00" level=info msg="Adding file /var/log/caddy/access/1311de0d-20ce-46da-909c-bb2265cc138b.log to datasources" type=file
time="2024-03-13T04:20:10+01:00" level=info msg="Adding file /var/log/caddy/access/146b4766-c6cc-4b2c-81de-64c19a394a89.log to datasources" type=file
time="2024-03-13T04:20:10+01:00" level=info msg="Adding file /var/log/caddy/access/353e05b6-1f79-4e60-9d1b-d702ce7a660e.log to datasources" type=file
time="2024-03-13T04:20:10+01:00" level=info msg="Adding file /var/log/caddy/access/4062d92e-0a79-4dfb-8ac1-a985f6cd794a.log to datasources" type=file
time="2024-03-13T04:20:10+01:00" level=info msg="Adding file /var/log/caddy/access/658a38ef-f651-4f56-b81e-32017e6d9e4c.log to datasources" type=file
time="2024-03-13T04:20:10+01:00" level=info msg="Adding file /var/log/caddy/access/7242359b-6e6f-43c2-909e-bc76da30596b.log to datasources" type=file
time="2024-03-13T04:20:10+01:00" level=info msg="Adding file /var/log/caddy/access/7b8c1671-5011-4842-9f5d-b2c462bf673e.log to datasources" type=file
time="2024-03-13T04:20:10+01:00" level=info msg="Adding file /var/log/caddy/access/896bd5d6-04c2-443b-9b1a-dcc13bcec81d.log to datasources" type=file
time="2024-03-13T04:20:10+01:00" level=info msg="Adding file /var/log/caddy/access/b2a1852c-39be-46e7-b7bc-bee915e45108.log to datasources" type=file
time="2024-03-13T04:20:10+01:00" level=info msg="Adding file /var/log/caddy/access/c48dacc8-47a3-4e3e-b616-7b75b237f90c.log to datasources" type=file
time="2024-03-13T04:20:10+01:00" level=info msg="Adding file /var/log/caddy/access/f20aaa1e-99da-4934-b550-99bf3bd5e055.log to datasources" type=file
time="2024-03-13T04:20:10+01:00" level=info msg="loading acquisition file : /usr/local/etc/crowdsec/acquis.d/opnsense.yaml"
time="2024-03-13T04:20:10+01:00" level=info msg="Force add watch on /var/log/audit" type=file
time="2024-03-13T04:20:10+01:00" level=info msg="Adding file /var/log/audit/latest.log to datasources" type=file
time="2024-03-13T04:20:10+01:00" level=info msg="Force add watch on /var/log/lighttpd" type=file
time="2024-03-13T04:20:10+01:00" level=info msg="Adding file /var/log/lighttpd/latest.log to datasources" type=file
time="2024-03-13T04:20:10+01:00" level=info msg="Force add watch on /var/log/filter" type=file
time="2024-03-13T04:20:10+01:00" level=info msg="Adding file /var/log/filter/latest.log to datasources" type=file
time="2024-03-13T04:20:10+01:00" level=info msg="Starting processing data"

Anything else we need to know?

No response

Crowdsec version

```console # cscli version 2024/03/15 11:39:45 version: v1.6.0-freebsd-4b8e6cd7 2024/03/15 11:39:45 Codename: alphaga 2024/03/15 11:39:45 BuildDate: 2024-02-20_01:09:28 2024/03/15 11:39:45 GoVersion: 1.21.7 2024/03/15 11:39:45 Platform: freebsd 2024/03/15 11:39:45 libre2: C++ 2024/03/15 11:39:45 Constraint_parser: >= 1.0, <= 3.0 2024/03/15 11:39:45 Constraint_scenario: >= 1.0, <= 3.0 2024/03/15 11:39:45 Constraint_api: v1 2024/03/15 11:39:45 Constraint_acquis: >= 1.0, < 2.0 ```

OS version

```console # opnsense-version OPNsense 24.1.3_1 # freebsd-version 13.2-RELEASE-p10 ```

Enabled collections and parsers

```console # cscli hub list -o raw name,status,version,description,type crowdsecurity/caddy-logs,enabled,0.7,Parse caddy logs,parsers crowdsecurity/dateparse-enrich,enabled,0.2,,parsers crowdsecurity/geoip-enrich,enabled,0.2,"Populate event with geoloc info : as, country, coords, source range.",parsers crowdsecurity/http-logs,enabled,1.2,"Parse more Specifically HTTP logs, such as HTTP Code, HTTP path, HTTP args and if its a static ressource",parsers crowdsecurity/opnsense-gui-logs,enabled,0.1,Parse OPNSense web auth logs,parsers crowdsecurity/sshd-logs,enabled,2.3,Parse openSSH logs,parsers crowdsecurity/syslog-logs,enabled,0.8,,parsers crowdsecurity/whitelists,"enabled,local",,,parsers firewallservices/pf-logs,enabled,0.5,Parse packet filter logs,parsers crowdsecurity/apache_log4j2_cve-2021-44228,enabled,0.5,Detect cve-2021-44228 exploitation attemps,scenarios crowdsecurity/CVE-2017-9841,enabled,0.2,Detect CVE-2017-9841 exploits,scenarios crowdsecurity/CVE-2019-18935,enabled,0.2,Detect Telerik CVE-2019-18935 exploitation attempts,scenarios crowdsecurity/CVE-2022-26134,enabled,0.2,Detect CVE-2022-26134 exploits,scenarios crowdsecurity/CVE-2022-35914,enabled,0.2,Detect CVE-2022-35914 exploits,scenarios crowdsecurity/CVE-2022-37042,enabled,0.2,Detect CVE-2022-37042 exploits,scenarios crowdsecurity/CVE-2022-40684,enabled,0.3,Detect cve-2022-40684 exploitation attempts,scenarios crowdsecurity/CVE-2022-41082,enabled,0.4,Detect CVE-2022-41082 exploits,scenarios crowdsecurity/CVE-2022-41697,enabled,0.2,Detect CVE-2022-41697 enumeration,scenarios crowdsecurity/CVE-2022-42889,enabled,0.3,Detect CVE-2022-42889 exploits (Text4Shell),scenarios crowdsecurity/CVE-2022-44877,enabled,0.3,Detect CVE-2022-44877 exploits,scenarios crowdsecurity/CVE-2022-46169,enabled,0.2,Detect CVE-2022-46169 brute forcing,scenarios crowdsecurity/CVE-2023-22515,enabled,0.1,Detect CVE-2023-22515 exploitation,scenarios crowdsecurity/CVE-2023-22518,enabled,0.2,Detect CVE-2023-22518 exploits,scenarios crowdsecurity/CVE-2023-49103,enabled,0.3,Detect owncloud CVE-2023-49103 exploitation attempts,scenarios crowdsecurity/f5-big-ip-cve-2020-5902,enabled,0.2,Detect cve-2020-5902 exploitation attemps,scenarios crowdsecurity/fortinet-cve-2018-13379,enabled,0.3,Detect cve-2018-13379 exploitation attemps,scenarios crowdsecurity/grafana-cve-2021-43798,enabled,0.2,Detect cve-2021-43798 exploitation attemps,scenarios crowdsecurity/http-admin-interface-probing,enabled,0.3,Detect generic HTTP admin interface probing,scenarios crowdsecurity/http-backdoors-attempts,enabled,0.5,Detect attempt to common backdoors,scenarios crowdsecurity/http-bad-user-agent,enabled,1.1,Detect usage of bad User Agent,scenarios crowdsecurity/http-crawl-non_statics,enabled,0.7,Detect aggressive crawl on non static resources,scenarios crowdsecurity/http-cve-2021-41773,enabled,0.2,cve-2021-41773,scenarios crowdsecurity/http-cve-2021-42013,enabled,0.2,cve-2021-42013,scenarios crowdsecurity/http-generic-bf,enabled,0.6,Detect generic http brute force,scenarios crowdsecurity/http-open-proxy,enabled,0.5,Detect scan for open proxy,scenarios crowdsecurity/http-path-traversal-probing,enabled,0.3,Detect path traversal attempt,scenarios crowdsecurity/http-probing,enabled,0.3,Detect site scanning/probing from a single ip,scenarios crowdsecurity/http-sensitive-files,enabled,0.3,"Detect attempt to access to sensitive files (.log, .db ..) or folders (.git)",scenarios crowdsecurity/http-sqli-probing,enabled,0.3,A scenario that detects SQL injection probing with minimal false positives,scenarios crowdsecurity/http-xss-probing,enabled,0.3,A scenario that detects XSS probing with minimal false positives,scenarios crowdsecurity/jira_cve-2021-26086,enabled,0.2,Detect Atlassian Jira CVE-2021-26086 exploitation attemps,scenarios crowdsecurity/netgear_rce,enabled,0.3,Detect Netgear RCE DGN1000/DGN220 exploitation attempts,scenarios crowdsecurity/opnsense-gui-bf,enabled,0.3,Detect bruteforce on opnsense web interface,scenarios crowdsecurity/pulse-secure-sslvpn-cve-2019-11510,enabled,0.3,Detect cve-2019-11510 exploitation attemps,scenarios crowdsecurity/spring4shell_cve-2022-22965,enabled,0.3,Detect cve-2022-22965 probing,scenarios crowdsecurity/ssh-bf,enabled,0.3,Detect ssh bruteforce,scenarios crowdsecurity/ssh-slow-bf,enabled,0.4,Detect slow ssh bruteforce,scenarios crowdsecurity/thinkphp-cve-2018-20062,enabled,0.5,Detect ThinkPHP CVE-2018-20062 exploitation attemps,scenarios crowdsecurity/vmware-cve-2022-22954,enabled,0.3,Detect Vmware CVE-2022-22954 exploitation attempts,scenarios crowdsecurity/vmware-vcenter-vmsa-2021-0027,enabled,0.2,Detect VMSA-2021-0027 exploitation attemps,scenarios firewallservices/pf-scan-multi_ports,enabled,0.4,ban IPs that are scanning us,scenarios ltsich/http-w00tw00t,enabled,0.2,detect w00tw00t,scenarios crowdsecurity/bf_base,enabled,0.1,,contexts crowdsecurity/firewall_base,enabled,0.2,,contexts crowdsecurity/http_base,enabled,0.2,,contexts crowdsecurity/base-http-scenarios,enabled,0.8,http common : scanners detection,collections crowdsecurity/caddy,enabled,0.1,caddy support : parser and generic http scenarios,collections crowdsecurity/freebsd,enabled,0.1,core freebsd support : syslog+geoip+ssh,collections crowdsecurity/http-cve,enabled,2.6,Detect CVE exploitation in http logs,collections crowdsecurity/opnsense,enabled,0.4,core opnsense support,collections crowdsecurity/opnsense-gui,enabled,0.1,OPNSense web authentication support,collections crowdsecurity/sshd,enabled,0.3,sshd support : parser and brute-force detection,collections firewallservices/pf,enabled,0.2,Parser and scenario for Packet Filter logs,collections ```

Acquisition config

```console # cat /usr/local/etc/crowdsec/acquis.d/* filenames: - /var/log/caddy/access/*.log force_inotify: true poll_without_inotify: true labels: type: caddy # # Before 22.1, OPNsense used circular logs under /var/log/*.log that # can still be around. They are old, in binary format and are not needed by crowdsec. # # For this reason we don't scan /var/log/*.log, but some plugins can write # their (plaintext) logs in that location, in such case add their pathnames too. # filenames: # DO NOT EDIT - to add new datasources (log locations), # create new files in /usr/local/etc/crowdsec/acquis.d/ # collection: crowdsecurity/sshd - /var/log/audit/latest.log # collection: crowdsecurity/opnsense-gui (web admin) - /var/log/lighttpd/latest.log # collection: firewallservices/pf - /var/log/filter/latest.log # When OPNsense is configured with /var/log in a RAM disk, # the log directories are created after crowdsec is run. # We force crowdsec to watch over directory creation as well # as file creation. FreeBSD has kqueue instead of inotify # but the option works with both. force_inotify: true # this option is required from crowdsec v1.5.0 to follow # changes in symlinks poll_without_inotify: true labels: type: syslog ```

Config show

```console # cscli config show Global: - Configuration Folder : /usr/local/etc/crowdsec - Data Folder : /var/db/crowdsec/data - Hub Folder : /usr/local/etc/crowdsec/hub - Simulation File : /usr/local/etc/crowdsec/simulation.yaml - Log Folder : /var/log/crowdsec - Log level : info - Log Media : file Crowdsec: - Acquisition File : /usr/local/etc/crowdsec/acquis.yaml - Parsers routines : 1 - Acquisition Folder : /usr/local/etc/crowdsec/acquis.d/ cscli: - Output : human - Hub Branch : API Client: - URL : http://127.0.0.1:8080/ - Login : localhost - Credentials File : /usr/local/etc/crowdsec/local_api_credentials.yaml Local API Server: - Listen URL : 127.0.0.1:8080 - Profile File : /usr/local/etc/crowdsec/profiles.yaml - Trusted IPs: - 127.0.0.1 - ::1 - Database: - Type : sqlite - Path : /var/db/crowdsec/data/crowdsec.db - Flush age : 7d - Flush size : 5000 ```

Prometheus metrics

```console $ cscli metrics # paste output here ```

Related custom configs versions (if applicable) : notification plugins, custom scenarios, parsers etc.

github-actions[bot] commented 5 months ago

@pmhausen: Thanks for opening an issue, it is currently awaiting triage.

In the meantime, you can:

  1. Check Crowdsec Documentation to see if your issue can be self resolved.
  2. You can also join our Discord.
  3. Check Releases to make sure your agent is on the latest version.
Details I am a bot created to help the [crowdsecurity](https://github.com/crowdsecurity) developers manage community feedback and contributions. You can check out my [manifest file](https://github.com/crowdsecurity/crowdsec/blob/master/.github/governance.yml) to understand my behavior and what I can do. If you want to use this for your project, you can check out the [BirthdayResearch/oss-governance-bot](https://github.com/BirthdayResearch/oss-governance-bot) repository.
rr404 commented 5 months ago

Same issue with lock and backupfiles on some services.

buixor commented 5 months ago

fixed in https://github.com/crowdsecurity/crowdsec/pull/2903

1.6.1 is coming very soon, stay tuned