Crowdsec Fails to Acquire /var/log/filter/latest.log on OPNsense Reboot with RAM Disk

yye347 commented 1 year ago

What happened?

When OPNsense is configured to use a RAM disk for /var/log, Crowdsec is unable to acquire the /var/log/filter/latest.log file upon system reboot. However, nginx logs under /var/log/nginx are working as expected. Manually restarting Crowdsec resolves the issue temporarily.

What did you expect to happen?

Crowdsec should automatically acquire the /var/log/filter/latest.log file after an OPNsense reboot, even when using a RAM disk for /var/log.

How can we reproduce it (as minimally and precisely as possible)?

Configure OPNsense to use a RAM disk for /var/log.
Reboot OPNsense.
Observe that Crowdsec does not acquire /var/log/filter/latest.log.

Anything else we need to know?

Nginx logs under /var/log/nginx are working correctly.
The issue only occurs when using a RAM disk for /var/log on OPNsense.

Crowdsec version

```console root@OPNsense:/home/admin # cscli version 2023/03/31 18:06:54 version: v1.4.6-c8cb9ac9 2023/03/31 18:06:54 Codename: alphaga 2023/03/31 18:06:54 BuildDate: 2023-03-21_03:09:29 2023/03/31 18:06:54 GoVersion: 1.19.7 2023/03/31 18:06:54 Platform: freebsd 2023/03/31 18:06:54 Constraint_parser: >= 1.0, <= 2.0 2023/03/31 18:06:54 Constraint_scenario: >= 1.0, < 3.0 2023/03/31 18:06:54 Constraint_api: v1 2023/03/31 18:06:54 Constraint_acquis: >= 1.0, < 2.0 ```

OS version

```console root@OPNsense:/home/admin # opnsense-version OPNsense 23.1.5_2 root@OPNsense:/home/admin # uname -a FreeBSD OPNsense.localdomain 13.1-RELEASE-p7 FreeBSD 13.1-RELEASE-p7 stable/23.1-n250411-85724e9ce22 SMP amd64 ```

Enabled collections and parsers

```console root@OPNsense:/home/admin # cscli hub list -o raw LePresidente/adguardhome,enabled,0.1,AdGuardHome Support : parser and brute-force detection,collections crowdsecurity/base-http-scenarios,"enabled,tainted",0.6,http common : scanners detection,collections crowdsecurity/freebsd,enabled,0.1,core freebsd support : syslog+geoip+ssh,collections crowdsecurity/http-cve,enabled,1.9,,collections crowdsecurity/opnsense,enabled,0.4,core opnsense support,collections crowdsecurity/opnsense-gui,enabled,0.1,OPNSense web authentication support,collections crowdsecurity/sshd,enabled,0.2,sshd support : parser and brute-force detection,collections crowdsecurity/suricata,enabled,0.1,suricata support : parser and automatic remediation on high/major alerts,collections firewallservices/pf,enabled,0.1,Parser and scenario for Packet Filter logs,collections LePresidente/adguardhome-logs,enabled,0.1,Parse adguardhome logs,parsers crowdsecurity/dateparse-enrich,enabled,0.2,,parsers crowdsecurity/geoip-enrich,enabled,0.2,"Populate event with geoloc info : as, country, coords, source range.",parsers crowdsecurity/http-logs,enabled,1.1,"Parse more Specifically HTTP logs, such as HTTP Code, HTTP path, HTTP args and if its a static ressource",parsers crowdsecurity/nginx-logs,enabled,1.3,Parse nginx access and error logs,parsers crowdsecurity/opnsense-gui-logs,enabled,0.1,Parse OPNSense web auth logs,parsers crowdsecurity/sshd-logs,enabled,2.0,Parse openSSH logs,parsers crowdsecurity/suricata-logs,enabled,0.6,Parse suricata fast.log,parsers crowdsecurity/syslog-logs,enabled,0.8,,parsers crowdsecurity/whitelists,enabled,0.2,Whitelist events from private ipv4 addresses,parsers firewallservices/pf-logs,enabled,0.3,Parse packet filter logs,parsers LePresidente/adguardhome-bf,enabled,0.1,Detect AdGuardHome bruteforce attacks,scenarios crowdsecurity/CVE-2022-26134,enabled,0.1,Detect CVE-2022-26134 exploits,scenarios crowdsecurity/CVE-2022-35914,enabled,0.1,Detect CVE-2022-35914 exploits,scenarios crowdsecurity/CVE-2022-37042,enabled,0.1,Detect CVE-2022-37042 exploits,scenarios crowdsecurity/CVE-2022-40684,enabled,0.2,Detect cve-2022-40684 exploitation attempts,scenarios crowdsecurity/CVE-2022-41082,enabled,0.3,Detect CVE-2022-41082 exploits,scenarios crowdsecurity/CVE-2022-41697,enabled,0.1,Detect CVE-2022-41697 enumeration,scenarios crowdsecurity/CVE-2022-42889,enabled,0.2,Detect CVE-2022-42889 exploits (Text4Shell),scenarios crowdsecurity/CVE-2022-44877,enabled,0.2,Detect CVE-2022-44877 exploits,scenarios crowdsecurity/CVE-2022-46169,enabled,0.1,Detect CVE-2022-46169 brute forcing,scenarios crowdsecurity/apache_log4j2_cve-2021-44228,enabled,0.4,Detect cve-2021-44228 exploitation attemps,scenarios crowdsecurity/f5-big-ip-cve-2020-5902,enabled,0.1,Detect cve-2020-5902 exploitation attemps,scenarios crowdsecurity/fortinet-cve-2018-13379,enabled,0.2,Detect cve-2018-13379 exploitation attemps,scenarios crowdsecurity/grafana-cve-2021-43798,enabled,0.1,Detect cve-2021-43798 exploitation attemps,scenarios crowdsecurity/http-backdoors-attempts,enabled,0.3,Detect attempt to common backdoors,scenarios crowdsecurity/http-crawl-non_statics,enabled,0.3,Detect aggressive crawl from single ip,scenarios crowdsecurity/http-cve-2021-41773,enabled,0.1,cve-2021-41773,scenarios crowdsecurity/http-cve-2021-42013,enabled,0.1,cve-2021-42013,scenarios crowdsecurity/http-generic-bf,enabled,0.4,Detect generic http brute force,scenarios crowdsecurity/http-open-proxy,enabled,0.3,Detect scan for open proxy,scenarios crowdsecurity/http-path-traversal-probing,enabled,0.2,Detect path traversal attempt,scenarios crowdsecurity/http-probing,enabled,0.2,Detect site scanning/probing from a single ip,scenarios crowdsecurity/http-sensitive-files,enabled,0.2,"Detect attempt to access to sensitive files (.log, .db ..) or folders (.git)",scenarios crowdsecurity/http-sqli-probing,enabled,0.2,A scenario that detects SQL injection probing with minimal false positives,scenarios crowdsecurity/http-xss-probing,enabled,0.2,A scenario that detects XSS probing with minimal false positives,scenarios crowdsecurity/jira_cve-2021-26086,enabled,0.1,Detect Atlassian Jira CVE-2021-26086 exploitation attemps,scenarios crowdsecurity/nginx-req-limit-exceeded,enabled,0.1,Detects IPs which violate nginx's user set request limit.,scenarios crowdsecurity/opnsense-gui-bf,enabled,0.1,Detect bruteforce on opnsense web interface,scenarios crowdsecurity/pulse-secure-sslvpn-cve-2019-11510,enabled,0.2,Detect cve-2019-11510 exploitation attemps,scenarios crowdsecurity/spring4shell_cve-2022-22965,enabled,0.2,Detect cve-2022-22965 probing,scenarios crowdsecurity/ssh-bf,enabled,0.1,Detect ssh bruteforce,scenarios crowdsecurity/ssh-slow-bf,enabled,0.2,Detect slow ssh bruteforce,scenarios crowdsecurity/suricata-alerts,enabled,0.3,Detect exploit attempts via emerging threat rules,scenarios crowdsecurity/thinkphp-cve-2018-20062,enabled,0.3,Detect ThinkPHP CVE-2018-20062 exploitation attemps,scenarios crowdsecurity/vmware-cve-2022-22954,enabled,0.2,Detect Vmware CVE-2022-22954 exploitation attempts,scenarios crowdsecurity/vmware-vcenter-vmsa-2021-0027,enabled,0.1,Detect VMSA-2021-0027 exploitation attemps,scenarios firewallservices/pf-scan-multi_ports,enabled,0.2,ban IPs that are scanning us,scenarios ltsich/http-w00tw00t,enabled,0.1,detect w00tw00t,scenarios ```

Acquisition config

````console root@OPNsense:/home/admin # cat /usr/local/etc/crowdsec/acquis.yaml filenames: - /var/log/nginx/*.log - ./tests/nginx/nginx.log #this is not a syslog log, indicate which kind of logs it is labels: type: nginx --- filenames: - /var/log/auth.log - /var/log/syslog labels: type: syslog --- filenames: - /var/log/httpd-access.log - /var/log/httpd-error.log labels: type: apache2 --- filename: /var/log/suricata/eve.json labels: type: suricata-evelogs --- filename: /var/log/filter/latest.log labels: type: syslog root@OPNsense:/home/admin # cat /usr/local/etc/crowdsec/acquis.d/* # # Before 22.1, OPNsense used circular logs under /var/log/*.log that # can still be around. They are old, in binary format and are not needed by crowdsec. # # For this reason we don't scan /var/log/*.log, but some plugins can write # their (plaintext) logs in that location, in such case add their pathnames too. # filenames: # DO NOT EDIT - to add new datasources (log locations), # create new files in /usr/local/etc/crowdsec/acquis.d/ # # collection: crowdsecurity/sshd - /var/log/audit/latest.log # collection: crowdsecurity/opnsense-gui (web admin) - /var/log/lighttpd/latest.log labels: type: syslog ````

Config show

```console root@OPNsense:/home/admin # cscli config show Global: - Configuration Folder : /usr/local/etc/crowdsec - Data Folder : /var/db/crowdsec/data - Hub Folder : /usr/local/etc/crowdsec/hub - Simulation File : /usr/local/etc/crowdsec/simulation.yaml - Log Folder : /var/log/crowdsec - Log level : info - Log Media : file Crowdsec: - Acquisition File : /usr/local/etc/crowdsec/acquis.yaml - Parsers routines : 1 - Acquisition Folder : /usr/local/etc/crowdsec/acquis.d/ cscli: - Output : human - Hub Branch : - Hub Folder : /usr/local/etc/crowdsec/hub Local API Server: - Listen URL : 192.168.100.1:8090 - Profile File : /usr/local/etc/crowdsec/profiles.yaml - Trusted IPs: - 127.0.0.1 - ::1 - Database: - Type : sqlite - Path : /var/db/crowdsec/data/crowdsec.db - Flush age : 7d - Flush size : 5000 ```

Prometheus metrics

```console root@OPNsense:/home/admin # cscli metrics Acquisition Metrics: ╭───────────────────────────────────────────────────┬────────────┬──────────────┬────────────────┬────────────────────────╮ │ Source │ Lines read │ Lines parsed │ Lines unparsed │ Lines poured to bucket │ ├───────────────────────────────────────────────────┼────────────┼──────────────┼────────────────┼────────────────────────┤ │ file:/var/log/nginx/error.log │ 1 │ - │ 1 │ - │ │ file:/var/log/nginx/*.access.log │ 8 │ 8 │ - │ - │ │ file:/var/log/nginx/tls_handshake.log │ 8 │ - │ 8 │ - │ ╰───────────────────────────────────────────────────┴────────────┴──────────────┴────────────────┴────────────────────────╯ ```

Related custom configs versions (if applicable) : notification plugins, custom scenarios, parsers etc.

github-actions[bot] commented 1 year ago

@yye347: Thanks for opening an issue, it is currently awaiting triage.

In the meantime, you can:

Check Crowdsec Documentation to see if your issue can be self resolved.
You can also join our Discord.
Check Releases to make sure your agent is on the latest version.

Details

I am a bot created to help the [crowdsecurity](https://github.com/crowdsecurity) developers manage community feedback and contributions. You can check out my [manifest file](https://github.com/crowdsecurity/crowdsec/blob/master/.github/governance.yml) to understand my behavior and what I can do. If you want to use this for your project, you can check out the [BirthdayResearch/oss-governance-bot](https://github.com/BirthdayResearch/oss-governance-bot) repository.

mmetc commented 1 year ago

Hi @yye347 , thanks for reporting

As you can see, the logs are created just a second after crowdsec is run (st_birthtime)

# stat -s /var/log/crowdsec/crowdsec.log
st_dev=18446744071679573761 st_ino=25 st_mode=0100600 st_nlink=1 st_uid=0 st_gid=0 st_rdev=18446744073709551615 st_size=5187 st_atime=1680853841 st_mtime=1680853499 st_ctime=1680853499 st_birthtime=1680853187 st_blksize=4096 st_blocks=16 st_flags=0
# stat -s /var/log/filter/latest.log
st_dev=18446744071679573761 st_ino=38 st_mode=0120750 st_nlink=1 st_uid=0 st_gid=0 st_rdev=18446744073709551615 st_size=35 st_atime=1680853644 st_mtime=1680853188 st_ctime=1680853188 st_birthtime=1680853188 st_blksize=4096 st_blocks=0 st_flags=0

For this case, there is an option I must add add in /usr/local/etc/crowdsec/acquis.d/opnsense.yaml

[...]
  # collection: crowdsecurity/sshd
  - /var/log/audit/latest.log
  # collection: crowdsecurity/opnsense-gui (web admin)
  - /var/log/lighttpd/latest.log
  # collection: firewallservices/pf
  - /var/log/filter/latest.log
force_inotify: true
labels:
  type: syslog

Just add force_inotify to your acquisition file and let me know. I'll add it to the 1.0.4 plugin. Even if freebsd has an alternative to inotify, the option covers both cases. Keep in mind the 1.0.3 has just been merged, it includes filter/latest.log so you'll have to remove it on your side. Parsing the same file twice can increase the sensitivity of scenarios.

yye347 commented 1 year ago

It works now, thanks!

crowdsecurity / crowdsec