Crowdsec crashes after startup with: runtime error: invalid memory address or nil pointer dereference

laugmanuel commented 1 month ago

What happened?

When starting Crowdsec as a from the dev container image without mounting a db volume (or empty volume), the service crashes when trying to send metrics to the APIC. The log shows:

[...]
crowdsec  | time="2024-08-08T13:08:23Z" level=debug msg="serving API after 1.056009297s ms"
crowdsec  | time="2024-08-08T13:08:23Z" level=info msg="Start push to CrowdSec Central API (interval: 13s once, then 10s)"
crowdsec  | time="2024-08-08T13:08:23Z" level=debug msg="loading hub idx /etc/crowdsec/hub/.index.json"
crowdsec  | time="2024-08-08T13:08:23Z" level=debug msg="1 scenarios for machine 1"
crowdsec  | time="2024-08-08T13:08:23Z" level=debug msg="Returning list of scenarios : []"
crowdsec  | time="2024-08-08T13:08:23Z" level=warning msg="scenario list is empty, will not pull yet"
crowdsec  | time="2024-08-08T13:08:23Z" level=info msg="Start sending metrics to CrowdSec Central API (interval: 42m25s once, then 30m0s)"
crowdsec  | time="2024-08-08T13:08:23Z" level=info msg="CrowdSec Local API listening on 0.0.0.0:8080"
crowdsec  | time="2024-08-08T13:08:23Z" level=error msg="crowdsec - goroutine lapi/metricsToAPIC crashed: runtime error: invalid memory address or nil pointer dereference"
crowdsec  | time="2024-08-08T13:08:23Z" level=error msg="please report this error to https://github.com/crowdsecurity/crowdsec/issues"
crowdsec  | time="2024-08-08T13:08:23Z" level=error msg="stacktrace/report is written to /var/lib/crowdsec/data/trace/crowdsec-crash.2342213429.txt: please join it to your issue"
crowdsec  | time="2024-08-08T13:08:23Z" level=fatal msg="crowdsec stopped"

When using a pre-populated db directory from (tag latest), the container starts up fine and works like it should.

What did you expect to happen?

Crowdsec to startup and work normally even when the db directory is empty.

How can we reproduce it (as minimally and precisely as possible)?

# compose.yml
services:
  crowdsec:
    container_name: crowdsec
    image: crowdsecurity/crowdsec:dev
    restart: unless-stopped
    environment:
      COLLECTIONS: "crowdsecurity/traefik crowdsecurity/http-cve crowdsecurity/whitelist-good-actors"
      PARSERS: "crowdsecurity/syslog-logs"
      BOUNCER_KEY_traefik: "traefik-bouncer-api-key"
      LEVEL_FATAL: true
      LEVEL_ERROR: true
      LEVEL_WARN: true
      LEVEL_INFO: true
      LEVEL_DEBUG: true
      LEVEL_TRACE: true

and run docker compose up -d

Anything else we need to know?

Starting tag dev with a pre-populated config and db directory from tag latest works just fine

Crowdsec version

```console $ cscli version version: v1.6.3-rc2-5-g1bc3b087 Codename: alphaga BuildDate: 2024-08-07_10:48:52 GoVersion: 1.22.5 Platform: docker libre2: C++ User-Agent: crowdsec/v1.6.3-rc2-5-g1bc3b087-docker Constraint_parser: >= 1.0, <= 3.0 Constraint_scenario: >= 1.0, <= 3.0 Constraint_api: v1 Constraint_acquis: >= 1.0, < 2.0 ```

OS version

Using the official docker image from dockerhub `crowdsecurity/crowdsec:dev` with sha `0985687e1f6b`

Enabled collections and parsers

```console $ cscli hub list -o raw name,status,version,description,type crowdsecurity/cri-logs,enabled,0.1,CRI logging format parser,parsers crowdsecurity/dateparse-enrich,enabled,0.2,,parsers crowdsecurity/docker-logs,enabled,0.1,docker json logs parser,parsers crowdsecurity/geoip-enrich,enabled,0.4,"Populate event with geoloc info : as, country, coords, source range.",parsers crowdsecurity/http-logs,enabled,1.2,"Parse more Specifically HTTP logs, such as HTTP Code, HTTP path, HTTP args and if its a static ressource",parsers crowdsecurity/sshd-logs,enabled,2.7,Parse openSSH logs,parsers crowdsecurity/syslog-logs,enabled,0.8,,parsers crowdsecurity/traefik-logs,enabled,0.9,Parse Traefik access logs,parsers crowdsecurity/whitelists,enabled,0.2,Whitelist events from private ipv4 addresses,parsers crowdsecurity/cdn-whitelist,enabled,0.4,Whitelist CDN providers,postoverflows crowdsecurity/rdns,enabled,0.3,Lookup the DNS associated to the source IP only for overflows,postoverflows crowdsecurity/seo-bots-whitelist,enabled,0.5,Whitelist good search engine crawlers,postoverflows crowdsecurity/apache_log4j2_cve-2021-44228,enabled,0.6,Detect cve-2021-44228 exploitation attemps,scenarios crowdsecurity/CVE-2017-9841,enabled,0.2,Detect CVE-2017-9841 exploits,scenarios crowdsecurity/CVE-2019-18935,enabled,0.2,Detect Telerik CVE-2019-18935 exploitation attempts,scenarios crowdsecurity/CVE-2022-26134,enabled,0.2,Detect CVE-2022-26134 exploits,scenarios crowdsecurity/CVE-2022-35914,enabled,0.2,Detect CVE-2022-35914 exploits,scenarios crowdsecurity/CVE-2022-37042,enabled,0.2,Detect CVE-2022-37042 exploits,scenarios crowdsecurity/CVE-2022-40684,enabled,0.3,Detect cve-2022-40684 exploitation attempts,scenarios crowdsecurity/CVE-2022-41082,enabled,0.4,Detect CVE-2022-41082 exploits,scenarios crowdsecurity/CVE-2022-41697,enabled,0.2,Detect CVE-2022-41697 enumeration,scenarios crowdsecurity/CVE-2022-42889,enabled,0.3,Detect CVE-2022-42889 exploits (Text4Shell),scenarios crowdsecurity/CVE-2022-44877,enabled,0.3,Detect CVE-2022-44877 exploits,scenarios crowdsecurity/CVE-2022-46169,enabled,0.2,Detect CVE-2022-46169 brute forcing,scenarios crowdsecurity/CVE-2023-22515,enabled,0.1,Detect CVE-2023-22515 exploitation,scenarios crowdsecurity/CVE-2023-22518,enabled,0.2,Detect CVE-2023-22518 exploits,scenarios crowdsecurity/CVE-2023-49103,enabled,0.3,Detect owncloud CVE-2023-49103 exploitation attempts,scenarios crowdsecurity/f5-big-ip-cve-2020-5902,enabled,0.2,Detect cve-2020-5902 exploitation attemps,scenarios crowdsecurity/fortinet-cve-2018-13379,enabled,0.3,Detect cve-2018-13379 exploitation attemps,scenarios crowdsecurity/grafana-cve-2021-43798,enabled,0.2,Detect cve-2021-43798 exploitation attemps,scenarios crowdsecurity/http-admin-interface-probing,enabled,0.4,Detect generic HTTP admin interface probing,scenarios crowdsecurity/http-backdoors-attempts,enabled,0.6,Detect attempt to common backdoors,scenarios crowdsecurity/http-bad-user-agent,enabled,1.2,Detect usage of bad User Agent,scenarios crowdsecurity/http-crawl-non_statics,enabled,0.7,Detect aggressive crawl on non static resources,scenarios crowdsecurity/http-cve-2021-41773,enabled,0.2,cve-2021-41773,scenarios crowdsecurity/http-cve-2021-42013,enabled,0.2,cve-2021-42013,scenarios crowdsecurity/http-cve-probing,enabled,0.2,Detect generic HTTP cve probing,scenarios crowdsecurity/http-generic-bf,enabled,0.6,Detect generic http brute force,scenarios crowdsecurity/http-open-proxy,enabled,0.5,Detect scan for open proxy,scenarios crowdsecurity/http-path-traversal-probing,enabled,0.4,Detect path traversal attempt,scenarios crowdsecurity/http-probing,enabled,0.4,Detect site scanning/probing from a single ip,scenarios crowdsecurity/http-sensitive-files,enabled,0.4,"Detect attempt to access to sensitive files (.log, .db ..) or folders (.git)",scenarios crowdsecurity/http-sqli-probing,enabled,0.4,A scenario that detects SQL injection probing with minimal false positives,scenarios crowdsecurity/http-wordpress-scan,enabled,0.2,Detect WordPress scan: vuln hunting,scenarios crowdsecurity/http-xss-probing,enabled,0.4,A scenario that detects XSS probing with minimal false positives,scenarios crowdsecurity/jira_cve-2021-26086,enabled,0.3,Detect Atlassian Jira CVE-2021-26086 exploitation attemps,scenarios crowdsecurity/netgear_rce,enabled,0.3,Detect Netgear RCE DGN1000/DGN220 exploitation attempts,scenarios crowdsecurity/pulse-secure-sslvpn-cve-2019-11510,enabled,0.3,Detect cve-2019-11510 exploitation attemps,scenarios crowdsecurity/spring4shell_cve-2022-22965,enabled,0.3,Detect cve-2022-22965 probing,scenarios crowdsecurity/ssh-bf,enabled,0.3,Detect ssh bruteforce,scenarios crowdsecurity/ssh-cve-2024-6387,enabled,0.2,Detect exploitation attempt of CVE-2024-6387,scenarios crowdsecurity/ssh-slow-bf,enabled,0.4,Detect slow ssh bruteforce,scenarios crowdsecurity/thinkphp-cve-2018-20062,enabled,0.6,Detect ThinkPHP CVE-2018-20062 exploitation attemps,scenarios crowdsecurity/vmware-cve-2022-22954,enabled,0.3,Detect Vmware CVE-2022-22954 exploitation attempts,scenarios crowdsecurity/vmware-vcenter-vmsa-2021-0027,enabled,0.2,Detect VMSA-2021-0027 exploitation attemps,scenarios ltsich/http-w00tw00t,enabled,0.2,detect w00tw00t,scenarios crowdsecurity/bf_base,enabled,0.1,,contexts crowdsecurity/http_base,enabled,0.2,,contexts crowdsecurity/base-http-scenarios,enabled,1.0,http common : scanners detection,collections crowdsecurity/http-cve,enabled,2.6,Detect CVE exploitation in http logs,collections crowdsecurity/linux,enabled,0.2,core linux support : syslog+geoip+ssh,collections crowdsecurity/sshd,enabled,0.5,sshd support : parser and brute-force detection,collections crowdsecurity/traefik,enabled,0.1,traefik support: parser and generic http scenarios,collections crowdsecurity/whitelist-good-actors,enabled,0.1,Good actors whitelists,collections ```

Acquisition config

```console # On Linux: $ cat /etc/crowdsec/acquis.yaml /etc/crowdsec/acquis.d/* filenames: - /var/log/nginx/*.log - ./tests/nginx/nginx.log #this is not a syslog log, indicate which kind of logs it is labels: type: nginx --- filenames: - /var/log/auth.log - /var/log/syslog labels: type: syslog --- filename: /var/log/apache2/*.log labels: type: apache2 cat: can't open '/etc/crowdsec/acquis.d/*': No such file or directory

Config show

```console $ cscli config show Global: - Configuration Folder : /etc/crowdsec - Data Folder : /var/lib/crowdsec/data - Hub Folder : /etc/crowdsec/hub - Simulation File : /etc/crowdsec/simulation.yaml - Log Folder : /var/log - Log level : info - Log Media : stdout Crowdsec: - Acquisition File : /etc/crowdsec/acquis.yaml - Parsers routines : 1 - Acquisition Folder : /etc/crowdsec/acquis.d cscli: - Output : human - Hub Branch : API Client: - URL : http://0.0.0.0:8080/ - Login : localhost - Credentials File : /etc/crowdsec/local_api_credentials.yaml Local API Server: - Listen URL : 0.0.0.0:8080 - Listen Socket : - Profile File : /etc/crowdsec/profiles.yaml - Allowed Agents OU : agent-ou - Allowed Bouncers OU : bouncer-ou - Trusted IPs: - 127.0.0.1 - ::1 - Database: - Type : sqlite - Path : /var/lib/crowdsec/data/crowdsec.db - Flush age : 7d - Flush size : 5000 ```

Prometheus metrics

```console $ cscli metrics # paste output here ```

Related custom configs versions (if applicable) : notification plugins, custom scenarios, parsers etc.

github-actions[bot] commented 1 month ago

@laugmanuel: Thanks for opening an issue, it is currently awaiting triage.

In the meantime, you can:

Check Crowdsec Documentation to see if your issue can be self resolved.
You can also join our Discord.
Check Releases to make sure your agent is on the latest version.

Details

I am a bot created to help the [crowdsecurity](https://github.com/crowdsecurity) developers manage community feedback and contributions. You can check out my [manifest file](https://github.com/crowdsecurity/crowdsec/blob/master/.github/governance.yml) to understand my behavior and what I can do. If you want to use this for your project, you can check out the [BirthdayResearch/oss-governance-bot](https://github.com/BirthdayResearch/oss-governance-bot) repository.

laugmanuel commented 1 month ago

I did some more digging and found out, that it is caused by setting BOUNCER_KEY_traefik as an environment variable for the automatic bouncer registration: https://github.com/crowdsecurity/crowdsec/blob/master/docker/README.md#automatic-bouncer-registration

LaurenceJJones commented 1 month ago

Will be fixed by #3170

LaurenceJJones commented 1 month ago

The TLDR; is when the bouncer is created in the database the LastPull timestamp is nil by default there was no check before attempting to use the value to convert it to a RFC timestamp. We are adding the check in the linked PR onced it is merged you can repull the dev image and it will fix the issue.

crowdsecurity / crowdsec