search

crowdsecurity / crowdsec

CrowdSec - the open-source and participative security solution offering crowdsourced protection against malicious IPs and access to the most advanced real-world CTI.
https://crowdsec.net
MIT License
8.87k stars 459 forks source link

Crowdsec 1.5.4 invalid memory address or nil pointer dereference (ProxMox LXC) #2499

Closed Mirabis closed 1 year ago

Mirabis commented 1 year ago

What happened?

Crowdsec crashes whenever I try to start it since moving to 1.5.4.

```console time="29-09-2023 15:47:00" level=error msg="crowdsec - goroutine lapi/pullFromAPIC crashed : runtime error: invalid memory address or nil pointer dereference" time="29-09-2023 15:47:00" level=error msg="please report this error to https://github.com/crowdsecurity/crowdsec/" time="29-09-2023 15:47:00" level=error msg="stacktrace/report is written to /tmp/crowdsec-crash.1951062260.txt : please join it to your issue" time="29-09-2023 15:47:00" level=fatal msg="crowdsec stopped" ```

Stacktrace/report:

```console root@crowdsec:/var/log# cat /tmp/crowdsec-crash.1951062260.txt error : runtime error: invalid memory address or nil pointer dereference version: v1.5.4-debian-pragmatic-amd64-e4dcdd25728b914823525f1efabf18d5c454902b BuildDate: 2023-09-20_12:15:26 GoVersion: 1.20.5 goroutine 118 [running]: runtime/debug.Stack() runtime/debug/stack.go:24 +0x65 github.com/crowdsecurity/go-cs-lib/trace.WriteStackTrace({0x1c819a0, 0x3072fc0}) github.com/crowdsecurity/go-cs-lib@v0.0.4/trace/trace.go:26 +0x24d github.com/crowdsecurity/go-cs-lib/trace.CatchPanic({0x1f069bd, 0x11}) github.com/crowdsecurity/go-cs-lib@v0.0.4/trace/trace.go:41 +0xfa panic({0x1c819a0, 0x3072fc0}) runtime/panic.go:884 +0x213 net.networkNumberAndMask(0xc000376272?) net/ip.go:498 net.(*IPNet).Contains(0xc000376272?, {0xc000e94000, 0x10, 0xc000a07958?}) net/ip.go:522 +0x27 github.com/crowdsecurity/crowdsec/pkg/apiserver.(*apic).whitelistedBy(0xc000002000, 0xc000a07a08?) github.com/crowdsecurity/crowdsec/pkg/apiserver/apic.go:629 +0xed github.com/crowdsecurity/crowdsec/pkg/apiserver.(*apic).ApplyApicWhitelists(0xc000002000, {0xc000ee4000?, 0xbb8, 0x1000}) github.com/crowdsecurity/crowdsec/pkg/apiserver/apic.go:648 +0xc7 github.com/crowdsecurity/crowdsec/pkg/apiserver.(*apic).PullTop(0xc000002000, 0x0) github.com/crowdsecurity/crowdsec/pkg/apiserver/apic.go:603 +0x905 github.com/crowdsecurity/crowdsec/pkg/apiserver.(*apic).Pull(0xc000002000) github.com/crowdsecurity/crowdsec/pkg/apiserver/apic.go:810 +0x154 github.com/crowdsecurity/crowdsec/pkg/apiserver.(*APIServer).Run.func2() github.com/crowdsecurity/crowdsec/pkg/apiserver/apiserver.go:351 +0x25 gopkg.in/tomb%2ev2.(*Tomb).run(0xc000002098, 0xc0005cf758?) gopkg.in/tomb.v2@v2.0.0-20161208151619-d5d1b5820637/tomb.go:163 +0x36 created by gopkg.in/tomb%2ev2.(*Tomb).Go gopkg.in/tomb.v2@v2.0.0-20161208151619-d5d1b5820637/tomb.go:159 +0xee ```

What did you expect to happen?

Crowdsec to actually run after doing systemctl start.

How can we reproduce it (as minimally and precisely as possible)?

Crowdsec 1.5.2 runs fine but 1.5.3 and 1.5.4 won't run at all.

Interestingly the installing steps crash too if I try to reproduce:

From 1.5.4 to 1.5.3

```console root@crowdsec:/var/log# apt install crowdsec=1.5.3 Reading package lists... Done Building dependency tree... Done Reading state information... Done The following packages will be DOWNGRADED: crowdsec 0 upgraded, 0 newly installed, 1 downgraded, 0 to remove and 0 not upgraded. Need to get 40.4 MB of archives. After this operation, 4096 B disk space will be freed. Do you want to continue? [Y/n] Y Get:1 https://packagecloud.io/crowdsec/crowdsec/debian bookworm/main amd64 crowdsec amd64 1.5.3 [40.4 MB] Fetched 40.4 MB in 4s (9940 kB/s) Preconfiguring packages ... dpkg: warning: downgrading crowdsec from 1.5.4 to 1.5.3 (Reading database ... 20523 files and directories currently installed.) Preparing to unpack .../crowdsec_1.5.3_amd64.deb ... You can always run the configuration again interactively by using '/usr/share/crowdsec/wizard.sh -c Unpacking crowdsec (1.5.3) over (1.5.4) ... Setting up crowdsec (1.5.3) ... Updating hub WARN[29-09-2023 15:56:59] Crowdsec is not the latest version. Current version is 'v1.5.3' and the latest stable version is 'v1.5.4'. Please update it! WARN[29-09-2023 15:56:59] As a result, you will not be able to use parsers/scenarios/collections added to Crowdsec Hub after CrowdSec v1.5.4 INFO[29-09-2023 15:56:59] Wrote new 813262 bytes index to /etc/crowdsec/hub/.index.json Job for crowdsec.service failed because the control process exited with error code. See "systemctl status crowdsec.service" and "journalctl -xeu crowdsec.service" for details. dpkg: error processing package crowdsec (--configure): installed crowdsec package post-installation script subprocess returned error exit status 1 Errors were encountered while processing: crowdsec E: Sub-process /usr/bin/dpkg returned an error code (1) ```

From 1.5.4 to 1.5.2

```console root@crowdsec:/var/log# apt install crowdsec=1.5.2 Reading package lists... Done Building dependency tree... Done Reading state information... Done The following packages will be DOWNGRADED: crowdsec 0 upgraded, 0 newly installed, 1 downgraded, 0 to remove and 0 not upgraded. 1 not fully installed or removed. Need to get 0 B/36.6 MB of archives. After this operation, 12.7 MB disk space will be freed. Do you want to continue? [Y/n] Y Preconfiguring packages ... dpkg: warning: downgrading crowdsec from 1.5.3 to 1.5.2 (Reading database ... 20523 files and directories currently installed.) Preparing to unpack .../crowdsec_1.5.2_amd64.deb ... You can always run the configuration again interactively by using '/usr/share/crowdsec/wizard.sh -c Unpacking crowdsec (1.5.2) over (1.5.3) ... Setting up crowdsec (1.5.2) ... Installing new version of config file /etc/crowdsec/simulation.yaml ... Updating hub WARN[29-09-2023 16:00:19] Crowdsec is not the latest version. Current version is 'v1.5.2' and the latest stable version is 'v1.5.4'. Please update it! WARN[29-09-2023 16:00:19] As a result, you will not be able to use parsers/scenarios/collections added to Crowdsec Hub after CrowdSec v1.5.4 INFO[29-09-2023 16:00:19] hub index is up to date INFO[29-09-2023 16:00:19] Wrote new 813262 bytes index to /etc/crowdsec/hub/.index.json You can always run the configuration again interactively by using '/usr/share/crowdsec/wizard.sh -c root@crowdsec:/var/log# ```

Anything else we need to know?

No response

Crowdsec version

```console $ cscli version 2023/09/29 15:48:05 version: v1.5.4-debian-pragmatic-amd64-e4dcdd25728b914823525f1efabf18d5c454902b 2023/09/29 15:48:05 Codename: alphaga 2023/09/29 15:48:05 BuildDate: 2023-09-20_12:17:53 2023/09/29 15:48:05 GoVersion: 1.20.5 2023/09/29 15:48:05 Platform: linux 2023/09/29 15:48:05 libre2: C++ 2023/09/29 15:48:05 Constraint_parser: >= 1.0, <= 2.0 2023/09/29 15:48:05 Constraint_scenario: >= 1.0, < 3.0 2023/09/29 15:48:05 Constraint_api: v1 2023/09/29 15:48:05 Constraint_acquis: >= 1.0, < 2.0 ```

OS version

```console # On Linux: $ cat /etc/os-release PRETTY_NAME="Debian GNU/Linux 12 (bookworm)" NAME="Debian GNU/Linux" VERSION_ID="12" VERSION="12 (bookworm)" VERSION_CODENAME=bookworm ID=debian HOME_URL="https://www.debian.org/" SUPPORT_URL="https://www.debian.org/support" BUG_REPORT_URL="https://bugs.debian.org/" $ uname -a Linux crowdsec 6.2.16-14-pve #1 SMP PREEMPT_DYNAMIC PMX 6.2.16-14 (2023-09-19T08:17Z) x86_64 GNU/Linux ```

Enabled collections and parsers

```console $ cscli hub list -o raw crowdsecurity/iptables,enabled,0.1,iptables support : logs and port-scans detection scenarios,collections crowdsecurity/linux,enabled,0.2,core linux support : syslog+geoip+ssh,collections crowdsecurity/sshd,enabled,0.2,sshd support : parser and brute-force detection,collections crowdsecurity/unifi,enabled,0.1,Unifi support: syslog parser + port scan + SSH BF detection,collections crowdsecurity/wireguard,enabled,0.1,wireguard auth detection,collections fulljackz/proxmox,enabled,0.1,Proxmox Web interface support : parser for brute force detection on Proxmox VE Web UI,collections crowdsecurity/dateparse-enrich,enabled,0.2,,parsers crowdsecurity/dropbear-logs,enabled,0.2,Parse dropbear logs,parsers crowdsecurity/geoip-enrich,enabled,0.2,"Populate event with geoloc info : as, country, coords, source range.",parsers crowdsecurity/iptables-logs,enabled,0.5,Parse iptables drop logs,parsers crowdsecurity/sshd-logs,enabled,2.2,Parse openSSH logs,parsers crowdsecurity/syslog-logs,enabled,0.8,,parsers crowdsecurity/unifi-logs,enabled,0.1,,parsers crowdsecurity/wireguard-logs,enabled,0.1,Parses wireguard log via dyndbg,parsers fulljackz/proxmox-logs,enabled,0.2,Parse proxmox logs for bruteforce attempts,parsers crowdsecurity/iptables-scan-multi_ports,enabled,0.1,ban IPs that are scanning us,scenarios crowdsecurity/ssh-bf,enabled,0.1,Detect ssh bruteforce,scenarios crowdsecurity/ssh-slow-bf,enabled,0.2,Detect slow ssh bruteforce,scenarios crowdsecurity/wireguard-auth,enabled,0.1,Detects rejected connections attempts and unauthorized packets through wireguard tunnels,scenarios fulljackz/proxmox-bf,enabled,0.1,Detect proxmox bruteforce,scenarios ```

Acquisition config

```console #Generated acquisition file - wizard.sh (service: ssh) / files : #journalctl_filter: # - _SYSTEMD_UNIT=ssh.service #labels: # type: syslog --- filenames: - /var/log/auth.log - /var/log/syslog labels: type: syslog # Proxmox Host filenames: - /var/log/remote/99-proxmox.log labels: type: syslog external_format: proxmox # https://hub.crowdsec.net/author/crowdsecurity/collections/unifi filenames: - /var/log/remote/99-unifi.log labels: type: unifi --- # Proxmox Host filenames: - /var/log/remote/99-wireguard.log labels: type: syslog external_format: wireguard ```

Config show

```console root@crowdsec:/var/log# cscli config show Global: - Configuration Folder : /etc/crowdsec - Data Folder : /var/lib/crowdsec/data - Hub Folder : /etc/crowdsec/hub - Simulation File : /etc/crowdsec/simulation.yaml - Log Folder : /var/log/ - Log level : info - Log Media : file Crowdsec: - Acquisition File : /etc/crowdsec/acquis.yaml - Parsers routines : 1 - Acquisition Folder : /etc/crowdsec/acquis.d cscli: - Output : human - Hub Branch : - Hub Folder : /etc/crowdsec/hub API Client: - URL : http://localhost:8080/ - Login : crowdsec.mirabis.lab - Credentials File : /etc/crowdsec/local_api_credentials.yaml Local API Server: - Listen URL : 0.0.0.0:8080 - Profile File : /etc/crowdsec/profiles.yaml - Trusted IPs: - 127.0.0.1 - 10.0.30.0/24 - 10.0.40.5 - ::1 - Database: - Type : mysql - Host : 20.0.20.104 - Port : 3306 - User : crowdsec - DB Name : crowdsec - Max Open Conns : 100 - Flush age : 7d - Flush size : 5000 ```

Prometheus metrics

```console root@crowdsec:/var/log# cscli metrics FATA[29-09-2023 15:53:03] could not fetch prometheus metrics: failed to fetch prometheus metrics: executing GET request for URL "http://20.0.20.102:6060/metrics" failed: Get "http://20.0.20.102:6060/metrics": dial tcp 20.0.20.102:6060: connect: connection refused ```

Related custom configs versions (if applicable) : notification plugins, custom scenarios, parsers etc.

N/A
github-actions[bot] commented 1 year ago

@Mirabis: Thanks for opening an issue, it is currently awaiting triage.

In the meantime, you can:

  1. Check Crowdsec Documentation to see if your issue can be self resolved.
  2. You can also join our Discord.
  3. Check Releases to make sure your agent is on the latest version.
Details I am a bot created to help the [crowdsecurity](https://github.com/crowdsecurity) developers manage community feedback and contributions. You can check out my [manifest file](https://github.com/crowdsecurity/crowdsec/blob/master/.github/governance.yml) to understand my behavior and what I can do. If you want to use this for your project, you can check out the [BirthdayResearch/oss-governance-bot](https://github.com/BirthdayResearch/oss-governance-bot) repository.
LaurenceJJones commented 1 year ago

Do you have any CAPI whitelist set?

Mirabis commented 1 year ago

Have not whitelisted anything as far as I'm aware. Is there a specific command you want me to run?

LaurenceJJones commented 1 year ago

Have not whitelisted anything as far as I'm aware. Is there a specific command you want me to run?

If you don't know then most likely not 👍🏻 Could you share your whole /etc/crowdsec/config.yaml but redact your DB_PASSWORD from it.

Mirabis commented 1 year ago

Have not whitelisted anything as far as I'm aware. Is there a specific command you want me to run?

If you don't know then most likely not 👍🏻 Could you share your whole /etc/crowdsec/config.yaml but redact your DB_PASSWORD from it.

`/etc/crowdsec/config.yaml'

```console root@crowdsec:/etc/crowdsec# cat config.yaml common: daemonize: true log_media: file log_level: info log_dir: /var/log/ log_max_size: 20 compress_logs: true log_max_files: 10 working_dir: . config_paths: config_dir: /etc/crowdsec/ data_dir: /var/lib/crowdsec/data/ simulation_path: /etc/crowdsec/simulation.yaml hub_dir: /etc/crowdsec/hub/ index_path: /etc/crowdsec/hub/.index.json notification_dir: /etc/crowdsec/notifications/ plugin_dir: /usr/lib/crowdsec/plugins/ crowdsec_service: #console_context_path: /etc/crowdsec/console/context.yaml acquisition_path: /etc/crowdsec/acquis.yaml acquisition_dir: /etc/crowdsec/acquis.d parser_routines: 1 cscli: output: human color: auto db_config: log_level: error type: mysql #db_path: /var/lib/crowdsec/data/crowdsec.db max_open_conns: 100 user: crowdsec password: ******************* db_name: crowdsec host: 20.0.20.104 port: 3306 flush: max_items: 5000 max_age: 7d plugin_config: user: nobody # plugin process would be ran on behalf of this user group: nogroup # plugin process would be ran on behalf of this group api: client: insecure_skip_verify: false credentials_path: /etc/crowdsec/local_api_credentials.yaml server: log_level: info listen_uri: 0.0.0.0:8080 use_forwarded_for_headers: true profiles_path: /etc/crowdsec/profiles.yaml console_path: /etc/crowdsec/console.yaml capi_whitelists_path: /etc/crowdsec/capi_whitelist.yaml online_client: # Central API credentials (to push signals and receive bad IPs) credentials_path: /etc/crowdsec/online_api_credentials.yaml trusted_ips: # IP ranges, or IPs which can have admin API access - 127.0.0.1 - 10.0.30.0/24 - 10.0.40.5 - ::1 # tls: # cert_file: /etc/crowdsec/ssl/cert.pem # key_file: /etc/crowdsec/ssl/key.pem prometheus: enabled: true level: aggregated #or full listen_addr: 20.0.20.102 listen_port: 6060 # # https://docs.crowdsec.net/docs/data_sources/syslog #source: syslog # listen_addr: 127.0.0.1 # listen_port: 4242 # labels: # type: syslog ```
mmetc commented 1 year ago

Can you try without this?

capi_whitelists_path: /etc/crowdsec/capi_whitelist.yaml
sriccio commented 1 year ago

I had the same issue and thought it was a false alert because the system disk was full and after freeing space the service started. https://github.com/crowdsecurity/crowdsec/issues/2481

But after a few restarts it crashed again, had to revert to 1.5.3

I'm indeed using capi whitelist:

capi_whitelists_path: /etc/crowdsec/capi_whitelists.yaml

root@crowdsec:~# cat /etc/crowdsec/capi_whitelists.yaml

ips:
 # Customer 1
 - w.x.y.z
cidrs:
 # Sucuri
 - 192.88.134.0/23
 - 185.93.228.0/22
 - 66.248.200.0/22
 - 2a02:fe80::/29
 - 208.109.0.0/22

Note: Ip of customer obfuscated for privacy concerns...

Mirabis commented 1 year ago

contents of the file:

$ root@crowdsec:/etc/crowdsec# cat /etc/crowdsec/capi_whitelist.yaml 
ips:
 - 1.1.1.1
 - 10.0.40.5
cidrs:
 - 10.0.30.0/24

Commented it out of the config.yaml and ran apt-upgrade from 1.5.2 to 1.5.4. Ran fine after the upgrade (and the odd install message didn't appear either.

mmetc commented 1 year ago

Thanks again, fixed in master and we may release 1.5.5-rc1 with it pretty soon.