search

crowdsecurity / crowdsec

CrowdSec - the open-source and participative security solution offering crowdsourced protection against malicious IPs and access to the most advanced real-world CTI.
https://crowdsec.net
MIT License
9.08k stars 470 forks source link

Invalid memory address or nil pointer dereference #3295

Closed priyankub closed 4 weeks ago

priyankub commented 4 weeks ago

What happened?

Crowdsec crashed.

What did you expect to happen?

It to not crash. Its been running stable for more than a year now

How can we reproduce it (as minimally and precisely as possible)?

Not sure

Anything else we need to know?

time="2024-10-23T17:51:51-04:00" level=error msg="capi metrics: failed: Post \"https://api.crowdsec.net/v3/metrics/\": performing jwt auth: dial tcp: lookup api.crowdsec.net on 127.0.0.11:53: read udp 127.0.0.1:46180->127.0.0.11:53: i/o timeout" time="2024-10-23T18:34:39-04:00" level=error msg="capi pull top: get stream: Get \"https://api.crowdsec.net/v3/decisions/stream?\": performing jwt auth: dial tcp: lookup api.crowdsec.net on 127.0.0.11:53: read udp 127.0.0.1:43876->127.0.0.11:53: i/o timeout" time="2024-10-23T19:22:03-04:00" level=error msg="capi metrics: failed: Post \"https://api.crowdsec.net/v3/metrics/\": performing jwt auth: dial tcp: lookup api.crowdsec.net on 127.0.0.11:53: read udp 127.0.0.1:33450->127.0.0.11:53: i/o timeout" time="2024-10-23T19:59:51-04:00" level=error msg="unable to send usage metrics: Post \"https://api.crowdsec.net/v3/usage-metrics\": performing jwt auth: dial tcp: lookup api.crowdsec.net on 127.0.0.11:53: read udp 127.0.0.1:37044->127.0.0.11:53: i/o timeout" time="2024-10-23T19:59:51-04:00" level=error msg="crowdsec - goroutine lapi/usageMetricsToAPIC crashed: runtime error: invalid memory address or nil pointer dereference" time="2024-10-23T19:59:51-04:00" level=error msg="please report this error to https://github.com/crowdsecurity/crowdsec/issues" time="2024-10-23T19:59:51-04:00" level=error msg="stacktrace/report is written to /var/lib/crowdsec/data/trace/crowdsec-crash.3536834419.txt: please join it to your issue" time="2024-10-23T19:59:51-04:00" level=fatal msg="crowdsec stopped" time="2024-10-24T00:49:21-04:00" level=debug msg="starting plugin" args="[/usr/

Crowdsec version

```console $ cscli version # paste output here ``` version: v1.6.3-4851945a Codename: alphaga BuildDate: 2024-09-12_09:39:08 GoVersion: 1.22.6 Platform: docker libre2: C++ User-Agent: crowdsec/v1.6.3-4851945a-docker Constraint_parser: >= 1.0, <= 3.0 Constraint_scenario: >= 1.0, <= 3.0 Constraint_api: v1 Constraint_acquis: >= 1.0, < 2.0

OS version

```console # On Linux: $ cat /etc/os-release # paste output here $ uname -a # paste output here # On Windows: C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture # paste output here ```

Docker on Unraid

Enabled collections and parsers

```console $ cscli hub list -o raw # paste output here ``` name,status,version,description,type crowdsecurity/appsec-logs,enabled,0.5,Parse Appsec events,parsers crowdsecurity/cri-logs,enabled,0.1,CRI logging format parser,parsers crowdsecurity/dateparse-enrich,enabled,0.2,,parsers crowdsecurity/docker-logs,enabled,0.1,docker json logs parser,parsers crowdsecurity/geoip-enrich,enabled,0.5,"Populate event with geoloc info : as, country, coords, source range.",parsers crowdsecurity/http-logs,enabled,1.2,"Parse more Specifically HTTP logs, such as HTTP Code, HTTP path, HTTP args and if its a static ressource",parsers crowdsecurity/mywhitelists,"enabled,local",,,parsers crowdsecurity/nextcloud-logs,enabled,0.3,Parse nextcloud logs,parsers crowdsecurity/nextcloud-whitelist,enabled,1.1,Whitelist events from nextcloud,parsers crowdsecurity/sshd-logs,enabled,2.8,Parse openSSH logs,parsers crowdsecurity/syslog-logs,enabled,0.8,,parsers crowdsecurity/traefik-logs,enabled,0.9,Parse Traefik access logs,parsers crowdsecurity/whitelists,enabled,0.2,Whitelist events from private ipv4 addresses,parsers Dominic-Wagner/vaultwarden-logs,enabled,0.1,Parse vaultwarden logs,parsers crowdsecurity/apache_log4j2_cve-2021-44228,enabled,0.6,Detect cve-2021-44228 exploitation attemps,scenarios crowdsecurity/appsec-vpatch,enabled,0.5,Identify attacks flagged by CrowdSec AppSec,scenarios crowdsecurity/CVE-2017-9841,enabled,0.2,Detect CVE-2017-9841 exploits,scenarios crowdsecurity/CVE-2019-18935,enabled,0.2,Detect Telerik CVE-2019-18935 exploitation attempts,scenarios crowdsecurity/CVE-2022-26134,enabled,0.2,Detect CVE-2022-26134 exploits,scenarios crowdsecurity/CVE-2022-35914,enabled,0.2,Detect CVE-2022-35914 exploits,scenarios crowdsecurity/CVE-2022-37042,enabled,0.2,Detect CVE-2022-37042 exploits,scenarios crowdsecurity/CVE-2022-40684,enabled,0.3,Detect cve-2022-40684 exploitation attempts,scenarios crowdsecurity/CVE-2022-41082,enabled,0.4,Detect CVE-2022-41082 exploits,scenarios crowdsecurity/CVE-2022-41697,enabled,0.2,Detect CVE-2022-41697 enumeration,scenarios crowdsecurity/CVE-2022-42889,enabled,0.3,Detect CVE-2022-42889 exploits (Text4Shell),scenarios crowdsecurity/CVE-2022-44877,enabled,0.3,Detect CVE-2022-44877 exploits,scenarios crowdsecurity/CVE-2022-46169,enabled,0.2,Detect CVE-2022-46169 brute forcing,scenarios crowdsecurity/CVE-2023-22515,enabled,0.1,Detect CVE-2023-22515 exploitation,scenarios crowdsecurity/CVE-2023-22518,enabled,0.2,Detect CVE-2023-22518 exploits,scenarios crowdsecurity/CVE-2023-49103,enabled,0.3,Detect owncloud CVE-2023-49103 exploitation attempts,scenarios crowdsecurity/CVE-2024-38475,enabled,0.1,Detect CVE-2024-38475 exploitation attempts,scenarios crowdsecurity/f5-big-ip-cve-2020-5902,enabled,0.2,Detect cve-2020-5902 exploitation attemps,scenarios crowdsecurity/fortinet-cve-2018-13379,enabled,0.3,Detect cve-2018-13379 exploitation attemps,scenarios crowdsecurity/grafana-cve-2021-43798,enabled,0.2,Detect cve-2021-43798 exploitation attemps,scenarios crowdsecurity/http-admin-interface-probing,enabled,0.4,Detect generic HTTP admin interface probing,scenarios crowdsecurity/http-backdoors-attempts,enabled,0.6,Detect attempt to common backdoors,scenarios crowdsecurity/http-bad-user-agent,enabled,1.2,Detect usage of bad User Agent,scenarios crowdsecurity/http-crawl-non_statics,enabled,0.7,Detect aggressive crawl on non static resources,scenarios crowdsecurity/http-cve-2021-41773,enabled,0.2,cve-2021-41773,scenarios crowdsecurity/http-cve-2021-42013,enabled,0.2,cve-2021-42013,scenarios crowdsecurity/http-cve-probing,enabled,0.2,Detect generic HTTP cve probing,scenarios crowdsecurity/http-generic-bf,enabled,0.6,Detect generic http brute force,scenarios crowdsecurity/http-open-proxy,enabled,0.5,Detect scan for open proxy,scenarios crowdsecurity/http-path-traversal-probing,enabled,0.4,Detect path traversal attempt,scenarios crowdsecurity/http-probing,enabled,0.4,Detect site scanning/probing from a single ip,scenarios crowdsecurity/http-sensitive-files,enabled,0.4,"Detect attempt to access to sensitive files (.log, .db ..) or folders (.git)",scenarios crowdsecurity/http-sqli-probing,enabled,0.4,A scenario that detects SQL injection probing with minimal false positives,scenarios crowdsecurity/http-wordpress-scan,enabled,0.2,Detect WordPress scan: vuln hunting,scenarios crowdsecurity/http-xss-probing,enabled,0.4,A scenario that detects XSS probing with minimal false positives,scenarios crowdsecurity/jira_cve-2021-26086,enabled,0.3,Detect Atlassian Jira CVE-2021-26086 exploitation attemps,scenarios crowdsecurity/netgear_rce,enabled,0.3,Detect Netgear RCE DGN1000/DGN220 exploitation attempts,scenarios crowdsecurity/nextcloud-bf,enabled,0.3,Detect Nextcloud bruteforce,scenarios crowdsecurity/pulse-secure-sslvpn-cve-2019-11510,enabled,0.3,Detect cve-2019-11510 exploitation attemps,scenarios crowdsecurity/spring4shell_cve-2022-22965,enabled,0.3,Detect cve-2022-22965 probing,scenarios crowdsecurity/ssh-bf,enabled,0.3,Detect ssh bruteforce,scenarios crowdsecurity/ssh-cve-2024-6387,enabled,0.2,Detect exploitation attempt of CVE-2024-6387,scenarios crowdsecurity/ssh-slow-bf,enabled,0.4,Detect slow ssh bruteforce,scenarios crowdsecurity/thinkphp-cve-2018-20062,enabled,0.6,Detect ThinkPHP CVE-2018-20062 exploitation attemps,scenarios crowdsecurity/vmware-cve-2022-22954,enabled,0.3,Detect Vmware CVE-2022-22954 exploitation attempts,scenarios crowdsecurity/vmware-vcenter-vmsa-2021-0027,enabled,0.2,Detect VMSA-2021-0027 exploitation attemps,scenarios Dominic-Wagner/vaultwarden-bf,enabled,0.2,Detect vaultwarden bruteforce,scenarios ltsich/http-w00tw00t,enabled,0.2,detect w00tw00t,scenarios crowdsecurity/appsec_base,enabled,0.2,,contexts crowdsecurity/bf_base,enabled,0.1,,contexts crowdsecurity/http_base,enabled,0.2,,contexts crowdsecurity/appsec-default,enabled,0.2,,appsec-configs crowdsecurity/virtual-patching,enabled,0.4,,appsec-configs crowdsecurity/base-config,enabled,0.1,,appsec-rules crowdsecurity/vpatch-connectwise-auth-bypass,enabled,0.3,Detect exploitation of auth bypass in ConnectWise ScreenConnect,appsec-rules crowdsecurity/vpatch-CVE-2017-9841,enabled,0.3,PHPUnit RCE (CVE-2017-9841),appsec-rulescrowdsecurity/vpatch-CVE-2018-1000861,enabled,0.1,Jenkins - RCE (CVE-2018-1000861),appsec-rules crowdsecurity/vpatch-CVE-2018-10562,enabled,0.2,Dasan GPON RCE (CVE-2018-10562),appsec-rules crowdsecurity/vpatch-CVE-2018-13379,enabled,0.2,Fortinet FortiOS - Credentials Disclosure (CVE-2018-13379),appsec-rules crowdsecurity/vpatch-CVE-2019-1003030,enabled,0.1,Jenkins - RCE (CVE-2019-1003030),appsec-rules crowdsecurity/vpatch-CVE-2019-12989,enabled,0.3,Citrix SQLi (CVE-2019-12989),appsec-rules crowdsecurity/vpatch-CVE-2019-18935,enabled,0.1,Telerik - RCE (CVE-2019-18935),appsec-rules crowdsecurity/vpatch-CVE-2020-11738,enabled,0.6,Wordpress Snap Creek Duplicator - Path Traversal (CVE-2020-11738),appsec-rules crowdsecurity/vpatch-CVE-2020-17496,enabled,0.1,vBulletin RCE (CVE-2020-17496),appsec-rules crowdsecurity/vpatch-CVE-2020-5902,enabled,0.1,F5 BIG-IP TMUI - RCE (CVE-2020-5902),appsec-rules crowdsecurity/vpatch-CVE-2021-22941,enabled,0.3,Citrix RCE (CVE-2021-22941),appsec-rules crowdsecurity/vpatch-CVE-2021-3129,enabled,0.4,Laravel with Ignition Debug Mode RCE (CVE-2021-3129),appsec-rules crowdsecurity/vpatch-CVE-2022-22954,enabled,0.2,VMWare Workspace ONE Access RCE (CVE-2022-22954),appsec-rules crowdsecurity/vpatch-CVE-2022-22965,enabled,0.2,Spring4Shell - RCE (CVE-2022-22965),appsec-rules crowdsecurity/vpatch-CVE-2022-26134,enabled,0.2,Confluence - RCE (CVE-2022-26134),appsec-rules crowdsecurity/vpatch-CVE-2022-27926,enabled,0.4,Zimbra Collaboration XSS (CVE-2022-27926),appsec-rules crowdsecurity/vpatch-CVE-2022-35914,enabled,0.5,GLPI RCE (CVE-2022-35914),appsec-rules crowdsecurity/vpatch-CVE-2022-41082,enabled,0.1,Microsoft Exchange - RCE (CVE-2022-41082),appsec-rules crowdsecurity/vpatch-CVE-2022-44877,enabled,0.2,CentOS Web Panel 7 RCE (CVE-2022-44877),appsec-rules crowdsecurity/vpatch-CVE-2022-46169,enabled,0.5,Cacti RCE (CVE-2022-46169),appsec-rulescrowdsecurity/vpatch-CVE-2023-1389,enabled,0.1,TP-Link Archer AX21 - RCE (CVE-2023-1389),appsec-rules crowdsecurity/vpatch-CVE-2023-20198,enabled,0.6,CISCO IOS XE Account Creation (CVE-2023-20198),appsec-rules crowdsecurity/vpatch-CVE-2023-22515,enabled,0.4,Atlassian Confluence Privesc (CVE-2023-22515),appsec-rules crowdsecurity/vpatch-CVE-2023-22527,enabled,0.2,RCE using SSTI in Confluence (CVE-2023-22527),appsec-rules crowdsecurity/vpatch-CVE-2023-23752,enabled,0.1,Joomla! Webservice - Password Disclosure (CVE-2023-23752),appsec-rules crowdsecurity/vpatch-CVE-2023-24489,enabled,0.2,Citrix ShareFile RCE (CVE-2023-24489),appsec-rules crowdsecurity/vpatch-CVE-2023-28121,enabled,0.1,WooCommerce auth bypass (CVE-2023-28121),appsec-rules crowdsecurity/vpatch-CVE-2023-33617,enabled,0.4,Atlassian Confluence Privesc (CVE-2023-33617),appsec-rules crowdsecurity/vpatch-CVE-2023-34362,enabled,0.6,MOVEit Transfer RCE (CVE-2023-34362),appsec-rules crowdsecurity/vpatch-CVE-2023-35078,enabled,0.1,MobileIron Core Remote Unauthenticated API Access (CVE-2023-35078),appsec-rules crowdsecurity/vpatch-CVE-2023-35082,enabled,0.2,MobileIron Core Remote Unauthenticated API Access (CVE-2023-35082),appsec-rules crowdsecurity/vpatch-CVE-2023-3519,enabled,0.3,Citrix RCE (CVE-2023-3519),appsec-rules crowdsecurity/vpatch-CVE-2023-38205,enabled,0.3,Adobe ColdFusion Access Control Bypass (CVE-2023-38205),appsec-rules crowdsecurity/vpatch-CVE-2023-40044,enabled,0.3,WS_FTP .NET deserialize RCE (CVE-2023-40044),appsec-rules crowdsecurity/vpatch-CVE-2023-42793,enabled,0.3,JetBrains Teamcity Auth Bypass (CVE-2023-42793),appsec-rules crowdsecurity/vpatch-CVE-2023-46805,enabled,0.4,Ivanti Connect Auth Bypass (CVE-2023-46805),appsec-rules crowdsecurity/vpatch-CVE-2023-47218,enabled,0.2,QNAP QTS - RCE (CVE-2023-47218),appsec-rules crowdsecurity/vpatch-CVE-2023-49070,enabled,0.1,Apache OFBiz - RCE (CVE-2023-49070),appsec-rules crowdsecurity/vpatch-CVE-2023-50164,enabled,0.6,Apache Struts2 Path Traversal (CVE-2023-50164),appsec-rules crowdsecurity/vpatch-CVE-2023-6553,enabled,0.1,Backup Migration plugin for WordPress RCE (CVE-2023-6553),appsec-rules crowdsecurity/vpatch-CVE-2023-7028,enabled,0.2,Gitlab Password Reset Account Takeover (CVE-2023-7028),appsec-rules crowdsecurity/vpatch-CVE-2024-1212,enabled,0.3,Progress Kemp LoadMaster Unauthenticated Command Injection (CVE-2024-1212),appsec-rules crowdsecurity/vpatch-CVE-2024-22024,enabled,0.1,Ivanti Connect Secure - XXE (CVE-2024-22024),appsec-rules crowdsecurity/vpatch-CVE-2024-23897,enabled,0.4,Jenkins CLI RCE (CVE-2024-23897),appsec-rules crowdsecurity/vpatch-CVE-2024-27198,enabled,0.5,Teamcity - Authentication Bypass (CVE-2024-27198),appsec-rules crowdsecurity/vpatch-CVE-2024-27348,enabled,0.1,Apache HugeGraph-Server - RCE (CVE-2024-27348),appsec-rules crowdsecurity/vpatch-CVE-2024-28255,enabled,0.1,OpenMetadata - Authentication Bypass (CVE-2024-28255),appsec-rules crowdsecurity/vpatch-CVE-2024-29824,enabled,0.1,Ivanti EPM - SQLi (CVE-2024-29824),appsec-rules crowdsecurity/vpatch-CVE-2024-29849,enabled,0.5,Veeam Backup Enterprise Manager - Authentication Bypass (CVE-2024-29849),appsec-rules crowdsecurity/vpatch-CVE-2024-29973,enabled,0.1,Zyxel - RCE (CVE-2024-29973),appsec-rules crowdsecurity/vpatch-CVE-2024-32113,enabled,0.1,Apache OFBiz - Path Traversal (CVE-2024-32113),appsec-rules crowdsecurity/vpatch-CVE-2024-3272,enabled,0.1," D-Link NAS - RCE (CVE-2024-3272)",appsec-rules crowdsecurity/vpatch-CVE-2024-3273,enabled,0.1,D-LINK NAS Command Injection (CVE-2024-3273),appsec-rules crowdsecurity/vpatch-CVE-2024-34102,enabled,0.1,Adobe Commerce & Magento - XXE (CVE-2024-34102),appsec-rules crowdsecurity/vpatch-CVE-2024-4577,enabled,0.1,PHP CGI Command Injection - CVE-2024-4577,appsec-rules crowdsecurity/vpatch-CVE-2024-8190,enabled,0.1,Ivanti Cloud Services Appliance - RCE (CVE-2024-8190),appsec-rules crowdsecurity/vpatch-env-access,enabled,0.1,Detect access to .env files,appsec-rules crowdsecurity/vpatch-git-config,enabled,0.2,Detect access to .git files,appsec-rules crowdsecurity/vpatch-laravel-debug-mode,enabled,0.3,Detect bots exploiting laravel debug mode,appsec-rules crowdsecurity/vpatch-symfony-profiler,enabled,0.1,Detect abuse of symfony profiler,appsec-rules crowdsecurity/appsec-virtual-patching,enabled,4.0,"a generic virtual patching collection, suitable for most web servers.",collections crowdsecurity/base-http-scenarios,enabled,1.0,http common : scanners detection,collections crowdsecurity/http-cve,enabled,2.7,Detect CVE exploitation in http logs,collections crowdsecurity/linux,enabled,0.2,core linux support : syslog+geoip+ssh,collections crowdsecurity/nextcloud,enabled,0.3,Nextcloud support : parser and brute-force detection,collections crowdsecurity/sshd,enabled,0.5,sshd support : parser and brute-force detection,collections crowdsecurity/traefik,enabled,0.1,traefik support: parser and generic http scenarios,collections Dominic-Wagner/vaultwarden,enabled,0.1,Vaultwarden support : parser and brute-force detection,collections

Acquisition config

```console # On Linux: $ cat /etc/crowdsec/acquis.yaml /etc/crowdsec/acquis.d/* # paste output here # On Windows: C:\> Get-Content C:\ProgramData\CrowdSec\config\acquis.yaml # paste output here

Config show

```console $ cscli config show # paste output here ```

Prometheus metrics

```console $ cscli metrics # paste output here ```

Related custom configs versions (if applicable) : notification plugins, custom scenarios, parsers etc.

github-actions[bot] commented 4 weeks ago

@priyankub: Thanks for opening an issue, it is currently awaiting triage.

In the meantime, you can:

  1. Check Crowdsec Documentation to see if your issue can be self resolved.
  2. You can also join our Discord.
  3. Check Releases to make sure your agent is on the latest version.
Details I am a bot created to help the [crowdsecurity](https://github.com/crowdsecurity) developers manage community feedback and contributions. You can check out my [manifest file](https://github.com/crowdsecurity/crowdsec/blob/master/.github/governance.yml) to understand my behavior and what I can do. If you want to use this for your project, you can check out the [BirthdayResearch/oss-governance-bot](https://github.com/BirthdayResearch/oss-governance-bot) repository.
priyankub commented 4 weeks ago

cat crowdsec-crash.1254139945.txt error: runtime error: invalid memory address or nil pointer dereference version: v1.6.3-4851945a BuildDate: 2024-09-12_09:37:12 GoVersion: 1.22.6 Platform: docker goroutine 189 [running]: runtime/debug.Stack() runtime/debug/stack.go:24 +0x5e github.com/crowdsecurity/go-cs-lib/trace.(*traceKeeper).writeStackTrace(0x382a180, {0x1e6c060, 0x3808360}) github.com/crowdsecurity/go-cs-lib@v0.0.15/trace/trace.go:152 +0x173 github.com/crowdsecurity/go-cs-lib/trace.(*traceKeeper).catchPanic(0x382a180, {0x21676aa, 0x17}) github.com/crowdsecurity/go-cs-lib@v0.0.15/trace/trace.go:168 +0x13d github.com/crowdsecurity/go-cs-lib/trace.CatchPanic(...) github.com/crowdsecurity/go-cs-lib@v0.0.15/trace/trace.go:37 panic({0x1e6c060?, 0x3808360?}) runtime/panic.go:770 +0x132 github.com/crowdsecurity/crowdsec/pkg/apiserver.(*apic).SendUsageMetrics(0xc000aea000) github.com/crowdsecurity/crowdsec/pkg/apiserver/apic_metrics.go:371 +0x28e github.com/crowdsecurity/crowdsec/pkg/apiserver.(*APIServer).Run.func6() github.com/crowdsecurity/crowdsec/pkg/apiserver/apiserver.go:373 +0x1b gopkg.in/tomb%2ev2.(*Tomb).run(0xc000aea0f0, 0x0?) gopkg.in/tomb.v2@v2.0.0-20161208151619-d5d1b5820637/tomb.go:163 +0x2b created by gopkg.in/tomb%2ev2.(*Tomb).Go in goroutine 155 gopkg.in/tomb.v2@v2.0.0-20161208151619-d5d1b5820637/tomb.go:159 +0xe5

priyankub commented 4 weeks ago

cat crowdsec-crash.3536834419.txt error: runtime error: invalid memory address or nil pointer dereference version: v1.6.3-4851945a BuildDate: 2024-09-12_09:37:12 GoVersion: 1.22.6 Platform: docker goroutine 203 [running]: runtime/debug.Stack() runtime/debug/stack.go:24 +0x5e github.com/crowdsecurity/go-cs-lib/trace.(*traceKeeper).writeStackTrace(0x382a180, {0x1e6c060, 0x3808360}) github.com/crowdsecurity/go-cs-lib@v0.0.15/trace/trace.go:152 +0x173 github.com/crowdsecurity/go-cs-lib/trace.(*traceKeeper).catchPanic(0x382a180, {0x21676aa, 0x17}) github.com/crowdsecurity/go-cs-lib@v0.0.15/trace/trace.go:168 +0x13d github.com/crowdsecurity/go-cs-lib/trace.CatchPanic(...) github.com/crowdsecurity/go-cs-lib@v0.0.15/trace/trace.go:37 panic({0x1e6c060?, 0x3808360?}) runtime/panic.go:770 +0x132 github.com/crowdsecurity/crowdsec/pkg/apiserver.(*apic).SendUsageMetrics(0xc000b8e000) github.com/crowdsecurity/crowdsec/pkg/apiserver/apic_metrics.go:371 +0x28e github.com/crowdsecurity/crowdsec/pkg/apiserver.(*APIServer).Run.func6() github.com/crowdsecurity/crowdsec/pkg/apiserver/apiserver.go:373 +0x1b gopkg.in/tomb%2ev2.(*Tomb).run(0xc000b8e0f0, 0x0?) gopkg.in/tomb.v2@v2.0.0-20161208151619-d5d1b5820637/tomb.go:163 +0x2b created by gopkg.in/tomb%2ev2.(*Tomb).Go in goroutine 128 gopkg.in/tomb.v2@v2.0.0-20161208151619-d5d1b5820637/tomb.go:159 +0xe5

priyankub commented 4 weeks ago

error: runtime error: invalid memory address or nil pointer dereference version: v1.6.3-4851945a BuildDate: 2024-09-12_09:37:12 GoVersion: 1.22.6 Platform: docker goroutine 169 [running]: runtime/debug.Stack() runtime/debug/stack.go:24 +0x5e github.com/crowdsecurity/go-cs-lib/trace.(traceKeeper).writeStackTrace(0x382a180, {0x1e6c060, 0x3808360}) github.com/crowdsecurity/go-cs-lib@v0.0.15/trace/trace.go:152 +0x173 github.com/crowdsecurity/go-cs-lib/trace.(traceKeeper).catchPanic(0x382a180, {0x21676aa, 0x17}) github.com/crowdsecurity/go-cs-lib@v0.0.15/trace/trace.go:168 +0x13d github.com/crowdsecurity/go-cs-lib/trace.CatchPanic(...) github.com/crowdsecurity/go-cs-lib@v0.0.15/trace/trace.go:37 panic({0x1e6c060?, 0x3808360?}) runtime/panic.go:770 +0x132 github.com/crowdsecurity/crowdsec/pkg/apiserver.(apic).SendUsageMetrics(0xc0009faa80) github.com/crowdsecurity/crowdsec/pkg/apiserver/apic_metrics.go:371 +0x28e github.com/crowdsecurity/crowdsec/pkg/apiserver.(APIServer).Run.func6() github.com/crowdsecurity/crowdsec/pkg/apiserver/apiserver.go:373 +0x1b gopkg.in/tomb%2ev2.(Tomb).run(0xc0009fab70, 0xc000810000?) gopkg.in/tomb.v2@v2.0.0-20161208151619-d5d1b5820637/tomb.go:163 +0x2b created by gopkg.in/tomb%2ev2.(Tomb).Go in goroutine 179 gopkg.in/tomb.v2@v2.0.0-20161208151619-d5d1b5820637/tomb.go:159 +0xe5

priyankub commented 4 weeks ago

Command from inside the container to test reachability to api.crowdsec.net

ad90c0accb7c:/# cscli version
version: v1.6.3-4851945a
Codename: alphaga
BuildDate: 2024-09-12_09:39:08
GoVersion: 1.22.6
Platform: docker
libre2: C++
User-Agent: crowdsec/v1.6.3-4851945a-docker
Constraint_parser: >= 1.0, <= 3.0
Constraint_scenario: >= 1.0, <= 3.0
Constraint_api: v1
Constraint_acquis: >= 1.0, < 2.0
ad90c0accb7c:/# ping api.crowdsec.net
PING api.crowdsec.net (18.245.96.48): 56 data bytes
64 bytes from 18.245.96.48: seq=0 ttl=247 time=19.355 ms
64 bytes from 18.245.96.48: seq=1 ttl=247 time=22.070 ms
64 bytes from 18.245.96.48: seq=2 ttl=247 time=15.487 ms
^C
--- api.crowdsec.net ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 15.487/18.970/22.070 ms
LaurenceJJones commented 4 weeks ago

Hey 👋🏻

Thank you for the report and I can see the nil ref straight away.

https://github.com/crowdsecurity/crowdsec/blob/d8bc17b17009467945dd47c9b90d061bbf6de899/pkg/apiclient/usagemetrics.go#L16-L19

@blotus @mmetc

The error returns but the resp is nil, on the upper scope we check the resp status but its a nil object.

https://github.com/crowdsecurity/crowdsec/blob/d8bc17b17009467945dd47c9b90d061bbf6de899/pkg/apiserver/apic_metrics.go#L371-L375

LaurenceJJones commented 4 weeks ago

Fix has been merged, however, wont be released to GA until 1.6.4 if you wish to use the fix for now you can point the docker tag to :dev for now.

priyankub commented 1 week ago

Hey, it crashed again recently. Here's the trace

error: runtime error: invalid memory address or nil pointer dereference version: v1.6.3-rc4-83-g5d414f58 BuildDate: 2024-11-08_14:13:15 GoVersion: 1.23.3 Platform: docker goroutine 254 [running]: runtime/debug.Stack() runtime/debug/stack.go:26 +0x5e github.com/crowdsecurity/go-cs-lib/trace.(*traceKeeper).writeStackTrace(0x35fa580, {0x1eb7bc0, 0x35d8400}) github.com/crowdsecurity/go-cs-lib@v0.0.15/trace/trace.go:152 +0x16e github.com/crowdsecurity/go-cs-lib/trace.(*traceKeeper).catchPanic(0x35fa580, {0x21bdfe0, 0x17}) github.com/crowdsecurity/go-cs-lib@v0.0.15/trace/trace.go:168 +0x134 github.com/crowdsecurity/go-cs-lib/trace.CatchPanic(...) github.com/crowdsecurity/go-cs-lib@v0.0.15/trace/trace.go:37 panic({0x1eb7bc0?, 0x35d8400?}) runtime/panic.go:785 +0x132 github.com/crowdsecurity/crowdsec/pkg/apiserver.(*apic).SendUsageMetrics(0xc000d66000, {0x25ebeb8, 0x36d95e0}) github.com/crowdsecurity/crowdsec/pkg/apiserver/apic_metrics.go:371 +0x2be github.com/crowdsecurity/crowdsec/pkg/apiserver.(*APIServer).initAPIC.func5() github.com/crowdsecurity/crowdsec/pkg/apiserver/apiserver.go:364 +0x25 gopkg.in/tomb%2ev2.(*Tomb).run(0xc000d660f0, 0x544547223a226575?) gopkg.in/tomb.v2@v2.0.0-20161208151619-d5d1b5820637/tomb.go:163 +0x2b created by gopkg.in/tomb%2ev2.(*Tomb).Go in goroutine 243 gopkg.in/tomb.v2@v2.0.0-20161208151619-d5d1b5820637/tomb.go:159 +0xdb

LaurenceJJones commented 1 week ago

Hey, it crashed again recently. Here's the trace

error: runtime error: invalid memory address or nil pointer dereference version: v1.6.3-rc4-83-g5d414f58 BuildDate: 2024-11-08_14:13:15 GoVersion: 1.23.3 Platform: docker goroutine 254 [running]: runtime/debug.Stack() runtime/debug/stack.go:26 +0x5e github.com/crowdsecurity/go-cs-lib/trace.(*traceKeeper).writeStackTrace(0x35fa580, {0x1eb7bc0, 0x35d8400}) github.com/crowdsecurity/go-cs-lib@v0.0.15/trace/trace.go:152 +0x16e github.com/crowdsecurity/go-cs-lib/trace.(*traceKeeper).catchPanic(0x35fa580, {0x21bdfe0, 0x17}) github.com/crowdsecurity/go-cs-lib@v0.0.15/trace/trace.go:168 +0x134 github.com/crowdsecurity/go-cs-lib/trace.CatchPanic(...) github.com/crowdsecurity/go-cs-lib@v0.0.15/trace/trace.go:37 panic({0x1eb7bc0?, 0x35d8400?}) runtime/panic.go:785 +0x132 github.com/crowdsecurity/crowdsec/pkg/apiserver.(*apic).SendUsageMetrics(0xc000d66000, {0x25ebeb8, 0x36d95e0}) github.com/crowdsecurity/crowdsec/pkg/apiserver/apic_metrics.go:371 +0x2be github.com/crowdsecurity/crowdsec/pkg/apiserver.(*APIServer).initAPIC.func5() github.com/crowdsecurity/crowdsec/pkg/apiserver/apiserver.go:364 +0x25 gopkg.in/tomb%2ev2.(*Tomb).run(0xc000d660f0, 0x544547223a226575?) gopkg.in/tomb.v2@v2.0.0-20161208151619-d5d1b5820637/tomb.go:163 +0x2b created by gopkg.in/tomb%2ev2.(*Tomb).Go in goroutine 243 gopkg.in/tomb.v2@v2.0.0-20161208151619-d5d1b5820637/tomb.go:159 +0xdb

From the panic output you are still using an older version. If you are running on docker can you pull the latest :dev image and redeploy.

priyankub commented 1 week ago

That's what it's pulling in :dev With:latest, it runs on 1.6.3 built in September, not the November build in my trace log

This is running latest : cscli version version: v1.6.3-4851945a Codename: alphaga BuildDate: 2024-09-12_09:39:08 GoVersion: 1.22.6 Platform: docker libre2: C++ User-Agent: crowdsec/v1.6.3-4851945a-docker Constraint_parser: >= 1.0, <= 3.0 Constraint_scenario: >= 1.0, <= 3.0 Constraint_api: v1 Constraint_acquis: >= 1.0, < 2.0

This is running:dev

cscli version version: v1.6.3-rc4-83-g5d414f58 Codename: alphaga BuildDate: 2024-11-08_14:14:15 GoVersion: 1.23.3 Platform: docker libre2: C++ User-Agent: crowdsec/v1.6.3-rc4-83-g5d414f58-docker Constraint_parser: >= 1.0, <= 3.0 Constraint_scenario: >= 1.0, <= 3.0 Constraint_api: v1 Constraint_acquis: >= 1.0, < 2.0 Built-in optional components: cscli_setup, datasource_appsec, datasource_cloudwatch, datasource_docker, datasource_file, datasource_http, datasource_journalctl, datasource_k8s-audit, datasource_kafka, datasource_kinesis, datasource_loki, datasource_s3, datasource_syslog, datasource_wineventlog