search

crowdsecurity / crowdsec

CrowdSec - the open-source and participative security solution offering crowdsourced protection against malicious IPs and access to the most advanced real-world CTI.
https://crowdsec.net
MIT License
9.14k stars 472 forks source link

Inappropriate parsing pattern for Email #3205

Closed intweed closed 2 months ago

intweed commented 3 months ago

What happened?

The official documentation specifies the following pattern for email parsing:

%{EMAILLOCALPART}@%{HOSTNAME}

where EMAILLOCALPART has the following regex pattern:

[a-zA-Z][a-zA-Z0-9_.+-=:]+

which does not include email addresses starting with a digits (for example, these are acceptable in Gmail). In this case, the parser cannot correctly process the log message and the corresponding scenario is not activated, no decisions are applied.

What did you expect to happen?

Crowdsec correctly processes log messages containing email addresses starting with digits.

How can we reproduce it (as minimally and precisely as possible)?

The above is observed on this configuration:

Crowdsec + Dominic-Wagner/vaultwarden collection + vaultwarden server (logs are collected via docker socket proxy)

Multiple login attempts with wrong credentials (where email address starting with digit) are not registered by Crowdsec. cscli metrics show acquisition displays the same value for "Lines read" and "Lines unparsed" for docker:/vaultwarden source. Thus, the bruteforce scenario does not apply.

Anything else we need to know?

No response

Crowdsec version

```console # cscli version version: v1.6.2-16bfab86 Codename: alphaga BuildDate: 2024-06-05_14:25:55 GoVersion: 1.22.3 Platform: docker libre2: C++ User-Agent: crowdsec/v1.6.2-16bfab86-docker Constraint_parser: >= 1.0, <= 3.0 Constraint_scenario: >= 1.0, <= 3.0 Constraint_api: v1 Constraint_acquis: >= 1.0, < 2.0 ```

OS version

Crowdsec runs in a Docker container.

Enabled collections and parsers

```console $ cscli hub list -o raw name,status,version,description,type crowdsecurity/cri-logs,enabled,0.1,CRI logging format parser,parsers crowdsecurity/dateparse-enrich,enabled,0.2,,parsers crowdsecurity/docker-logs,enabled,0.1,docker json logs parser,parsers crowdsecurity/geoip-enrich,enabled,0.4,"Populate event with geoloc info : as, country, coords, source range.",parsers crowdsecurity/http-logs,enabled,1.2,"Parse more Specifically HTTP logs, such as HTTP Code, HTTP path, HTTP args and if its a static ressource",parsers crowdsecurity/nginx-logs,enabled,1.6,Parse nginx access and error logs,parsers crowdsecurity/sshd-logs,enabled,2.8,Parse openSSH logs,parsers crowdsecurity/syslog-logs,enabled,0.8,,parsers Dominic-Wagner/vaultwarden-logs,enabled,0.1,Parse vaultwarden logs,parsers crowdsecurity/apache_log4j2_cve-2021-44228,enabled,0.6,Detect cve-2021-44228 exploitation attemps,scenarios crowdsecurity/CVE-2017-9841,enabled,0.2,Detect CVE-2017-9841 exploits,scenarios crowdsecurity/CVE-2019-18935,enabled,0.2,Detect Telerik CVE-2019-18935 exploitation attempts,scenarios crowdsecurity/CVE-2022-26134,enabled,0.2,Detect CVE-2022-26134 exploits,scenarios crowdsecurity/CVE-2022-35914,enabled,0.2,Detect CVE-2022-35914 exploits,scenarios crowdsecurity/CVE-2022-37042,enabled,0.2,Detect CVE-2022-37042 exploits,scenarios crowdsecurity/CVE-2022-40684,enabled,0.3,Detect cve-2022-40684 exploitation attempts,scenarios crowdsecurity/CVE-2022-41082,enabled,0.4,Detect CVE-2022-41082 exploits,scenarios crowdsecurity/CVE-2022-41697,enabled,0.2,Detect CVE-2022-41697 enumeration,scenarios crowdsecurity/CVE-2022-42889,enabled,0.3,Detect CVE-2022-42889 exploits (Text4Shell),scenarios crowdsecurity/CVE-2022-44877,enabled,0.3,Detect CVE-2022-44877 exploits,scenarios crowdsecurity/CVE-2022-46169,enabled,0.2,Detect CVE-2022-46169 brute forcing,scenarios crowdsecurity/CVE-2023-22515,enabled,0.1,Detect CVE-2023-22515 exploitation,scenarios crowdsecurity/CVE-2023-22518,enabled,0.2,Detect CVE-2023-22518 exploits,scenarios crowdsecurity/CVE-2023-49103,enabled,0.3,Detect owncloud CVE-2023-49103 exploitation attempts,scenarios crowdsecurity/CVE-2024-38475,enabled,0.1,Detect CVE-2024-38475 exploitation attempts,scenarios crowdsecurity/f5-big-ip-cve-2020-5902,enabled,0.2,Detect cve-2020-5902 exploitation attemps,scenarios crowdsecurity/fortinet-cve-2018-13379,enabled,0.3,Detect cve-2018-13379 exploitation attemps,scenarios crowdsecurity/grafana-cve-2021-43798,enabled,0.2,Detect cve-2021-43798 exploitation attemps,scenarios crowdsecurity/http-admin-interface-probing,enabled,0.4,Detect generic HTTP admin interface probing,scenarios crowdsecurity/http-backdoors-attempts,enabled,0.6,Detect attempt to common backdoors,scenarios crowdsecurity/http-bad-user-agent,enabled,1.2,Detect usage of bad User Agent,scenarios crowdsecurity/http-crawl-non_statics,enabled,0.7,Detect aggressive crawl on non static resources,scenarios crowdsecurity/http-cve-2021-41773,enabled,0.2,cve-2021-41773,scenarios crowdsecurity/http-cve-2021-42013,enabled,0.2,cve-2021-42013,scenarios crowdsecurity/http-cve-probing,enabled,0.2,Detect generic HTTP cve probing,scenarios crowdsecurity/http-generic-bf,enabled,0.6,Detect generic http brute force,scenarios crowdsecurity/http-open-proxy,enabled,0.5,Detect scan for open proxy,scenarios crowdsecurity/http-path-traversal-probing,enabled,0.4,Detect path traversal attempt,scenarios crowdsecurity/http-probing,enabled,0.4,Detect site scanning/probing from a single ip,scenarios crowdsecurity/http-sensitive-files,enabled,0.4,"Detect attempt to access to sensitive files (.log, .db ..) or folders (.git)",scenarios crowdsecurity/http-sqli-probing,enabled,0.4,A scenario that detects SQL injection probing with minimal false positives,scenarios crowdsecurity/http-wordpress-scan,enabled,0.2,Detect WordPress scan: vuln hunting,scenarios crowdsecurity/http-xss-probing,enabled,0.4,A scenario that detects XSS probing with minimal false positives,scenarios crowdsecurity/jira_cve-2021-26086,enabled,0.3,Detect Atlassian Jira CVE-2021-26086 exploitation attemps,scenarios crowdsecurity/netgear_rce,enabled,0.3,Detect Netgear RCE DGN1000/DGN220 exploitation attempts,scenarios crowdsecurity/nginx-req-limit-exceeded,enabled,0.3,Detects IPs which violate nginx's user set request limit.,scenarios crowdsecurity/pulse-secure-sslvpn-cve-2019-11510,enabled,0.3,Detect cve-2019-11510 exploitation attemps,scenarios crowdsecurity/spring4shell_cve-2022-22965,enabled,0.3,Detect cve-2022-22965 probing,scenarios crowdsecurity/ssh-bf,enabled,0.3,Detect ssh bruteforce,scenarios crowdsecurity/ssh-cve-2024-6387,enabled,0.2,Detect exploitation attempt of CVE-2024-6387,scenarios crowdsecurity/ssh-slow-bf,enabled,0.4,Detect slow ssh bruteforce,scenarios crowdsecurity/thinkphp-cve-2018-20062,enabled,0.6,Detect ThinkPHP CVE-2018-20062 exploitation attemps,scenarios crowdsecurity/vmware-cve-2022-22954,enabled,0.3,Detect Vmware CVE-2022-22954 exploitation attempts,scenarios crowdsecurity/vmware-vcenter-vmsa-2021-0027,enabled,0.2,Detect VMSA-2021-0027 exploitation attemps,scenarios Dominic-Wagner/vaultwarden-bf,enabled,0.2,Detect vaultwarden bruteforce,scenarios ltsich/http-w00tw00t,enabled,0.2,detect w00tw00t,scenarios crowdsecurity/bf_base,enabled,0.1,,contexts crowdsecurity/http_base,enabled,0.2,,contexts crowdsecurity/base-http-scenarios,enabled,1.0,http common : scanners detection,collections crowdsecurity/http-cve,enabled,2.7,Detect CVE exploitation in http logs,collections crowdsecurity/linux,enabled,0.2,core linux support : syslog+geoip+ssh,collections crowdsecurity/nginx,enabled,0.2,nginx support : parser and generic http scenarios,collections crowdsecurity/sshd,enabled,0.5,sshd support : parser and brute-force detection,collections Dominic-Wagner/vaultwarden,enabled,0.1,Vaultwarden support : parser and brute-force detection,collections ```

Acquisition config

```console $ cat /etc/crowdsec/acquis.yaml /etc/crowdsec/acquis.d/* source: docker container_name_regexp: - .nginx labels: type: nginx --- source: docker container_name_regexp: - .vaultwarden labels: type: Vaultwarden cat: can't open '/etc/crowdsec/acquis.d/*': No such file or directory

Config show

```console $ cscli config show Global: - Configuration Folder : /etc/crowdsec - Data Folder : /var/lib/crowdsec/data - Hub Folder : /etc/crowdsec/hub - Simulation File : /etc/crowdsec/simulation.yaml - Log Folder : /var/log - Log level : info - Log Media : stdout Crowdsec: - Acquisition File : /etc/crowdsec/acquis.yaml - Parsers routines : 1 - Acquisition Folder : /etc/crowdsec/acquis.d cscli: - Output : human - Hub Branch : API Client: - URL : http://0.0.0.0:8080/ - Login : localhost - Credentials File : /etc/crowdsec/local_api_credentials.yaml Local API Server: - Listen URL : 0.0.0.0:8080 - Listen Socket : - Profile File : /etc/crowdsec/profiles.yaml - Trusted IPs: - 127.0.0.1 - ::1 - Database: - Type : sqlite - Path : /var/lib/crowdsec/data/crowdsec.db - Flush age : 7d - Flush size : 5000 ```

Prometheus metrics

```console $ cscli metrics Acquisition Metrics: ╭─────────────────────┬────────────┬──────────────┬────────────────┬────────────────────────┬───────────────────╮ │ Source │ Lines read │ Lines parsed │ Lines unparsed │ Lines poured to bucket │ Lines whitelisted │ ├─────────────────────┼────────────┼──────────────┼────────────────┼────────────────────────┼───────────────────┤ │ docker:/nginx │ 77 │ 68 │ 9 │ 13 │ - │ │ docker:/vaultwarden │ 119 │ - │ 119 │ - │ - │ ╰─────────────────────┴────────────┴──────────────┴────────────────┴────────────────────────┴───────────────────╯ Local API Metrics: ╭──────────────────────┬────────┬──────╮ │ Route │ Method │ Hits │ ├──────────────────────┼────────┼──────┤ │ /v1/decisions/stream │ GET │ 77 │ │ /v1/heartbeat │ GET │ 6 │ │ /v1/watchers/login │ POST │ 1 │ ╰──────────────────────┴────────┴──────╯ Local API Bouncers Metrics: ╭──────────┬──────────────────────┬────────┬──────╮ │ Bouncer │ Route │ Method │ Hits │ ├──────────┼──────────────────────┼────────┼──────┤ │ mikrotik │ /v1/decisions/stream │ GET │ 77 │ ╰──────────┴──────────────────────┴────────┴──────╯ Local API Machines Metrics: ╭───────────┬───────────────┬────────┬──────╮ │ Machine │ Route │ Method │ Hits │ ├───────────┼───────────────┼────────┼──────┤ │ localhost │ /v1/heartbeat │ GET │ 6 │ ╰───────────┴───────────────┴────────┴──────╯ Parser Metrics: ╭───────────────────────────────────────┬──────┬────────┬──────────╮ │ Parsers │ Hits │ Parsed │ Unparsed │ ├───────────────────────────────────────┼──────┼────────┼──────────┤ │ Dominic-Wagner/vaultwarden-logs │ 119 │ - │ 119 │ │ child-Dominic-Wagner/vaultwarden-logs │ 357 │ - │ 357 │ │ child-crowdsecurity/http-logs │ 204 │ 140 │ 64 │ │ child-crowdsecurity/nginx-logs │ 95 │ 68 │ 27 │ │ crowdsecurity/dateparse-enrich │ 68 │ 68 │ - │ │ crowdsecurity/geoip-enrich │ 68 │ 68 │ - │ │ crowdsecurity/http-logs │ 68 │ 68 │ - │ │ crowdsecurity/nginx-logs │ 77 │ 68 │ 9 │ │ crowdsecurity/non-syslog │ 196 │ 196 │ - │ ╰───────────────────────────────────────┴──────┴────────┴──────────╯ Scenario Metrics: ╭──────────────────────────────────────┬───────────────┬───────────┬──────────────┬────────┬─────────╮ │ Scenario │ Current Count │ Overflows │ Instantiated │ Poured │ Expired │ ├──────────────────────────────────────┼───────────────┼───────────┼──────────────┼────────┼─────────┤ │ crowdsecurity/http-crawl-non_statics │ - │ - │ 4 │ 12 │ 4 │ │ crowdsecurity/http-probing │ 1 │ - │ 1 │ 1 │ - │ ╰──────────────────────────────────────┴───────────────┴───────────┴──────────────┴────────┴─────────╯ ```

Related custom configs versions (if applicable) : notification plugins, custom scenarios, parsers etc.

github-actions[bot] commented 3 months ago

@intweed: Thanks for opening an issue, it is currently awaiting triage.

In the meantime, you can:

  1. Check Crowdsec Documentation to see if your issue can be self resolved.
  2. You can also join our Discord.
  3. Check Releases to make sure your agent is on the latest version.
Details I am a bot created to help the [crowdsecurity](https://github.com/crowdsecurity) developers manage community feedback and contributions. You can check out my [manifest file](https://github.com/crowdsecurity/crowdsec/blob/master/.github/governance.yml) to understand my behavior and what I can do. If you want to use this for your project, you can check out the [BirthdayResearch/oss-governance-bot](https://github.com/BirthdayResearch/oss-governance-bot) repository.
LaurenceJJones commented 3 months ago

Upstream issue: https://github.com/crowdsecurity/grokky/issues/3

Please read comment as to why it difficult to fix until the next release. In future we plan to remove all hard coded patterns and extend them to the normal patterns directory.

LaurenceJJones commented 2 months ago

Version: 1.6.3 is planned release for monday which will include this fix.