search

crowdsecurity / crowdsec

CrowdSec - the open-source and participative security solution offering crowdsourced protection against malicious IPs and access to the most advanced real-world CTI.
https://crowdsec.net
MIT License
9.09k stars 470 forks source link

Blacklisting 127.0.0.1 should be disabled by default #3090

Closed codiflow closed 5 months ago

codiflow commented 5 months ago

What happened?

I'm using crowdsec on many servers for more than a year now and there were no problems with blacklisting – until today. During the day parts of my server were unavailable and I couldn't find any reason for this.

Looking deeper into my crowdsec Telegram log I found out that Crowdsec blocked itself (127.0.0.1) and therefore prevented any action with cscli as those requests are sent to 127.0.0.1 as well:

image

From my perspective this should be prevented by default as it is very very unlikely that there is a real threat incoming from 127.0.0.1. This would mean the machine itself is not trustworthy anymore.

I found 127.0.0.1 being listed in /etc/crowdsec/hub/parsers/s02-enrich/crowdsecurity/whitelists.yaml so I really don't know how this could happen if the IP itself is listed there.

What did you expect to happen?

It is not possible that crowdsec blacklists 127.0.0.1 by default.

How can we reproduce it (as minimally and precisely as possible)?

The only services which send regular requests to 127.0.0.1 on my server are monit, nginx as reverse proxy and connections to the mysql server. So maybe triggering crowdsecurity/http-bad-user-agent on 127.0.0.1 somehow reproduces this?

Anything else we need to know?

I never had this issue ever before on one of my machines and they all share the same configuration.

I tried to temporarily "resolve" this by replacing all occurences of 127.0.0.1 in the configuration files with 127.0.1.1, restarting crowdsec and then trying to remove the 127.0.0.1 ban.

This was the output of cscli decisions list (it worked) image

Removal with cscli decisions delete -i 127.0.0.1 did not work: image

I needed to "chain" both commands to make the deletion of the decision possible: systemctl restart crowdsec.service crowdsec-firewall-bouncer.service ; cscli decisions delete -i 127.0.0.1

Crowdsec version

```console version: v1.6.2-debian-pragmatic-amd64-16bfab86 Codename: alphaga BuildDate: 2024-05-31_09:18:01 GoVersion: 1.22.2 Platform: linux libre2: C++ User-Agent: crowdsec/v1.6.2-debian-pragmatic-amd64-16bfab86-linux Constraint_parser: >= 1.0, <= 3.0 Constraint_scenario: >= 1.0, <= 3.0 Constraint_api: v1 Constraint_acquis: >= 1.0, < 2.0 ```

OS version

```console PRETTY_NAME="Debian GNU/Linux 11 (bullseye)" NAME="Debian GNU/Linux" VERSION_ID="11" VERSION="11 (bullseye)" VERSION_CODENAME=bullseye ID=debian HOME_URL="https://www.debian.org/" SUPPORT_URL="https://www.debian.org/support" BUG_REPORT_URL="https://bugs.debian.org/" Linux server 5.10.0-30-amd64 #1 SMP Debian 5.10.218-1 (2024-06-01) x86_64 GNU/Linux ```

Enabled collections and parsers

```console name,status,version,description,type crowdsecurity/apache2-logs,enabled,1.4,Parse Apache2 access and error logs,parsers crowdsecurity/dateparse-enrich,enabled,0.2,,parsers crowdsecurity/geoip-enrich,enabled,0.4,"Populate event with geoloc info : as, country, coords, source range.",parsers crowdsecurity/http-logs,enabled,1.2,"Parse more Specifically HTTP logs, such as HTTP Code, HTTP path, HTTP args and if its a static ressource",parsers crowdsecurity/mysql-logs,enabled,0.4,Parse MySQL logs,parsers crowdsecurity/nginx-logs,enabled,1.5,Parse nginx access and error logs,parsers crowdsecurity/sshd-logs,enabled,2.3,Parse openSSH logs,parsers crowdsecurity/syslog-logs,enabled,0.8,,parsers crowdsecurity/whitelists,"enabled,local",,,parsers crowdsecurity/seo-bots-whitelist,enabled,0.5,Whitelist good search engine crawlers,postoverflows crowdsecurity/apache_log4j2_cve-2021-44228,enabled,0.6,Detect cve-2021-44228 exploitation attemps,scenarios crowdsecurity/CVE-2017-9841,enabled,0.2,Detect CVE-2017-9841 exploits,scenarios crowdsecurity/CVE-2019-18935,enabled,0.2,Detect Telerik CVE-2019-18935 exploitation attempts,scenarios crowdsecurity/CVE-2022-26134,enabled,0.2,Detect CVE-2022-26134 exploits,scenarios crowdsecurity/CVE-2022-35914,enabled,0.2,Detect CVE-2022-35914 exploits,scenarios crowdsecurity/CVE-2022-37042,enabled,0.2,Detect CVE-2022-37042 exploits,scenarios crowdsecurity/CVE-2022-40684,enabled,0.3,Detect cve-2022-40684 exploitation attempts,scenarios crowdsecurity/CVE-2022-41082,enabled,0.4,Detect CVE-2022-41082 exploits,scenarios crowdsecurity/CVE-2022-41697,enabled,0.2,Detect CVE-2022-41697 enumeration,scenarios crowdsecurity/CVE-2022-42889,enabled,0.3,Detect CVE-2022-42889 exploits (Text4Shell),scenarios crowdsecurity/CVE-2022-44877,enabled,0.3,Detect CVE-2022-44877 exploits,scenarios crowdsecurity/CVE-2022-46169,enabled,0.2,Detect CVE-2022-46169 brute forcing,scenarios crowdsecurity/CVE-2023-22515,enabled,0.1,Detect CVE-2023-22515 exploitation,scenarios crowdsecurity/CVE-2023-22518,enabled,0.2,Detect CVE-2023-22518 exploits,scenarios crowdsecurity/CVE-2023-49103,enabled,0.3,Detect owncloud CVE-2023-49103 exploitation attempts,scenarios crowdsecurity/f5-big-ip-cve-2020-5902,enabled,0.2,Detect cve-2020-5902 exploitation attemps,scenarios crowdsecurity/fortinet-cve-2018-13379,enabled,0.3,Detect cve-2018-13379 exploitation attemps,scenarios crowdsecurity/grafana-cve-2021-43798,enabled,0.2,Detect cve-2021-43798 exploitation attemps,scenarios crowdsecurity/http-admin-interface-probing,enabled,0.4,Detect generic HTTP admin interface probing,scenarios crowdsecurity/http-backdoors-attempts,enabled,0.6,Detect attempt to common backdoors,scenarios crowdsecurity/http-bad-user-agent,enabled,1.2,Detect usage of bad User Agent,scenarios crowdsecurity/http-crawl-non_statics,enabled,0.7,Detect aggressive crawl on non static resources,scenarios crowdsecurity/http-cve-2021-41773,enabled,0.2,cve-2021-41773,scenarios crowdsecurity/http-cve-2021-42013,enabled,0.2,cve-2021-42013,scenarios crowdsecurity/http-cve-probing,enabled,0.2,Detect generic HTTP cve probing,scenarios crowdsecurity/http-generic-bf,enabled,0.6,Detect generic http brute force,scenarios crowdsecurity/http-open-proxy,enabled,0.5,Detect scan for open proxy,scenarios crowdsecurity/http-path-traversal-probing,enabled,0.4,Detect path traversal attempt,scenarios crowdsecurity/http-probing,enabled,0.4,Detect site scanning/probing from a single ip,scenarios crowdsecurity/http-sensitive-files,enabled,0.4,"Detect attempt to access to sensitive files (.log, .db ..) or folders (.git)",scenarios crowdsecurity/http-sqli-probing,enabled,0.4,A scenario that detects SQL injection probing with minimal false positives,scenarios crowdsecurity/http-wordpress-scan,enabled,0.2,Detect WordPress scan: vuln hunting,scenarios crowdsecurity/http-xss-probing,enabled,0.4,A scenario that detects XSS probing with minimal false positives,scenarios crowdsecurity/jira_cve-2021-26086,enabled,0.3,Detect Atlassian Jira CVE-2021-26086 exploitation attemps,scenarios crowdsecurity/mysql-bf,enabled,0.2,Detect mysql bruteforce,scenarios crowdsecurity/netgear_rce,enabled,0.3,Detect Netgear RCE DGN1000/DGN220 exploitation attempts,scenarios crowdsecurity/nginx-req-limit-exceeded,enabled,0.3,Detects IPs which violate nginx's user set request limit.,scenarios crowdsecurity/pulse-secure-sslvpn-cve-2019-11510,enabled,0.3,Detect cve-2019-11510 exploitation attemps,scenarios crowdsecurity/spring4shell_cve-2022-22965,enabled,0.3,Detect cve-2022-22965 probing,scenarios crowdsecurity/ssh-bf,enabled,0.3,Detect ssh bruteforce,scenarios crowdsecurity/ssh-slow-bf,enabled,0.4,Detect slow ssh bruteforce,scenarios crowdsecurity/thinkphp-cve-2018-20062,enabled,0.6,Detect ThinkPHP CVE-2018-20062 exploitation attemps,scenarios crowdsecurity/vmware-cve-2022-22954,enabled,0.3,Detect Vmware CVE-2022-22954 exploitation attempts,scenarios crowdsecurity/vmware-vcenter-vmsa-2021-0027,enabled,0.2,Detect VMSA-2021-0027 exploitation attemps,scenarios ltsich/http-w00tw00t,enabled,0.2,detect w00tw00t,scenarios crowdsecurity/bf_base,enabled,0.1,,contexts crowdsecurity/http_base,enabled,0.2,,contexts crowdsecurity/apache2,enabled,0.1,apache2 support : parser and generic http scenarios ,collections crowdsecurity/base-http-scenarios,enabled,1.0,http common : scanners detection,collections crowdsecurity/http-cve,enabled,2.6,Detect CVE exploitation in http logs,collections crowdsecurity/linux,enabled,0.2,core linux support : syslog+geoip+ssh,collections crowdsecurity/mysql,enabled,0.1,mysql support : logs and brute-force scenarios,collections crowdsecurity/nginx,enabled,0.2,nginx support : parser and generic http scenarios,collections crowdsecurity/sshd,enabled,0.3,sshd support : parser and brute-force detection,collections ```

Acquisition config

```console # On Linux: $ cat /etc/crowdsec/acquis.yaml /etc/crowdsec/acquis.d/* # paste output here # On Windows: C:\> Get-Content C:\ProgramData\CrowdSec\config\acquis.yaml # paste output here

Config show

```console Global: - Configuration Folder : /etc/crowdsec - Data Folder : /var/lib/crowdsec/data - Hub Folder : /etc/crowdsec/hub - Simulation File : /etc/crowdsec/simulation.yaml - Log Folder : /var/log - Log level : info - Log Media : file Crowdsec: - Acquisition File : /etc/crowdsec/acquis.yaml - Parsers routines : 1 - Acquisition Folder : /etc/crowdsec/acquis.d cscli: - Output : human - Hub Branch : API Client: - URL : http://127.0.0.1:8080/ - Login : REDACTED - Credentials File : /etc/crowdsec/local_api_credentials.yaml Local API Server: - Listen URL : 127.0.0.1:8080 - Listen Socket : - Profile File : /etc/crowdsec/profiles.yaml - Trusted IPs: - 127.0.0.1 - ::1 - Database: - Type : sqlite - Path : /var/lib/crowdsec/data/crowdsec.db - Flush age : 7d - Flush size : 5000 ```

Prometheus metrics

```console Not working because 127.0.0.1 is blocked... FATA failed to fetch metrics: executing GET request for URL "http://127.0.0.1:6060/metrics" failed: Get "http://127.0.0.1:6060/metrics": dial tcp 127.0.0.1:6060: i/o timeout Was also not working with my "127.0.1.1" fix: FATA failed to fetch metrics: executing GET request for URL "http://127.0.1.1:6060/metrics" failed: Get "http://127.0.1.1:6060/metrics": dial tcp 127.0.1.1:6060: i/o timeout ```

Related custom configs versions (if applicable) : notification plugins, custom scenarios, parsers etc.

Telegram notifications are activated
github-actions[bot] commented 5 months ago

@codiflow: Thanks for opening an issue, it is currently awaiting triage.

In the meantime, you can:

  1. Check Crowdsec Documentation to see if your issue can be self resolved.
  2. You can also join our Discord.
  3. Check Releases to make sure your agent is on the latest version.
Details I am a bot created to help the [crowdsecurity](https://github.com/crowdsecurity) developers manage community feedback and contributions. You can check out my [manifest file](https://github.com/crowdsecurity/crowdsec/blob/master/.github/governance.yml) to understand my behavior and what I can do. If you want to use this for your project, you can check out the [BirthdayResearch/oss-governance-bot](https://github.com/BirthdayResearch/oss-governance-bot) repository.
LaurenceJJones commented 5 months ago

The only services which send regular requests to 127.0.0.1 on my server are monit, nginx as reverse proxy and connections to the mysql server. So maybe triggering crowdsecurity/http-bad-user-agent on 127.0.0.1 somehow reproduces this?

Hmmm are you using any upstream servers that you take the forwarded remote IP as a header? such as cloudflare for example?

We have been floating the idea but this would mean a hardcoded cidr list for 127.0.0.1/8 and ::1/128 as there both loopbacks. I will forward this too the team.

I would be interested if we can see the context around the ban just so we can look for there is a wl issue?

Could you run these commands:

cscli alerts list

Find the Alert ID from the table then run:

cscli alerts inspect <id> -d

[!NOTE] Replace <id> with the ID

This will provide all context are the loopback being banned as the scenario is quite concerning.

codiflow commented 5 months ago

There you go:

################################################################################################

 - ID           : 12741
 - Date         : 2024-06-19T13:23:48Z
 - Machine      : REDACTED
 - Simulation   : false
 - Reason       : crowdsecurity/http-bad-user-agent
 - Events Count : 2
 - Scope:Value  : Ip:127.0.0.1
 - Country      : 
 - AS           : 
 - Begin        : 2024-06-19 13:23:44.761388285 +0000 UTC
 - End          : 2024-06-19 13:23:48.531502038 +0000 UTC
 - UUID         : a1f4f724-56d7-40ed-944a-c3735b4e4181

 - Context  :
╭────────────┬──────────────────────────────────────────────────────────────╮
│     Key    │                             Value                            │
├────────────┼──────────────────────────────────────────────────────────────┤
│ method     │ GET                                                          │
│ status     │ 403                                                          │
│ target_uri │ //wp-admin/setup-config.php?step=1                           │
│ target_uri │ //wordpress/wp-admin/setup-config.php?step=1                 │
│ user_agent │ Mozlila/5.0 (Linux; Android 7.0; SM-G892A Bulid/NRD90M; wv)  │
│            │ AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0           │
│            │ Chrome/60.0.3112.107 Moblie Safari/537.36                    │
╰────────────┴──────────────────────────────────────────────────────────────╯

 - Events  :

- Date: 2024-06-19 15:23:44 +0200 +0200
╭─────────────────┬──────────────────────────────────────────────────────────────╮
│       Key       │                             Value                            │
├─────────────────┼──────────────────────────────────────────────────────────────┤
│ datasource_path │ /var/log/apache2/other_vhosts_access.log                     │
│ datasource_type │ file                                                         │
│ http_args_len   │ 6                                                            │
│ http_path       │ //wp-admin/setup-config.php?step=1                           │
│ http_status     │ 403                                                          │
│ http_user_agent │ Mozlila/5.0 (Linux; Android 7.0; SM-G892A Bulid/NRD90M; wv)  │
│                 │ AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0           │
│                 │ Chrome/60.0.3112.107 Moblie Safari/537.36                    │
│ http_verb       │ GET                                                          │
│ log_type        │ http_access-log                                              │
│ service         │ http                                                         │
│ source_ip       │ 127.0.0.1                                                    │
│ target_fqdn     │ 127.0.1.1                                                    │
│ timestamp       │ 2024-06-19T15:23:44+02:00                                    │
╰─────────────────┴──────────────────────────────────────────────────────────────╯

- Date: 2024-06-19 15:23:48 +0200 +0200
╭─────────────────┬──────────────────────────────────────────────────────────────╮
│       Key       │                             Value                            │
├─────────────────┼──────────────────────────────────────────────────────────────┤
│ datasource_path │ /var/log/apache2/other_vhosts_access.log                     │
│ datasource_type │ file                                                         │
│ http_args_len   │ 6                                                            │
│ http_path       │ //wordpress/wp-admin/setup-config.php?step=1                 │
│ http_status     │ 403                                                          │
│ http_user_agent │ Mozlila/5.0 (Linux; Android 7.0; SM-G892A Bulid/NRD90M; wv)  │
│                 │ AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0           │
│                 │ Chrome/60.0.3112.107 Moblie Safari/537.36                    │
│ http_verb       │ GET                                                          │
│ log_type        │ http_access-log                                              │
│ service         │ http                                                         │
│ source_ip       │ 127.0.0.1                                                    │
│ target_fqdn     │ 127.0.1.1                                                    │
│ timestamp       │ 2024-06-19T15:23:48+02:00                                    │
╰─────────────────┴──────────────────────────────────────────────────────────────╯

Looks like these requests came in via nginx and port 80 – this are the access log entries from the corresponding nginx log (see the typos "Mozlila" and "Moblie" in the user agent string):

┌104.248.159.240 - - [19/Jun/2024:15:23:43 +0200] "GET //wp-admin/setup-config.php?step=1 HTTP/1.1" 301 162 "www.google.com" "Mozlila/5.0 (Linux; Android 7.0; SM-G892A Bulid/NRD90M; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/60.0.3112.107 Moblie Safari/537.36"  │
└104.248.159.240 - - [19/Jun/2024:15:23:45 +0200] "GET //wordpress/wp-admin/setup-config.php?step=1 HTTP/1.1" 301 162 "www.google.com" "Mozlila/5.0 (Linux; Android 7.0; SM-G892A Bulid/NRD90M; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/60.0.3112.107 Moblie Safari/537.36"

These requests came in via nginx and were redirected to apache2 on the same machine (I know, weird setup but it should not cause those kind of problems) which hosts one specific application. The nginx configuration looks like this:

[...]
        location / {
                default_type text/html;
                proxy_redirect off;
                proxy_set_header Host $http_host;
                proxy_set_header X-REQUEST_URI $request_uri;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_set_header X-Forwarded-Proto $scheme;
                # CHANGE-SERVER-PORT-HERE
                proxy_pass http://127.0.0.1:8089;
                client_max_body_size 50m;
                client_body_buffer_size 256k;
                proxy_connect_timeout 120;
                proxy_send_timeout 90;
                proxy_read_timeout 180;
                proxy_buffer_size 4k;
                proxy_buffers 16 32k;
                proxy_busy_buffers_size 64k;
                proxy_temp_file_write_size 64k;
        }

The apache2 access log looks like this:

127.0.1.1:80 127.0.0.1 - - [19/Jun/2024:15:23:44 +0200] "GET //wp-admin/setup-config.php?step=1 HTTP/1.0" 403 363 "www.google.com" "Mozlila/5.0 (Linux; Android 7.0; SM-G892A Bulid/NRD90M; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/60.0.3112.107 Moblie Safari/537.36"
127.0.1.1:80 127.0.0.1 - - [19/Jun/2024:15:23:48 +0200] "GET //wordpress/wp-admin/setup-config.php?step=1 HTTP/1.0" 403 363 "www.google.com" "Mozlila/5.0 (Linux; Android 7.0; SM-G892A Bulid/NRD90M; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/60.0.3112.107 Moblie Safari/537.36"

So the "double reverse proxy" seems to be the issue here :/

But shouldn't the crowdsec whitelist prevent such situations?

LaurenceJJones commented 5 months ago

There you go:

################################################################################################

 - ID           : 12741
 - Date         : 2024-06-19T13:23:48Z
 - Machine      : REDACTED
 - Simulation   : false
 - Reason       : crowdsecurity/http-bad-user-agent
 - Events Count : 2
 - Scope:Value  : Ip:127.0.0.1
 - Country      : 
 - AS           : 
 - Begin        : 2024-06-19 13:23:44.761388285 +0000 UTC
 - End          : 2024-06-19 13:23:48.531502038 +0000 UTC
 - UUID         : a1f4f724-56d7-40ed-944a-c3735b4e4181

 - Context  :
╭────────────┬──────────────────────────────────────────────────────────────╮
│     Key    │                             Value                            │
├────────────┼──────────────────────────────────────────────────────────────┤
│ method     │ GET                                                          │
│ status     │ 403                                                          │
│ target_uri │ //wp-admin/setup-config.php?step=1                           │
│ target_uri │ //wordpress/wp-admin/setup-config.php?step=1                 │
│ user_agent │ Mozlila/5.0 (Linux; Android 7.0; SM-G892A Bulid/NRD90M; wv)  │
│            │ AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0           │
│            │ Chrome/60.0.3112.107 Moblie Safari/537.36                    │
╰────────────┴──────────────────────────────────────────────────────────────╯

 - Events  :

- Date: 2024-06-19 15:23:44 +0200 +0200
╭─────────────────┬──────────────────────────────────────────────────────────────╮
│       Key       │                             Value                            │
├─────────────────┼──────────────────────────────────────────────────────────────┤
│ datasource_path │ /var/log/apache2/other_vhosts_access.log                     │
│ datasource_type │ file                                                         │
│ http_args_len   │ 6                                                            │
│ http_path       │ //wp-admin/setup-config.php?step=1                           │
│ http_status     │ 403                                                          │
│ http_user_agent │ Mozlila/5.0 (Linux; Android 7.0; SM-G892A Bulid/NRD90M; wv)  │
│                 │ AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0           │
│                 │ Chrome/60.0.3112.107 Moblie Safari/537.36                    │
│ http_verb       │ GET                                                          │
│ log_type        │ http_access-log                                              │
│ service         │ http                                                         │
│ source_ip       │ 127.0.0.1                                                    │
│ target_fqdn     │ 127.0.1.1                                                    │
│ timestamp       │ 2024-06-19T15:23:44+02:00                                    │
╰─────────────────┴──────────────────────────────────────────────────────────────╯

- Date: 2024-06-19 15:23:48 +0200 +0200
╭─────────────────┬──────────────────────────────────────────────────────────────╮
│       Key       │                             Value                            │
├─────────────────┼──────────────────────────────────────────────────────────────┤
│ datasource_path │ /var/log/apache2/other_vhosts_access.log                     │
│ datasource_type │ file                                                         │
│ http_args_len   │ 6                                                            │
│ http_path       │ //wordpress/wp-admin/setup-config.php?step=1                 │
│ http_status     │ 403                                                          │
│ http_user_agent │ Mozlila/5.0 (Linux; Android 7.0; SM-G892A Bulid/NRD90M; wv)  │
│                 │ AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0           │
│                 │ Chrome/60.0.3112.107 Moblie Safari/537.36                    │
│ http_verb       │ GET                                                          │
│ log_type        │ http_access-log                                              │
│ service         │ http                                                         │
│ source_ip       │ 127.0.0.1                                                    │
│ target_fqdn     │ 127.0.1.1                                                    │
│ timestamp       │ 2024-06-19T15:23:48+02:00                                    │
╰─────────────────┴──────────────────────────────────────────────────────────────╯

Looks like these requests came in via nginx and port 80 – this are the access log entries from the corresponding nginx log (see the typos "Mozlila" and "Moblie" in the user agent string):

┌104.248.159.240 - - [19/Jun/2024:15:23:43 +0200] "GET //wp-admin/setup-config.php?step=1 HTTP/1.1" 301 162 "www.google.com" "Mozlila/5.0 (Linux; Android 7.0; SM-G892A Bulid/NRD90M; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/60.0.3112.107 Moblie Safari/537.36"  │
└104.248.159.240 - - [19/Jun/2024:15:23:45 +0200] "GET //wordpress/wp-admin/setup-config.php?step=1 HTTP/1.1" 301 162 "www.google.com" "Mozlila/5.0 (Linux; Android 7.0; SM-G892A Bulid/NRD90M; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/60.0.3112.107 Moblie Safari/537.36"

These requests came in via nginx and were redirected to apache2 on the same machine (I know, weird setup but it should not cause those kind of problems) which hosts one specific application. The nginx configuration looks like this:

[...]
        location / {
                default_type text/html;
                proxy_redirect off;
                proxy_set_header Host $http_host;
                proxy_set_header X-REQUEST_URI $request_uri;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_set_header X-Forwarded-Proto $scheme;
                # CHANGE-SERVER-PORT-HERE
                proxy_pass http://127.0.0.1:8089;
                client_max_body_size 50m;
                client_body_buffer_size 256k;
                proxy_connect_timeout 120;
                proxy_send_timeout 90;
                proxy_read_timeout 180;
                proxy_buffer_size 4k;
                proxy_buffers 16 32k;
                proxy_busy_buffers_size 64k;
                proxy_temp_file_write_size 64k;
        }

The apache2 access log looks like this:

127.0.1.1:80 127.0.0.1 - - [19/Jun/2024:15:23:44 +0200] "GET //wp-admin/setup-config.php?step=1 HTTP/1.0" 403 363 "www.google.com" "Mozlila/5.0 (Linux; Android 7.0; SM-G892A Bulid/NRD90M; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/60.0.3112.107 Moblie Safari/537.36"
127.0.1.1:80 127.0.0.1 - - [19/Jun/2024:15:23:48 +0200] "GET //wordpress/wp-admin/setup-config.php?step=1 HTTP/1.0" 403 363 "www.google.com" "Mozlila/5.0 (Linux; Android 7.0; SM-G892A Bulid/NRD90M; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/60.0.3112.107 Moblie Safari/537.36"

So the "double reverse proxy" seems to be the issue here :/

But shouldn't the crowdsec whitelist prevent such situations?

Yes the whitelist should, but the file you linked was inside the hub folder that doesn't mean it is active. Could you provide the output of cscli parsers list

codiflow commented 5 months ago

Seems like my own whitelist "overrides" the default one? 🤯

INFO Ignoring file /etc/crowdsec/parsers/s02-enrich/whitelists.yaml of type parsers 
INFO Ignoring file /etc/crowdsec/hub/parsers/s02-enrich/crowdsecurity/whitelists.yaml of type parsers 

PARSERS
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
 Name                            📦 Status          Version  Local Path                                                 
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
 crowdsecurity/apache2-logs      ✔️  enabled        1.4      /etc/crowdsec/parsers/s01-parse/apache2-logs.yaml          
 crowdsecurity/dateparse-enrich  ✔️  enabled        0.2      /etc/crowdsec/parsers/s02-enrich/dateparse-enrich.yaml     
 crowdsecurity/geoip-enrich      ✔️  enabled        0.4      /etc/crowdsec/parsers/s02-enrich/geoip-enrich.yaml         
 crowdsecurity/http-logs         ✔️  enabled        1.2      /etc/crowdsec/parsers/s02-enrich/http-logs.yaml            
 crowdsecurity/mysql-logs        ✔️  enabled        0.4      /etc/crowdsec/parsers/s01-parse/mysql-logs.yaml            
 crowdsecurity/nginx-logs        ✔️  enabled        1.5      /etc/crowdsec/parsers/s01-parse/nginx-logs.yaml            
 crowdsecurity/sshd-logs         ✔️  enabled        2.3      /etc/crowdsec/parsers/s01-parse/sshd-logs.yaml             
 crowdsecurity/syslog-logs       ✔️  enabled        0.8      /etc/crowdsec/parsers/s00-raw/syslog-logs.yaml             
 crowdsecurity/whitelists        🏠  enabled,local           /etc/crowdsec/parsers/s02-enrich/custom-internal-whitelist.yaml 
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
codiflow commented 5 months ago

I just followed the instructions from the documentation like stated here: https://docs.crowdsec.net/docs/v1.4.0/whitelist/create/

Didn't know that this kicks out the default one 😅

LaurenceJJones commented 5 months ago

Seems like my own whitelist "overrides" the default one? 🤯

INFO Ignoring file /etc/crowdsec/parsers/s02-enrich/whitelists.yaml of type parsers 
INFO Ignoring file /etc/crowdsec/hub/parsers/s02-enrich/crowdsecurity/whitelists.yaml of type parsers 

PARSERS
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
 Name                            📦 Status          Version  Local Path                                                 
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
 crowdsecurity/apache2-logs      ✔️  enabled        1.4      /etc/crowdsec/parsers/s01-parse/apache2-logs.yaml          
 crowdsecurity/dateparse-enrich  ✔️  enabled        0.2      /etc/crowdsec/parsers/s02-enrich/dateparse-enrich.yaml     
 crowdsecurity/geoip-enrich      ✔️  enabled        0.4      /etc/crowdsec/parsers/s02-enrich/geoip-enrich.yaml         
 crowdsecurity/http-logs         ✔️  enabled        1.2      /etc/crowdsec/parsers/s02-enrich/http-logs.yaml            
 crowdsecurity/mysql-logs        ✔️  enabled        0.4      /etc/crowdsec/parsers/s01-parse/mysql-logs.yaml            
 crowdsecurity/nginx-logs        ✔️  enabled        1.5      /etc/crowdsec/parsers/s01-parse/nginx-logs.yaml            
 crowdsecurity/sshd-logs         ✔️  enabled        2.3      /etc/crowdsec/parsers/s01-parse/sshd-logs.yaml             
 crowdsecurity/syslog-logs       ✔️  enabled        0.8      /etc/crowdsec/parsers/s00-raw/syslog-logs.yaml             
 crowdsecurity/whitelists        🏠  enabled,local           /etc/crowdsec/parsers/s02-enrich/custom-internal-whitelist.yaml 
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

Yes, if you use the same name key it will override whatever node was previously parsed so the names need to be unique. I am going to create a change to update all documentation to use a non used name so if you copy and paste it wont break everything.

We will most likely in 2.0 version of CrowdSec make this an error as we wont want to run if the user makes this type of configuration.

Plus also I would look at the apache2 configuration to make sure it gets the remote IP address from the correct headers as currently it seems it logging all at localhost.

codiflow commented 5 months ago

Yes, if you use the same name key it will override whatever node was previously parsed so the names need to be unique.

Really good to know – couldn't derive that from the docs 😅

Plus also I would look at the apache2 configuration to make sure it gets the remote IP address from the correct headers as currently it seems it logging all at localhost.

Will look into it, thanks for the hint.

So we can close this as it was an issue by "misconfiguration" due to misleading documentation and not a bug.

TL;DR Make sure you are using a custom name for your whitelist (e.g. crowdsecurity/whitelists-custom) otherwise the default whitelist (crowdsecurity/whitelists) will be overridden by your custom whitelist.

You can check the lists being used by issuing the following command: cscli parsers list

Thanks 🙏