search

crowdsecurity / crowdsec

CrowdSec - the open-source and participative security solution offering crowdsourced protection against malicious IPs and access to the most advanced real-world CTI.
https://crowdsec.net
MIT License
8.75k stars 451 forks source link

Bug/ #1586

Closed maksim-morozov142 closed 2 years ago

maksim-morozov142 commented 2 years ago

Hello!

Problem: No active alerts Server: Oracle Linux Server release 8.5 Web server: nginx version: nginx/1.14.1

Installed the crowdsec using the instructions: https://docs.crowdsec.net/docs/getting_started/install_crowdsec/ https://packagecloud.io/crowdsec/crowdsec/install#manual-rpm

Downloaded and installed the crowdsec collections:

sudo cscli collections install crowdsecurity/nginx
sudo cscli collections install crowdsecurity/iptables

Crowdsec service started:

sudo systemctl status crowdsec
● crowdsec.service - Crowdsec agent
   Loaded: loaded (/usr/lib/systemd/system/crowdsec.service; enabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/crowdsec.service.d
           └─http-proxy.conf
   Active: active (running) since Thu 2022-06-09 16:44:52 MSK; 1 day 1h ago
 Main PID: 1580161 (crowdsec)
    Tasks: 10 (limit: 12396)
   Memory: 57.3M
   CGroup: /system.slice/crowdsec.service
           └─1580161 /usr/bin/crowdsec -c /etc/crowdsec/config.yaml

Scanned the Nginx web server with Wapiti and Nikto. /var/log/nginx/access.log contains logs with anomalous requests.

But sudo cscli alerts list and sudo cscli decisions list don't have any activity information. "No active alerts", "No active decisions"

What is the problem?

AlteredCoder commented 2 years ago

Hello,

Can you paste the output of cscli metrics please ?

maksim-morozov142 commented 2 years ago

cscli metrics

INFO[10-06-2022 06:34:02 PM] Acquisition Metrics:
+--------------------------------+------------+--------------+----------------+------------------------+
|             SOURCE             | LINES READ | LINES PARSED | LINES UNPARSED | LINES POURED TO BUCKET |
+--------------------------------+------------+--------------+----------------+------------------------+
| file:/var/log/messages         | 18.84k     | -            | 18.84k         | -                      |
| file:/var/log/nginx/access.log | 7.93k      | -            | 7.93k          | -                      |
| file:/var/log/nginx/error.log  | 7.77k      | -            | 7.77k          | -                      |
| file:/var/log/secure           | 182        | -            | 182            | -                      |
+--------------------------------+------------+--------------+----------------+------------------------+
INFO[10-06-2022 06:34:02 PM] Local Api Metrics:
+--------------------+--------+------+
|       ROUTE        | METHOD | HITS |
+--------------------+--------+------+
| /v1/alerts         | GET    | 6    |
| /v1/watchers/login | POST   | 10   |
+--------------------+--------+------+
INFO[10-06-2022 06:34:02 PM] Local Api Machines Metrics:
+--------------------------------------------------+------------+--------+------+
|                     MACHINE                      |   ROUTE    | METHOD | HITS |
+--------------------------------------------------+------------+--------+------+
| 80232ace620f4d07aa00080d1b07b30d9mPwK2nGrj90Bu0j | /v1/alerts | GET    | 6    |
+--------------------------------------------------+------------+--------+------+
AlteredCoder commented 2 years ago

Do you have the crowdsecurity/syslog-logs parser installed ? If not you should installed else no logs will be parsed.

maksim-morozov142 commented 2 years ago

Yes, the crowdsecurity/syslog-logs parser is installed:

cscli hub list
INFO[22-06-2022 11:38:03 AM] Loaded 54 collecs, 60 parsers, 74 scenarios, 3 post-overflow parsers
COLLECTIONS
------------------------------------------------------------------------------------------------------------
 NAME                               📦 STATUS   VERSION  LOCAL PATH
------------------------------------------------------------------------------------------------------------
 crowdsecurity/base-http-scenarios  ✔️  enabled  0.6      /etc/crowdsec/collections/base-http-scenarios.yaml
 crowdsecurity/http-cve             ✔️  enabled  1.0      /etc/crowdsec/collections/http-cve.yaml
 crowdsecurity/iptables             ✔️  enabled  0.1      /etc/crowdsec/collections/iptables.yaml
 crowdsecurity/linux                ✔️  enabled  0.2      /etc/crowdsec/collections/linux.yaml
 crowdsecurity/nginx                ✔️  enabled  0.2      /etc/crowdsec/collections/nginx.yaml
 crowdsecurity/sshd                 ✔️  enabled  0.2      /etc/crowdsec/collections/sshd.yaml
------------------------------------------------------------------------------------------------------------
PARSERS
-------------------------------------------------------------------------------------------------------------
 NAME                            📦 STATUS   VERSION  LOCAL PATH
-------------------------------------------------------------------------------------------------------------
 crowdsecurity/dateparse-enrich  ✔️  enabled  0.2      /etc/crowdsec/parsers/s02-enrich/dateparse-enrich.yaml
 crowdsecurity/geoip-enrich      ✔️  enabled  0.2      /etc/crowdsec/parsers/s02-enrich/geoip-enrich.yaml
 crowdsecurity/http-logs         ✔️  enabled  0.8      /etc/crowdsec/parsers/s02-enrich/http-logs.yaml
 crowdsecurity/iptables-logs     ✔️  enabled  0.3      /etc/crowdsec/parsers/s01-parse/iptables-logs.yaml
 crowdsecurity/nginx-logs        ✔️  enabled  1.2      /etc/crowdsec/parsers/s01-parse/nginx-logs.yaml
 crowdsecurity/sshd-logs         ✔️  enabled  1.9      /etc/crowdsec/parsers/s01-parse/sshd-logs.yaml
 crowdsecurity/syslog-logs       ✔️  enabled  0.8      /etc/crowdsec/parsers/s00-raw/syslog-logs.yaml
 crowdsecurity/whitelists        ✔️  enabled  0.2      /etc/crowdsec/parsers/s02-enrich/whitelists.yaml
`
blotus commented 2 years ago

Hello,

Can you paste the content of /etc/crowdsec/acquis.yaml and the full output of cscli metrics ?

I suspect you have the wrong label type for your nginx log (it should be nginx).

maksim-morozov142 commented 2 years ago

cat /etc/crowdsec/acquis.yaml

#Generated acquisition file - wizard.sh (service: nginx) / files : /var/log/nginx/access.log /var/log/nginx/error.log
filenames:
  - /var/log/nginx/access.log
  - /var/log/nginx/error.log
labels:
  type: nginx
---
#Generated acquisition file - wizard.sh (service: sshd) / files : /var/log/secure
filenames:
  - /var/log/secure
labels:
  type: syslog
---
#Generated acquisition file - wizard.sh (service: linux) / files : /var/log/messages
filenames:
  - /var/log/messages
labels:
  type: syslog
---

sudo cscli metrics

INFO[22-06-2022 12:13:01 PM] Acquisition Metrics:
+--------------------------------+------------+--------------+----------------+------------------------+
|             SOURCE             | LINES READ | LINES PARSED | LINES UNPARSED | LINES POURED TO BUCKET |
+--------------------------------+------------+--------------+----------------+------------------------+
| file:/var/log/messages         | 927        | -            | 927            | -                      |
| file:/var/log/nginx/access.log | 7.96k      | 7.95k        | 8              | -                      |
| file:/var/log/nginx/error.log  | 7.76k      | 7.55k        | 211            | -                      |
| file:/var/log/secure           | 129        | -            | 129            | -                      |
+--------------------------------+------------+--------------+----------------+------------------------+
INFO[22-06-2022 12:13:01 PM] Parser Metrics:
+---------------------------------+--------+--------+----------+
|             PARSERS             |  HITS  | PARSED | UNPARSED |
+---------------------------------+--------+--------+----------+
| child-crowdsecurity/http-logs   | 46.49k | 37.81k | 8.68k    |
| child-crowdsecurity/nginx-logs  | 23.49k | 15.50k | 7.99k    |
| child-crowdsecurity/sshd-logs   | 450    | -      | 450      |
| child-crowdsecurity/syslog-logs | 1.06k  | 1.06k  | -        |
| crowdsecurity/dateparse-enrich  | 15.50k | 15.50k | -        |
| crowdsecurity/geoip-enrich      | 15.50k | 15.50k | -        |
| crowdsecurity/http-logs         | 15.50k | 15.39k | 110      |
| crowdsecurity/nginx-logs        | 15.72k | 15.50k | 219      |
| crowdsecurity/non-syslog        | 15.72k | 15.72k | -        |
| crowdsecurity/sshd-logs         | 45     | -      | 45       |
| crowdsecurity/syslog-logs       | 1.06k  | 1.06k  | -        |
| crowdsecurity/whitelists        | 15.50k | 15.50k | -        |
+---------------------------------+--------+--------+----------+
INFO[22-06-2022 12:13:01 PM] Local Api Metrics:
+----------------------+--------+------+
|        ROUTE         | METHOD | HITS |
+----------------------+--------+------+
| /v1/alerts           | GET    | 11   |
| /v1/decisions/stream | GET    | 332  |
| /v1/watchers/login   | POST   | 15   |
+----------------------+--------+------+
INFO[22-06-2022 12:13:01 PM] Local Api Machines Metrics:
+--------------------------------------------------+------------+--------+------+
|                     MACHINE                      |   ROUTE    | METHOD | HITS |
+--------------------------------------------------+------------+--------+------+
| 80232ace620f4d07aa00080d1b07b30dXx0GIeptHLIV3Xzd | /v1/alerts | GET    | 11   |
+--------------------------------------------------+------------+--------+------+
INFO[22-06-2022 12:13:01 PM] Local Api Bouncers Metrics:
+----------------------------+----------------------+--------+------+
|          BOUNCER           |        ROUTE         | METHOD | HITS |
+----------------------------+----------------------+--------+------+
| FirewallBouncer-1655885865 | /v1/decisions/stream | GET    | 332  |
+----------------------------+----------------------+--------+------+
AlteredCoder commented 2 years ago

Hello,

from what i see in your cscli metrics , it works now ?

Also, it seems that all your logs are passing the crowdsecurity/whitelists which means that they will not be evaluate in the scenarios.

maksim-morozov142 commented 2 years ago

Yes, the problem was whitelisted! Everything is working now, thanks a lot!