search

crowdsecurity / hub

Main repository for crowdsec scenarios/parsers
https://hub.crowdsec.net
158 stars 149 forks source link

gitea parser don't parse with docker source file #479

Closed vincentDcmps closed 2 years ago

vincentDcmps commented 2 years ago

hi @LePresidente I try to implement gitea parser on my instance I use docker source I try to generate some authentification issue I have the log in my docker log 

2022/05/26 16:04:50 ...ers/web/auth/auth.go:200:SignInPost() [I] Failed authentication attempt for vincent from 192.168.1.15:0: user does not exist [uid: 1, name: vincent, keyid: 0]

but when I check cscli metrics he doesn't parse any lines

INFO[26-05-2022 04:21:21 PM] Acquisition Metrics:

+----------------------------------------------------+------------+--------------+----------------+------------------------+
|                       SOURCE                       | LINES READ | LINES PARSED | LINES UNPARSED | LINES POURED TO BUCKET |
+----------------------------------------------------+------------+--------------+----------------+------------------------+
| docker:/gitea-594f42cc-070d-3dd8-3d9f-693ae484461a | 672        | -            | 672            | -                      |
| docker:/hass-9dee9f03-404d-d936-d657-865ff6b47ea3  | 52         | 2            | 50             | -                      |
+----------------------------------------------------+------------+--------------+----------------+------------------------+
INFO[26-05-2022 04:21:21 PM] Parser Metrics:
+-----------------------------------------+------+--------+----------+
|                 PARSERS                 | HITS | PARSED | UNPARSED |
+-----------------------------------------+------+--------+----------+
| LePresidente/gitea-logs                 | 672  | -      | 672      |
| child-LePresidente/gitea-logs           | 672  | -      | 672      |
| child-crowdsecurity/home-assistant-logs | 52   | 2      | 50       |

here my acquis config

---
source: docker
container_name_regexp:
  - gitea-*
labels:
  type: gitea
buixor commented 2 years ago

Hello,

Looking at your log example, the logs seem to match the existing tests of the parser (https://github.com/crowdsecurity/hub/blob/master/.tests/gitea-logs/gitea-logs.log).

Can you show your list of parsers (cscli hub list) and the complete cscli metrics output ? Maybe you are missing the docker log parser or such ?

Thanks

vincentDcmps commented 2 years ago

yes I have note that log match it's for that I ask myself if issue is not from docker source here output asked

INFO[27-05-2022 10:09:54 AM] Acquisition Metrics:
+----------------------------------------------------+------------+--------------+----------------+------------------------+
|                       SOURCE                       | LINES READ | LINES PARSED | LINES UNPARSED | LINES POURED TO BUCKET |
+----------------------------------------------------+------------+--------------+----------------+------------------------+
| docker:/gitea-594f42cc-070d-3dd8-3d9f-693ae484461a | 11.42k     | -            | 11.42k         | -                      |
| docker:/hass-9dee9f03-404d-d936-d657-865ff6b47ea3  | 3.56k      | 2            | 3.55k          | -                      |
+----------------------------------------------------+------------+--------------+----------------+------------------------+
INFO[27-05-2022 10:09:54 AM] Parser Metrics:
+-----------------------------------------+--------+--------+----------+
|                 PARSERS                 |  HITS  | PARSED | UNPARSED |
+-----------------------------------------+--------+--------+----------+
| LePresidente/gitea-logs                 | 11.42k | -      | 11.42k   |
| child-LePresidente/gitea-logs           | 11.42k | -      | 11.42k   |
| child-crowdsecurity/home-assistant-logs | 3.56k  | 2      | 3.55k    |
| crowdsecurity/dateparse-enrich          | 2      | 2      | -        |
| crowdsecurity/geoip-enrich              | 2      | 2      | -        |
| crowdsecurity/home-assistant-logs       | 3.56k  | 2      | 3.55k    |
| crowdsecurity/non-syslog                | 14.98k | 14.98k | -        |
| crowdsecurity/whitelists                | 2      | 2      | -        |
+-----------------------------------------+--------+--------+----------+

################################################################################################

 - ID         : 18
 - Date       : 2022-05-27T02:00:30Z
 - Machine    : corwin
 - Simulation : false
 - Reason     : crowdsecurity/http-probing
 - Events Count : 11
 - Scope:Value: Ip:130.61.226.117
 - Country    : DE
 - AS         : ORACLE-BMC-31898
 - Begin      : 2022-05-27 02:00:27.272399116 +0000 UTC
 - End        : 2022-05-27 02:00:29.986913154 +0000 UTC

 - Active Decisions  :
+--------+-------------------+--------+-----------------+----------------------+
|   ID   |    SCOPE:VALUE    | ACTION |   EXPIRATION    |      CREATED AT      |
+--------+-------------------+--------+-----------------+----------------------+
| 174102 | Ip:130.61.226.117 | ban    | 7m15.823102598s | 2022-05-27T02:00:30Z |
+--------+-------------------+--------+-----------------+----------------------+
 vincent@fixe-pc    ~  nomad job allocs …                                ✔  10372  07:53:14 
 vincent@fixe-pc    ~  nomad job allocs crowdsec-api                     ✔  10372  07:53:14 
ID        Node ID   Task Group    Version  Desired  Status   Created     Modified
490e9ef8  a7e0fc8c  crowdsec-api  9        run      running  21h10m ago  21h9m ago
 vincent@fixe-pc    ~  nomad job allocs crowdsec-…                       ✔  10373  12:08:52 
 vincent@fixe-pc    ~  nomad job allocs crowdsec-agent                   ✔  10373  12:08:52 
ID        Node ID   Task Group      Version  Desired  Status   Created     Modified
3c2df020  a7e0fc8c  crowdsec-agent  22       run      running  18h29m ago  18h29m ago
9a764702  b5bb7bd9  crowdsec-agent  22       run      running  18h29m ago  18h29m ago
9ff67a40  882342d5  crowdsec-agent  22       run      running  18h29m ago  18h29m ago
 vincent@fixe-pc    ~  nomad exec 9ff67a40 cscli metrics                 ✔  10374  12:08:54 
INFO[27-05-2022 10:09:15 AM] Acquisition Metrics:
+------------------------------------------------------+------------+--------------+----------------+------------------------+
|                        SOURCE                        | LINES READ | LINES PARSED | LINES UNPARSED | LINES POURED TO BUCKET |
+------------------------------------------------------+------------+--------------+----------------+------------------------+
| docker:/traefik-4a62c500-b5c7-6a14-f5ea-8abcc74aa801 | 10         | -            | 10             | -                      |
+------------------------------------------------------+------------+--------------+----------------+------------------------+
INFO[27-05-2022 10:09:15 AM] Parser Metrics:
+----------------------------------------+------+--------+----------+
|                PARSERS                 | HITS | PARSED | UNPARSED |
+----------------------------------------+------+--------+----------+
| child-child-crowdsecurity/traefik-logs | 130  | 30     | 100      |
| child-crowdsecurity/traefik-logs       | 20   | -      | 20       |
| crowdsecurity/non-syslog               | 10   | 10     | -        |
| crowdsecurity/traefik-logs             | 10   | -      | 10       |
+----------------------------------------+------+--------+----------+
 vincent@fixe-pc    ~  nomad exec 3c2df cscli metrics                    ✔  10375  12:09:15 
INFO[27-05-2022 10:09:29 AM] Acquisition Metrics:
+----------------------------------------------------+------------+--------------+----------------+------------------------+
|                       SOURCE                       | LINES READ | LINES PARSED | LINES UNPARSED | LINES POURED TO BUCKET |
+----------------------------------------------------+------------+--------------+----------------+------------------------+
| docker:/gitea-594f42cc-070d-3dd8-3d9f-693ae484461a | 11.40k     | -            | 11.40k         | -                      |
| docker:/hass-9dee9f03-404d-d936-d657-865ff6b47ea3  | 3.56k      | 2            | 3.55k          | -                      |
+----------------------------------------------------+------------+--------------+----------------+------------------------+
INFO[27-05-2022 10:09:29 AM] Parser Metrics:
+-----------------------------------------+--------+--------+----------+
|                 PARSERS                 |  HITS  | PARSED | UNPARSED |
+-----------------------------------------+--------+--------+----------+
| LePresidente/gitea-logs                 | 11.40k | -      | 11.40k   |
| child-LePresidente/gitea-logs           | 11.40k | -      | 11.40k   |
| child-crowdsecurity/home-assistant-logs | 3.56k  | 2      | 3.55k    |
| crowdsecurity/dateparse-enrich          | 2      | 2      | -        |
| crowdsecurity/geoip-enrich              | 2      | 2      | -        |
| crowdsecurity/home-assistant-logs       | 3.56k  | 2      | 3.55k    |
| crowdsecurity/non-syslog                | 14.96k | 14.96k | -        |
| crowdsecurity/whitelists                | 2      | 2      | -        |
+-----------------------------------------+--------+--------+----------+
 vincent@fixe-pc    ~                                                                                                                                                                   ✔  10376  12:09:29 
 vincent@fixe-pc    ~  nomad exec 3c2df cscli metrics                                                                                                                         SIGINT(2) ↵  10376  12:09:41 
INFO[27-05-2022 10:09:54 AM] Acquisition Metrics:
+----------------------------------------------------+------------+--------------+----------------+------------------------+
|                       SOURCE                       | LINES READ | LINES PARSED | LINES UNPARSED | LINES POURED TO BUCKET |
+----------------------------------------------------+------------+--------------+----------------+------------------------+
| docker:/gitea-594f42cc-070d-3dd8-3d9f-693ae484461a | 11.42k     | -            | 11.42k         | -                      |
| docker:/hass-9dee9f03-404d-d936-d657-865ff6b47ea3  | 3.56k      | 2            | 3.55k          | -                      |
+----------------------------------------------------+------------+--------------+----------------+------------------------+
INFO[27-05-2022 10:09:54 AM] Parser Metrics:
+-----------------------------------------+--------+--------+----------+
|                 PARSERS                 |  HITS  | PARSED | UNPARSED |
+-----------------------------------------+--------+--------+----------+
| LePresidente/gitea-logs                 | 11.42k | -      | 11.42k   |
| child-LePresidente/gitea-logs           | 11.42k | -      | 11.42k   |
| child-crowdsecurity/home-assistant-logs | 3.56k  | 2      | 3.55k    |
| crowdsecurity/dateparse-enrich          | 2      | 2      | -        |
| crowdsecurity/geoip-enrich              | 2      | 2      | -        |
| crowdsecurity/home-assistant-logs       | 3.56k  | 2      | 3.55k    |
| crowdsecurity/non-syslog                | 14.98k | 14.98k | -        |
| crowdsecurity/whitelists                | 2      | 2      | -        |
+-----------------------------------------+--------+--------+----------+
 vincent@fixe-pc    ~                                                                                                                                                                   ✔  10376  12:09:54 
 vincent@fixe-pc    ~  cscli hub list                                                                                                                                         SIGINT(2) ↵  10376  12:10:31 
zsh: command not found: cscli
 vincent@fixe-pc    ~  cscli hub list                                                                                                                                               127 ↵  10377  12:10:38 
 vincent@fixe-pc    ~  nomad exec 3c2df cscli metrics                                                                                                                         SIGINT(2) ↵  10377  12:10:41 
INFO[27-05-2022 10:10:43 AM] Acquisition Metrics:
+----------------------------------------------------+------------+--------------+----------------+------------------------+
|                       SOURCE                       | LINES READ | LINES PARSED | LINES UNPARSED | LINES POURED TO BUCKET |
+----------------------------------------------------+------------+--------------+----------------+------------------------+
| docker:/gitea-594f42cc-070d-3dd8-3d9f-693ae484461a | 11.43k     | -            | 11.43k         | -                      |
| docker:/hass-9dee9f03-404d-d936-d657-865ff6b47ea3  | 3.56k      | 2            | 3.55k          | -                      |
+----------------------------------------------------+------------+--------------+----------------+------------------------+
INFO[27-05-2022 10:10:43 AM] Parser Metrics:
+-----------------------------------------+--------+--------+----------+
|                 PARSERS                 |  HITS  | PARSED | UNPARSED |
+-----------------------------------------+--------+--------+----------+
| LePresidente/gitea-logs                 | 11.43k | -      | 11.43k   |
| child-LePresidente/gitea-logs           | 11.43k | -      | 11.43k   |
| child-crowdsecurity/home-assistant-logs | 3.56k  | 2      | 3.55k    |
| crowdsecurity/dateparse-enrich          | 2      | 2      | -        |
| crowdsecurity/geoip-enrich              | 2      | 2      | -        |
| crowdsecurity/home-assistant-logs       | 3.56k  | 2      | 3.55k    |
| crowdsecurity/non-syslog                | 14.99k | 14.99k | -        |
| crowdsecurity/whitelists                | 2      | 2      | -        |
+-----------------------------------------+--------+--------+----------+
 vincent@fixe-pc    ~  nomad exec 3c2df cscli hub list                                                                                                                                  ✔  10378  12:10:43 
INFO[27-05-2022 10:10:49 AM] Loaded 53 collecs, 59 parsers, 73 scenarios, 3 post-overflow parsers
COLLECTIONS
------------------------------------------------------------------------------------------------------------
 NAME                               📦 STATUS   VERSION  LOCAL PATH
------------------------------------------------------------------------------------------------------------
 LePresidente/gitea                 ✔️  enabled  0.2      /etc/crowdsec/collections/gitea.yml
 crowdsecurity/base-http-scenarios  ✔️  enabled  0.6      /etc/crowdsec/collections/base-http-scenarios.yaml
 crowdsecurity/home-assistant       ✔️  enabled  0.1      /etc/crowdsec/collections/home-assistant.yaml
 crowdsecurity/http-cve             ✔️  enabled  1.0      /etc/crowdsec/collections/http-cve.yaml
 crowdsecurity/linux                ✔️  enabled  0.2      /etc/crowdsec/collections/linux.yaml
 crowdsecurity/sshd                 ✔️  enabled  0.2      /etc/crowdsec/collections/sshd.yaml
 crowdsecurity/traefik              ✔️  enabled  0.1      /etc/crowdsec/collections/traefik.yaml
------------------------------------------------------------------------------------------------------------
PARSERS
------------------------------------------------------------------------------------------------------------------
 NAME                               📦 STATUS   VERSION  LOCAL PATH
------------------------------------------------------------------------------------------------------------------
 LePresidente/gitea-logs            ✔️  enabled  0.1      /etc/crowdsec/parsers/s01-parse/gitea-logs.yaml
 crowdsecurity/dateparse-enrich     ✔️  enabled  0.2      /etc/crowdsec/parsers/s02-enrich/dateparse-enrich.yaml
 crowdsecurity/docker-logs          ✔️  enabled  0.1      /etc/crowdsec/parsers/s00-raw/docker-logs.yaml
 crowdsecurity/geoip-enrich         ✔️  enabled  0.2      /etc/crowdsec/parsers/s02-enrich/geoip-enrich.yaml
 crowdsecurity/home-assistant-logs  ✔️  enabled  0.4      /etc/crowdsec/parsers/s01-parse/home-assistant-logs.yaml
 crowdsecurity/http-logs            ✔️  enabled  0.8      /etc/crowdsec/parsers/s02-enrich/http-logs.yaml
 crowdsecurity/sshd-logs            ✔️  enabled  1.9      /etc/crowdsec/parsers/s01-parse/sshd-logs.yaml
 crowdsecurity/syslog-logs          ✔️  enabled  0.8      /etc/crowdsec/parsers/s00-raw/syslog-logs.yaml
 crowdsecurity/traefik-logs         ✔️  enabled  0.4      /etc/crowdsec/parsers/s01-parse/traefik-logs.yaml
 crowdsecurity/whitelists           ✔️  enabled  0.2      /etc/crowdsec/parsers/s02-enrich/whitelists.yaml
------------------------------------------------------------------------------------------------------------------
SCENARIOS
----------------------------------------------------------------------------------------------------------------------------------------
 NAME                                              📦 STATUS   VERSION  LOCAL PATH
----------------------------------------------------------------------------------------------------------------------------------------
 LePresidente/gitea-bf                             ✔️  enabled  0.1      /etc/crowdsec/scenarios/gitea-bf.yaml
 crowdsecurity/apache_log4j2_cve-2021-44228        ✔️  enabled  0.4      /etc/crowdsec/scenarios/apache_log4j2_cve-2021-44228.yaml
 crowdsecurity/f5-big-ip-cve-2020-5902             ✔️  enabled  0.1      /etc/crowdsec/scenarios/f5-big-ip-cve-2020-5902.yaml
 crowdsecurity/fortinet-cve-2018-13379             ✔️  enabled  0.2      /etc/crowdsec/scenarios/fortinet-cve-2018-13379.yaml
 crowdsecurity/grafana-cve-2021-43798              ✔️  enabled  0.1      /etc/crowdsec/scenarios/grafana-cve-2021-43798.yaml
 crowdsecurity/home-assistant-bf                   ✔️  enabled  0.2      /etc/crowdsec/scenarios/home-assistant-bf.yaml
 crowdsecurity/http-backdoors-attempts             ✔️  enabled  0.2      /etc/crowdsec/scenarios/http-backdoors-attempts.yaml
 crowdsecurity/http-bad-user-agent                 ✔️  enabled  0.7      /etc/crowdsec/scenarios/http-bad-user-agent.yaml
 crowdsecurity/http-crawl-non_statics              ✔️  enabled  0.3      /etc/crowdsec/scenarios/http-crawl-non_statics.yaml
 crowdsecurity/http-cve-2021-41773                 ✔️  enabled  0.1      /etc/crowdsec/scenarios/http-cve-2021-41773.yaml
 crowdsecurity/http-cve-2021-42013                 ✔️  enabled  0.1      /etc/crowdsec/scenarios/http-cve-2021-42013.yaml
 crowdsecurity/http-generic-bf                     ✔️  enabled  0.2      /etc/crowdsec/scenarios/http-generic-bf.yaml
 crowdsecurity/http-open-proxy                     ✔️  enabled  0.2      /etc/crowdsec/scenarios/http-open-proxy.yaml
 crowdsecurity/http-path-traversal-probing         ✔️  enabled  0.2      /etc/crowdsec/scenarios/http-path-traversal-probing.yaml
 crowdsecurity/http-probing                        ✔️  enabled  0.2      /etc/crowdsec/scenarios/http-probing.yaml
 crowdsecurity/http-sensitive-files                ✔️  enabled  0.2      /etc/crowdsec/scenarios/http-sensitive-files.yaml
 crowdsecurity/http-sqli-probing                   ✔️  enabled  0.2      /etc/crowdsec/scenarios/http-sqli-probing.yaml
 crowdsecurity/http-xss-probing                    ✔️  enabled  0.2      /etc/crowdsec/scenarios/http-xss-probing.yaml
 crowdsecurity/jira_cve-2021-26086                 ✔️  enabled  0.1      /etc/crowdsec/scenarios/jira_cve-2021-26086.yaml
 crowdsecurity/pulse-secure-sslvpn-cve-2019-11510  ✔️  enabled  0.2      /etc/crowdsec/scenarios/pulse-secure-sslvpn-cve-2019-11510.yaml
 crowdsecurity/spring4shell_cve-2022-22965         ✔️  enabled  0.2      /etc/crowdsec/scenarios/spring4shell_cve-2022-22965.yaml
 crowdsecurity/ssh-bf                              ✔️  enabled  0.1      /etc/crowdsec/scenarios/ssh-bf.yaml
 crowdsecurity/ssh-slow-bf                         ✔️  enabled  0.2      /etc/crowdsec/scenarios/ssh-slow-bf.yaml
 crowdsecurity/thinkphp-cve-2018-20062             ✔️  enabled  0.3      /etc/crowdsec/scenarios/thinkphp-cve-2018-20062.yaml
 crowdsecurity/vmware-cve-2022-22954               ✔️  enabled  0.2      /etc/crowdsec/scenarios/vmware-cve-2022-22954.yaml
 crowdsecurity/vmware-vcenter-vmsa-2021-0027       ✔️  enabled  0.1      /etc/crowdsec/scenarios/vmware-vcenter-vmsa-2021-0027.yaml
 ltsich/http-w00tw00t                              ✔️  enabled  0.1      /etc/crowdsec/scenarios/http-w00tw00t.yaml
----------------------------------------------------------------------------------------------------------------------------------------
POSTOVERFLOWS
--------------------------------------
 NAME  📦 STATUS  VERSION  LOCAL PATH
--------------------------------------
--------------------------------------
LePresidente commented 2 years ago

Change your aquis file

---
source: docker
container_name_regexp:
  - gitea-*
labels:
  type: docker
  program: gitea
vincentDcmps commented 2 years ago

I change my acquis file with configuration given, but seem log are always not parsed when authent failled

LePresidente commented 2 years ago

Ok i finally got around to testing this

Your original aquis.yaml file was correct

my aquis.yaml file for testing

source: docker
container_name:
  - Gitea      
labels:              
  type: gitea

output

/ # cscli parsers inspect LePresidente/gitea-logs
type: parsers
stage: s01-parse
name: LePresidente/gitea-logs
filename: gitea-logs.yaml
description: Parse gitea logs
author: LePresidente
belongs_to_collections:
- LePresidente/gitea
remote_path: parsers/s01-parse/LePresidente/gitea-logs.yaml
version: "0.1"
local_path: /etc/crowdsec/parsers/s01-parse/gitea-logs.yaml
localversion: "0.1"
localhash: 61733cf559c01d68ad3ee7d571c836273a0f26e03d1ac7d3b6c5f80783f802de
installed: true
downloaded: true
uptodate: true
tainted: false
local: false

Current metrics : 

 - (Parser) LePresidente/gitea-logs: 
+--------------+------+--------+----------+
|   PARSERS    | HITS | PARSED | UNPARSED |
+--------------+------+--------+----------+
| docker:Gitea |    2 |      1 |        1 |
+--------------+------+--------+----------+