search

crowdsecurity / hub

Main repository for crowdsec scenarios/parsers
https://hub.crowdsec.net
149 stars 147 forks source link

NGINX Log Parsing Issue in Ubuntu 18.04.4 (missing evt.StrTime) #817

Open oyale opened 1 year ago

oyale commented 1 year ago

What happened?

After installing CrowdSec v1.5.2 on Ubuntu 18.04.4 using the provided repository and script, and then installing crowdsec and crowdsec-firewall-bouncer-iptables, the NGINX logs are not being parsed correctly.

What did you expect to happen?

  1. Install CrowdSec on Ubuntu using the repository.
  2. Install crowdsec and crowdsec-firewall-bouncer-iptables.
  3. Attempt to parse NGINX logs.

How can we reproduce it (as minimally and precisely as possible)?

I used the cscli explain command with the following log:

cscli explain --log '185.132.xx.xxx - - [Aug/31/2023:13:57:03 +0200] "GET /cable HTTP/1.1" 101 24 6.890 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Safari/537.36"' --type nginx --verbose

And received the following error:

WARN[31-08-2023 15:02:41] Line 0/1 is missing evt.StrTime. It is most likely a mistake as it will prevent your logs to be processed in time-machine/forensic mode. 
line: 185.132.xx.xxx - - [Aug/31/2023:13:57:03 +0200] "GET /cable HTTP/1.1" 101 24 6.890 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Safari/537.36"
    ├ s00-raw
    |   ├ 🟢 crowdsecurity/non-syslog (+5 ~8)
    |       ├ update evt.ExpectMode : %!s(int=0) -> 1
    |       ├ update evt.Stage :  -> s01-parse
    |       ├ update evt.Line.Raw :  -> 185.132.xx.xxx - - [Aug/31/2023:13:57:03 +0200] "GET /cable HTTP/1.1" 101 24 6.890 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Safari/537.36"
    |       ├ update evt.Line.Src :  -> /tmp/cscli_explain4205633973/cscli_test_tmp.log
    |       ├ update evt.Line.Time : 0001-01-01 00:00:00 +0000 UTC -> 2023-08-31 13:02:38.290108895 +0000 UTC
    |       ├ create evt.Line.Labels.type : nginx
    |       ├ update evt.Line.Process : %!s(bool=false) -> true
    |       ├ update evt.Line.Module :  -> file
    |       ├ create evt.Parsed.message : 185.132.xx.xxx - - [Aug/31/2023:13:57:03 +0200] "GET /cable HTTP/1.1" 101 24 6.890 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Safari/537.36"
    |       ├ create evt.Parsed.program : nginx
    |       ├ update evt.Time : 0001-01-01 00:00:00 +0000 UTC -> 2023-08-31 13:02:38.290168537 +0000 UTC
    |       ├ create evt.Meta.datasource_type : file
    |       ├ create evt.Meta.datasource_path : /tmp/cscli_explain4205633973/cscli_test_tmp.log
    |   └ 🔴 crowdsecurity/syslog-logs
    ├ s01-parse
    |   ├ 🔴 crowdsecurity/nginx-logs
    |   └ 🔴 crowdsecurity/sshd-logs
    └-------- parser failure 🔴

Anything else we need to know?

nginx's log_format:

log_format nginx '$remote_addr - $remote_user [$time_local] ' '"$request" $status $body_bytes_sent $request_time ' '"$http_referer" "$http_user_agent"';

nginx -v

nginx/1.18.0

Crowdsec version

```console $ cscli version 2023/08/31 14:49:38 version: v1.5.2-debian-pragmatic-4fbc3402fba932c8bd34b671527dcf7909d264c0 2023/08/31 14:49:38 Codename: alphaga 2023/08/31 14:49:38 BuildDate: 2023-05-26_16:13:25 2023/08/31 14:49:38 GoVersion: 1.20.1 2023/08/31 14:49:38 Platform: linux 2023/08/31 14:49:38 Constraint_parser: >= 1.0, <= 2.0 2023/08/31 14:49:38 Constraint_scenario: >= 1.0, < 3.0 2023/08/31 14:49:38 Constraint_api: v1 2023/08/31 14:49:38 Constraint_acquis: >= 1.0, < 2.0 ```

OS version

```console # On Linux: $ cat /etc/os-release NAME="Ubuntu" VERSION="18.04.4 LTS (Bionic Beaver)" ID=ubuntu ID_LIKE=debian PRETTY_NAME="Ubuntu 18.04.4 LTS" VERSION_ID="18.04" HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" VERSION_CODENAME=bionic UBUNTU_CODENAME=bionic $ uname -a Linux server-redacted 4.15.0-196-generic crowdsecurity/crowdsec#207-Ubuntu SMP Thu Oct 27 21:24:58 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux ```

Enabled collections and parsers

```console $ cscli hub list -o raw crowdsecurity/base-http-scenarios,enabled,0.6,http common : scanners detection,collections crowdsecurity/http-cve,enabled,2.1,,collections crowdsecurity/linux,enabled,0.2,core linux support : syslog+geoip+ssh,collections crowdsecurity/nginx,enabled,0.2,nginx support : parser and generic http scenarios,collections crowdsecurity/sshd,enabled,0.2,sshd support : parser and brute-force detection,collections crowdsecurity/dateparse-enrich,enabled,0.2,,parsers crowdsecurity/geoip-enrich,enabled,0.2,"Populate event with geoloc info : as, country, coords, source range.",parsers crowdsecurity/http-logs,enabled,1.1,"Parse more Specifically HTTP logs, such as HTTP Code, HTTP path, HTTP args and if its a static ressource",parsers crowdsecurity/nginx-logs,enabled,1.4,Parse nginx access and error logs,parsers crowdsecurity/sshd-logs,enabled,2.2,Parse openSSH logs,parsers crowdsecurity/syslog-logs,enabled,0.8,,parsers crowdsecurity/whitelists,enabled,0.2,Whitelist events from private ipv4 addresses,parsers crowdsecurity/CVE-2019-18935,enabled,0.1,Detect Telerik CVE-2019-18935 exploitation attempts,scenarios crowdsecurity/CVE-2022-26134,enabled,0.1,Detect CVE-2022-26134 exploits,scenarios crowdsecurity/CVE-2022-35914,enabled,0.1,Detect CVE-2022-35914 exploits,scenarios crowdsecurity/CVE-2022-37042,enabled,0.1,Detect CVE-2022-37042 exploits,scenarios crowdsecurity/CVE-2022-40684,enabled,0.2,Detect cve-2022-40684 exploitation attempts,scenarios crowdsecurity/CVE-2022-41082,enabled,0.3,Detect CVE-2022-41082 exploits,scenarios crowdsecurity/CVE-2022-41697,enabled,0.1,Detect CVE-2022-41697 enumeration,scenarios crowdsecurity/CVE-2022-42889,enabled,0.2,Detect CVE-2022-42889 exploits (Text4Shell),scenarios crowdsecurity/CVE-2022-44877,enabled,0.2,Detect CVE-2022-44877 exploits,scenarios crowdsecurity/CVE-2022-46169,enabled,0.1,Detect CVE-2022-46169 brute forcing,scenarios crowdsecurity/apache_log4j2_cve-2021-44228,enabled,0.4,Detect cve-2021-44228 exploitation attemps,scenarios crowdsecurity/f5-big-ip-cve-2020-5902,enabled,0.1,Detect cve-2020-5902 exploitation attemps,scenarios crowdsecurity/fortinet-cve-2018-13379,enabled,0.2,Detect cve-2018-13379 exploitation attemps,scenarios crowdsecurity/grafana-cve-2021-43798,enabled,0.1,Detect cve-2021-43798 exploitation attemps,scenarios crowdsecurity/http-backdoors-attempts,enabled,0.3,Detect attempt to common backdoors,scenarios crowdsecurity/http-bad-user-agent,enabled,0.8,Detect bad user-agents,scenarios crowdsecurity/http-crawl-non_statics,enabled,0.3,Detect aggressive crawl from single ip,scenarios crowdsecurity/http-cve-2021-41773,enabled,0.1,cve-2021-41773,scenarios crowdsecurity/http-cve-2021-42013,enabled,0.1,cve-2021-42013,scenarios crowdsecurity/http-generic-bf,enabled,0.4,Detect generic http brute force,scenarios crowdsecurity/http-open-proxy,enabled,0.3,Detect scan for open proxy,scenarios crowdsecurity/http-path-traversal-probing,enabled,0.2,Detect path traversal attempt,scenarios crowdsecurity/http-probing,enabled,0.2,Detect site scanning/probing from a single ip,scenarios crowdsecurity/http-sensitive-files,enabled,0.2,"Detect attempt to access to sensitive files (.log, .db ..) or folders (.git)",scenarios crowdsecurity/http-sqli-probing,enabled,0.2,A scenario that detects SQL injection probing with minimal false positives,scenarios crowdsecurity/http-xss-probing,enabled,0.2,A scenario that detects XSS probing with minimal false positives,scenarios crowdsecurity/jira_cve-2021-26086,enabled,0.1,Detect Atlassian Jira CVE-2021-26086 exploitation attemps,scenarios crowdsecurity/netgear_rce,enabled,0.2,Detect Netgear RCE DGN1000/DGN220 exploitation attempts,scenarios crowdsecurity/nginx-req-limit-exceeded,enabled,0.1,Detects IPs which violate nginx's user set request limit.,scenarios crowdsecurity/pulse-secure-sslvpn-cve-2019-11510,enabled,0.2,Detect cve-2019-11510 exploitation attemps,scenarios crowdsecurity/spring4shell_cve-2022-22965,enabled,0.2,Detect cve-2022-22965 probing,scenarios crowdsecurity/ssh-bf,enabled,0.1,Detect ssh bruteforce,scenarios crowdsecurity/ssh-slow-bf,enabled,0.2,Detect slow ssh bruteforce,scenarios crowdsecurity/thinkphp-cve-2018-20062,enabled,0.3,Detect ThinkPHP CVE-2018-20062 exploitation attemps,scenarios crowdsecurity/vmware-cve-2022-22954,enabled,0.2,Detect Vmware CVE-2022-22954 exploitation attempts,scenarios crowdsecurity/vmware-vcenter-vmsa-2021-0027,enabled,0.1,Detect VMSA-2021-0027 exploitation attemps,scenarios ltsich/http-w00tw00t,enabled,0.1,detect w00tw00t,scenarios ```

Acquisition config

```console # On Linux: $ cat /etc/crowdsec/acquis.yaml /etc/crowdsec/acquis.d/* #Generated acquisition file - wizard.sh (service: nginx) / files : /var/log/nginx/access.log /var/log/nginx/error.log filenames: - /var/log/nginx/access.log - /var/log/nginx/error.log labels: type: nginx --- #Generated acquisition file - wizard.sh (service: ssh) / files : /var/log/auth.log filenames: - /var/log/auth.log labels: type: syslog --- #Generated acquisition file - wizard.sh (service: linux) / files : /var/log/syslog filenames: - /var/log/syslog labels: type: syslog --- cat: '/etc/crowdsec/acquis.d/*': No such file or directory ```

Config show

```console $ cscli config show Global: - Configuration Folder : /etc/crowdsec - Configuration Folder : /etc/crowdsec - Data Folder : /var/lib/crowdsec/data - Hub Folder : /etc/crowdsec/hub - Simulation File : /etc/crowdsec/simulation.yaml - Log Folder : /var/log/ - Log level : info - Log Media : file Crowdsec: - Acquisition File : /etc/crowdsec/acquis.yaml - Parsers routines : 1 - Acquisition Folder : /etc/crowdsec/acquis.d cscli: - Output : human - Hub Branch : - Hub Folder : /etc/crowdsec/hub API Client: - URL : http://127.0.0.1:8080/ - Login : bd854b54992945c7a769a8b2e2db5c77in5EoS25l5kmhqy0 - Credentials File : /etc/crowdsec/local_api_credentials.yaml Local API Server: - Listen URL : 127.0.0.1:8080 - Profile File : /etc/crowdsec/profiles.yaml - Trusted IPs: - 127.0.0.1 - ::1 - Database: - Type : sqlite - Path : /var/lib/crowdsec/data/crowdsec.db - Flush age : 7d - Flush size : 5000 ```

Prometheus metrics

```console $ cscli metrics Acquisition Metrics: ╭────────────────────────────────┬────────────┬──────────────┬────────────────┬────────────────────────╮ │ Source │ Lines read │ Lines parsed │ Lines unparsed │ Lines poured to bucket │ ├────────────────────────────────┼────────────┼──────────────┼────────────────┼────────────────────────┤ │ file:/var/log/auth.log │ 173 │ 3 │ 170 │ 10 │ │ file:/var/log/nginx/access.log │ 574 │ - │ 574 │ - │ │ file:/var/log/syslog │ 34 │ - │ 34 │ - │ ╰────────────────────────────────┴────────────┴──────────────┴────────────────┴────────────────────────╯ Bucket Metrics: ╭─────────────────────────────────────┬───────────────┬───────────┬──────────────┬────────┬─────────╮ │ Bucket │ Current Count │ Overflows │ Instantiated │ Poured │ Expired │ ├─────────────────────────────────────┼───────────────┼───────────┼──────────────┼────────┼─────────┤ │ crowdsecurity/ssh-bf │ - │ - │ 2 │ 3 │ 2 │ │ crowdsecurity/ssh-bf_user-enum │ - │ - │ 2 │ 2 │ 2 │ │ crowdsecurity/ssh-slow-bf │ - │ - │ 2 │ 3 │ 2 │ │ crowdsecurity/ssh-slow-bf_user-enum │ - │ - │ 2 │ 2 │ 2 │ ╰─────────────────────────────────────┴───────────────┴───────────┴──────────────┴────────┴─────────╯ Parser Metrics: ╭─────────────────────────────────┬───────┬────────┬──────────╮ │ Parsers │ Hits │ Parsed │ Unparsed │ ├─────────────────────────────────┼───────┼────────┼──────────┤ │ child-crowdsecurity/nginx-logs │ 1.15k │ - │ 1.15k │ │ child-crowdsecurity/sshd-logs │ 158 │ 3 │ 155 │ │ child-crowdsecurity/syslog-logs │ 207 │ 207 │ - │ │ crowdsecurity/dateparse-enrich │ 3 │ 3 │ - │ │ crowdsecurity/geoip-enrich │ 3 │ 3 │ - │ │ crowdsecurity/nginx-logs │ 574 │ - │ 574 │ │ crowdsecurity/non-syslog │ 574 │ 574 │ - │ │ crowdsecurity/sshd-logs │ 16 │ 3 │ 13 │ │ crowdsecurity/syslog-logs │ 207 │ 207 │ - │ │ crowdsecurity/whitelists │ 3 │ 3 │ - │ ╰─────────────────────────────────┴───────┴────────┴──────────╯ Local Api Metrics: ╭──────────────────────┬────────┬──────╮ │ Route │ Method │ Hits │ ├──────────────────────┼────────┼──────┤ │ /v1/decisions/stream │ GET │ 432 │ │ /v1/heartbeat │ GET │ 72 │ │ /v1/watchers/login │ POST │ 9 │ ╰──────────────────────┴────────┴──────╯ Local Api Machines Metrics: ╭──────────────────────────────────────────────────┬───────────────┬────────┬──────╮ │ Machine │ Route │ Method │ Hits │ ├──────────────────────────────────────────────────┼───────────────┼────────┼──────┤ │ bd854b54992945c7a769a8b2e2db5c77in5EoS25l5kmhqy0 │ /v1/heartbeat │ GET │ 72 │ ╰──────────────────────────────────────────────────┴───────────────┴────────┴──────╯ Local Api Bouncers Metrics: ╭────────────────────────────────┬──────────────────────┬────────┬──────╮ │ Bouncer │ Route │ Method │ Hits │ ├────────────────────────────────┼──────────────────────┼────────┼──────┤ │ cs-firewall-bouncer-1693480692 │ /v1/decisions/stream │ GET │ 432 │ ╰────────────────────────────────┴──────────────────────┴────────┴──────╯ Local Api Decisions: ╭────────────────────────────────────────────┬────────┬────────┬───────╮ │ Reason │ Origin │ Action │ Count │ ├────────────────────────────────────────────┼────────┼────────┼───────┤ │ crowdsecurity/fortinet-cve-2018-13379 │ CAPI │ ban │ 97 │ │ crowdsecurity/http-generic-bf │ CAPI │ ban │ 11 │ │ crowdsecurity/nginx-req-limit-exceeded │ CAPI │ ban │ 131 │ │ firehol_botscout_7d │ lists │ ban │ 6067 │ │ crowdsecurity/CVE-2019-18935 │ CAPI │ ban │ 36 │ │ crowdsecurity/spring4shell_cve-2022-22965 │ CAPI │ ban │ 1 │ │ crowdsecurity/thinkphp-cve-2018-20062 │ CAPI │ ban │ 29 │ │ crowdsecurity/CVE-2022-26134 │ CAPI │ ban │ 150 │ │ crowdsecurity/CVE-2022-41082 │ CAPI │ ban │ 867 │ │ crowdsecurity/CVE-2022-42889 │ CAPI │ ban │ 14 │ │ crowdsecurity/grafana-cve-2021-43798 │ CAPI │ ban │ 69 │ │ crowdsecurity/http-path-traversal-probing │ CAPI │ ban │ 56 │ │ crowdsecurity/http-sensitive-files │ CAPI │ ban │ 7 │ │ crowdsecurity/CVE-2022-35914 │ CAPI │ ban │ 49 │ │ crowdsecurity/http-backdoors-attempts │ CAPI │ ban │ 795 │ │ crowdsecurity/netgear_rce │ CAPI │ ban │ 23 │ │ crowdsecurity/CVE-2022-37042 │ CAPI │ ban │ 20 │ │ crowdsecurity/http-cve-2021-41773 │ CAPI │ ban │ 13 │ │ crowdsecurity/http-bad-user-agent │ CAPI │ ban │ 5458 │ │ crowdsecurity/apache_log4j2_cve-2021-44228 │ CAPI │ ban │ 372 │ │ crowdsecurity/http-probing │ CAPI │ ban │ 2003 │ │ crowdsecurity/jira_cve-2021-26086 │ CAPI │ ban │ 20 │ │ crowdsecurity/ssh-bf │ CAPI │ ban │ 4043 │ │ crowdsecurity/ssh-slow-bf │ CAPI │ ban │ 8 │ │ crowdsecurity/http-open-proxy │ CAPI │ ban │ 292 │ │ crowdsecurity/http-crawl-non_statics │ CAPI │ ban │ 415 │ │ crowdsecurity/f5-big-ip-cve-2020-5902 │ CAPI │ ban │ 21 │ ```

Related custom configs versions (if applicable) : notification plugins, custom scenarios, parsers etc.

No response

github-actions[bot] commented 1 year ago

@oyale: Thanks for opening an issue, it is currently awaiting triage.

In the meantime, you can:

  1. Check Crowdsec Documentation to see if your issue can be self resolved.
  2. You can also join our Discord.
  3. Check Releases to make sure your agent is on the latest version.
Details I am a bot created to help the [crowdsecurity](https://github.com/crowdsecurity) developers manage community feedback and contributions. You can check out my [manifest file](https://github.com/crowdsecurity/crowdsec/blob/master/.github/governance.yml) to understand my behavior and what I can do. If you want to use this for your project, you can check out the [BirthdayResearch/oss-governance-bot](https://github.com/BirthdayResearch/oss-governance-bot) repository.
oyale commented 1 year ago

I've realized that the difference was the response time in the log. I've managed to temporally fix it by adding:

  RESPONSETIME: '[0-9]+\.[0-9]+'

to the pattern_syntax and modifying the grok pattern by adding it:

      pattern: '(%{IPORHOST:target_fqdn} )?%{IPORHOST:remote_addr} - %{NGCUSTOMUSER:remote_user}? \[%{HTTPDATE:time_local}\] "%{WORD:verb} %{DATA:request} HTTP/%{NUMBER:http_version}" %{NUMBER:status} %{NUMBER:body_bytes_sent} (%{RESPONSETIME:response_time})? "%{NOTDQUOTE:http_referer}" "%{NOTDQUOTE:http_user_agent}"( %{NUMBER:request_length} %{NUMBER:request_time} \[%{DATA:proxy_upstream_name}\] \[%{DATA:proxy_alternative_upstream_name}\])?'

(both of them on /etc/crowdsec/parsers/s01-parse/nginx-logs.yaml).

I wonder what is the proper way to fix the expression because I guess those patches will be overridden in future updates.

Could you help me with that? Should I write a new parser?

LaurenceJJones commented 1 year ago

Never knew the response time could be a float, I will transfer to the hub which can handle this update.