search

crowdsecurity / hub

Main repository for crowdsec scenarios/parsers
https://hub.crowdsec.net
156 stars 147 forks source link

crowdsecurity/nginx-proxy-manager-logs parser failure #987

Closed ventra007 closed 6 months ago

ventra007 commented 6 months ago

Hi,

I'm getting the error "parser failure" when I run the following command: cscli explain --verbose --log '[27/Feb/2024:06:42:58 +0000] - 101 101 - GET https www.example.com "/websocket" [Client 10.0.0.1] [Length 2070] [Gzip -] [Sent-to 10.0.0.2] "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:123.0) Gecko/20100101 Firefox/123.0" "-"' --type nginx

The error output:

WARN[2024-02-28T08:09:46+02:00] Line 0/1 is missing evt.StrTime. It is most likely a mistake as it will prevent your logs to be processed in time-machine/forensic mode.
line: [27/Feb/2024:06:42:58 +0000] - 101 101 - GET https www.example.com "/websocket" [Client 10.0.0.1] [Length 2070] [Gzip -] [Sent-to 10.0.0.2] "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:123.0) Gecko/20100101 Firefox/123.0" "-"
        ├ s00-raw
        |       ├ 🔴 crowdsecurity/syslog-logs
        |       └ 🟢 crowdsecurity/non-syslog (+5 ~8)
        |               â”” update evt.ExpectMode : %!s(int=0) -> 1
        |               â”” update evt.Stage :  -> s01-parse
        |               â”” update evt.Line.Raw :  -> [27/Feb/2024:06:42:58 +0000] - 101 101 - GET https www.example.com "/websocket" [Client 10.0.0.1] [Length 2070] [Gzip -] [Sent-to 10.0.0.2] "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:123.0) Gecko/20100101 Firefox/123.0" "-"
        |               â”” update evt.Line.Src :  -> /tmp/cscli_explain1724764241/cscli_test_tmp.log
        |               â”” update evt.Line.Time : 0001-01-01 00:00:00 +0000 UTC -> 2024-02-28 06:09:43.185279311 +0000 UTC
        |               â”” create evt.Line.Labels.type : nginx
        |               â”” update evt.Line.Process : %!s(bool=false) -> true
        |               â”” update evt.Line.Module :  -> file
        |               â”” create evt.Parsed.message : [27/Feb/2024:06:42:58 +0000] - 101 101 - GET https www.example.com "/websocket" [Client 10.0.0.1] [Length 2070] [Gzip -] [Sent-to 10.0.0.2] "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:123.0) Gecko/20100101 Firefox/123.0" "-"
        |               â”” create evt.Parsed.program : nginx
        |               â”” update evt.Time : 0001-01-01 00:00:00 +0000 UTC -> 2024-02-28 06:09:43.185360471 +0000 UTC
        |               â”” create evt.Meta.datasource_type : file
        |               â”” create evt.Meta.datasource_path : /tmp/cscli_explain1724764241/cscli_test_tmp.log
        ├ s01-parse
        |       ├ 🔴 crowdsecurity/nginx-logs
        |       ├ 🔴 crowdsecurity/nginx-proxy-manager-logs
        |       └ 🔴 crowdsecurity/sshd-logs
        └-------- parser failure 🔴

I followed the instruction at https://doc.crowdsec.net/docs/next/getting_started/install_crowdsec and https://www.crowdsec.net/blog/crowdsec-with-nginx-proxy-manager

Running Ubuntu Server 22.04 

uname -a
Linux ubuntu 5.15.0-97-generic #107-Ubuntu SMP Wed Feb 7 13:26:48 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Any assistance will be greatly appreciated. Please let me know if you require other information.

LaurenceJJones commented 6 months ago

The type should be nginx-proxy-manager as defined in the collection

Here the details of using the correct type

``` $ cscli explain --log '[27/Feb/2024:06:42:58 +0000] - 101 101 - GET https www.example.com "/websocket" [Client 10.0.0.1] [Length 2070] [Gzip -] [Sent-to 10.0.0.2] "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:123.0) Gecko/20100101 Firefox/123.0" "-"' --type nginx-proxy-manager -v line: [27/Feb/2024:06:42:58 +0000] - 101 101 - GET https www.example.com "/websocket" [Client 10.0.0.1] [Length 2070] [Gzip -] [Sent-to 10.0.0.2] "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:123.0) Gecko/20100101 Firefox/123.0" "-" ├ s00-raw | └ 🟢 crowdsecurity/non-syslog (+5 ~8) | └ update evt.ExpectMode : %!s(int=0) -> 1 | └ update evt.Stage : -> s01-parse | └ update evt.Line.Raw : -> [27/Feb/2024:06:42:58 +0000] - 101 101 - GET https www.example.com "/websocket" [Client 10.0.0.1] [Length 2070] [Gzip -] [Sent-to 10.0.0.2] "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:123.0) Gecko/20100101 Firefox/123.0" "-" | └ update evt.Line.Src : -> /tmp/cscli_explain2569564032/cscli_test_tmp.log | └ update evt.Line.Time : 0001-01-01 00:00:00 +0000 UTC -> 2024-02-28 08:14:21.369152388 +0000 UTC | └ create evt.Line.Labels.type : nginx-proxy-manager | └ update evt.Line.Process : %!s(bool=false) -> true | └ update evt.Line.Module : -> file | └ create evt.Parsed.message : [27/Feb/2024:06:42:58 +0000] - 101 101 - GET https www.example.com "/websocket" [Client 10.0.0.1] [Length 2070] [Gzip -] [Sent-to 10.0.0.2] "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:123.0) Gecko/20100101 Firefox/123.0" "-" | └ create evt.Parsed.program : nginx-proxy-manager | └ update evt.Time : 0001-01-01 00:00:00 +0000 UTC -> 2024-02-28 08:14:21.369180324 +0000 UTC | └ create evt.Meta.datasource_type : file | └ create evt.Meta.datasource_path : /tmp/cscli_explain2569564032/cscli_test_tmp.log ├ s01-parse | └ 🟢 crowdsecurity/nginx-proxy-manager-logs (+22 ~2) | └ update evt.Stage : s01-parse -> s02-enrich | └ create evt.Parsed.body_bytes_sent : 2070 | └ create evt.Parsed.verb : GET | └ create evt.Parsed.http_referer : - | └ create evt.Parsed.request : /websocket | └ create evt.Parsed.target_fqdn : www.example.com | └ create evt.Parsed.target_server : 10.0.0.2 | └ create evt.Parsed.time_local : 27/Feb/2024:06:42:58 +0000 | └ create evt.Parsed.upstream_cache_status : - | └ create evt.Parsed.upstream_status : 101 | └ create evt.Parsed.gzip_ratio : - | └ create evt.Parsed.http_user_agent : Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:123.0) Gecko/20100101 Firefox/123.0 | └ create evt.Parsed.remote_addr : 10.0.0.1 | └ create evt.Parsed.scheme : https | └ create evt.Parsed.status : 101 | └ update evt.StrTime : -> 27/Feb/2024:06:42:58 +0000 | └ create evt.Meta.service : http | └ create evt.Meta.source_ip : 10.0.0.1 | └ create evt.Meta.target_fqdn : www.example.com | └ create evt.Meta.http_path : /websocket | └ create evt.Meta.http_status : 101 | └ create evt.Meta.http_verb : GET | └ create evt.Meta.http_user_agent : Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:123.0) Gecko/20100101 Firefox/123.0 | └ create evt.Meta.log_type : http_access-log ├ s02-enrich | ├ 🟢 crowdsecurity/dateparse-enrich (+2 ~2) | ├ create evt.Enriched.MarshaledTime : 2024-02-27T06:42:58Z | ├ update evt.Time : 2024-02-28 08:14:21.369180324 +0000 UTC -> 2024-02-27 06:42:58 +0000 UTC | ├ update evt.MarshaledTime : -> 2024-02-27T06:42:58Z | ├ create evt.Meta.timestamp : 2024-02-27T06:42:58Z | ├ 🟢 crowdsecurity/geoip-enrich (+9) | ├ create evt.Enriched.IsoCode : | ├ create evt.Enriched.Latitude : 0.000000 | ├ create evt.Enriched.Longitude : 0.000000 | ├ create evt.Enriched.ASNNumber : 0 | ├ create evt.Enriched.ASNOrg : | ├ create evt.Enriched.ASNumber : 0 | ├ create evt.Enriched.IsInEU : false | ├ create evt.Meta.ASNNumber : 0 | ├ create evt.Meta.IsInEU : false | ├ 🟢 crowdsecurity/http-logs (+7) | ├ create evt.Parsed.file_frag : websocket | ├ create evt.Parsed.file_name : websocket | ├ create evt.Parsed.file_dir : / | ├ create evt.Parsed.file_ext : | ├ create evt.Parsed.impact_completion : true | ├ create evt.Parsed.static_ressource : false | ├ create evt.Meta.http_args_len : 0 | ├ 🟢 crowdsecurity/jellyfin-whitelist (unchanged) | ├ 🟢 crowdsecurity/nextcloud-whitelist (unchanged) | └ 🟢 crowdsecurity/whitelists (~2 [whitelisted]) | └ update evt.Whitelisted : %!s(bool=false) -> true | └ update evt.WhitelistReason : -> private ipv4/ipv6 ip/ranges └-------- parser success, ignored by whitelist (private ipv4/ipv6 ip/ranges) 🟢 ```
ventra007 commented 6 months ago

Ok. I thought I tried that as well and got the same result, but perhaps when I tried to reinstall everything it came right. My initial issue was that testing the captcha and banning decisions wasn't working and thought the log parsing had something to do with it. I will test again and log a separate issue if it continues. Thank you.