false-positives due to nextcloud preview image 404s

crowdsecurity / hub

Main repository for crowdsec scenarios/parsers

https://hub.crowdsec.net

159 stars 147 forks source link

false-positives due to nextcloud preview image 404s #694

Closed chadsell closed 1 year ago

chadsell commented 1 year ago

The nextcloud-whitelist is not working properly for me, since I am still getting banned while file browsing through folders that contains files with no preview images.

Here's an example output from "cscli alerts inspect..."

- Date: 2023-03-28 19:36:35 -0700 -0700
╭───────────────┬───────────────────────────────────────────────────────────────────────────────────────────╮
│      Key      │                                           Value                                           │
├───────────────┼───────────────────────────────────────────────────────────────────────────────────────────┤
│ ASNNumber     │ 209                                                                                       │
├───────────────┼───────────────────────────────────────────────────────────────────────────────────────────┤
│ ASNOrg        │ CENTURYLINK-US-LEGACY-QWEST                                                               │
├───────────────┼───────────────────────────────────────────────────────────────────────────────────────────┤
│ IsInEU        │ false                                                                                     │
├───────────────┼───────────────────────────────────────────────────────────────────────────────────────────┤
│ IsoCode       │ US                                                                                        │
├───────────────┼───────────────────────────────────────────────────────────────────────────────────────────┤
│ SourceRange   │ 97.112.0.0/12                                                                             │
├───────────────┼───────────────────────────────────────────────────────────────────────────────────────────┤
│ http_args_len │ 75                                                                                        │
├───────────────┼───────────────────────────────────────────────────────────────────────────────────────────┤
│ http_path     │ /core/preview?fileId=30857&c=ab392b1c22d4aa00c68300fc4b2998f0&x=500&y=500&forceIcon=0&a=0 │
├───────────────┼───────────────────────────────────────────────────────────────────────────────────────────┤
│ http_status   │ 404                                                                                       │
├───────────────┼───────────────────────────────────────────────────────────────────────────────────────────┤
│ log_type      │ http_access-log                                                                           │
├───────────────┼───────────────────────────────────────────────────────────────────────────────────────────┤
│ service       │ http                                                                                      │
├───────────────┼───────────────────────────────────────────────────────────────────────────────────────────┤
│ source_ip     │ X.X.X.X                                                                                   │
╰───────────────┴───────────────────────────────────────────────────────────────────────────────────────────╯

And here's the output of cscli explain for one of the triggering log lines:


line: <my ip address> - - [28/Mar/2023:19:36:36 -0700] "GET /core/preview?fileId=30881&c=d8410df2495251c45a937c1e6c6e6a52&x=500&y=500&forceIcon=0&a=0 HTTP/1.1" 404 2706 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36"
        ├ s00-raw
        |       ├ 🟢 crowdsecurity/non-syslog (first_parser)
        |       └ 🔴 crowdsecurity/syslog-logs
        ├ s01-parse
        |       └ 🟢 crowdsecurity/apache2-logs (+18 ~2)
        ├ s02-enrich
        |       ├ 🟢 crowdsecurity/dateparse-enrich (+1 ~1)
        |       ├ 🟢 crowdsecurity/geoip-enrich (+13)
        |       ├ 🟢 crowdsecurity/http-logs (+8 ~1)
        |       ├ 🔴 crowdsecurity/naxsi-logs
        |       ├ 🟢 crowdsecurity/nextcloud-whitelist (unchanged)
        |       └ 🟢 crowdsecurity/whitelists (unchanged)
        ├-------- parser success 🟢
        ├ Scenarios
                ├ 🟢 crowdsecurity/http-crawl-non_statics
                └ 🟢 crowdsecurity/http-probing
'''

LaurenceJJones commented 1 year ago

The http_verb seems to be missing from the meta inside the example you provided. Could you re run the explain command but pass the -v flag to show what it parses.

chadsell commented 1 year ago

Sure, here's the verbose output of that same line.

line: <my ip address> - - [28/Mar/2023:19:36:36 -0700] "GET /core/preview?fileId=30881&c=d8410df2495251c45a937c1e6c6e6a52&x=500&y=500&forceIcon=0&a=0 HTTP/1.1" 404 2706 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36"
        ├ s00-raw
        |       ├ 🟢 crowdsecurity/non-syslog (first_parser)
        |       └ 🔴 crowdsecurity/syslog-logs
        ├ s01-parse
        |       └ 🟢 crowdsecurity/apache2-logs (+18 ~2)
        |               └ update evt.Stage : s01-parse -> s02-enrich
        |               └ create evt.Parsed.bytes : 2706
        |               └ create evt.Parsed.ident : -
        |               └ create evt.Parsed.rawrequest :
        |               └ create evt.Parsed.auth : -
        |               └ create evt.Parsed.target_fqdn :
        |               └ create evt.Parsed.verb : GET
        |               └ create evt.Parsed.httpversion : 1.1
        |               └ create evt.Parsed.referrer : "-"
        |               └ create evt.Parsed.request : /core/preview?fileId=30881&c=d8410df2495251c45a937c1e6c6e6a52&x=500&y=500&forceIcon=0&a=0
        |               └ create evt.Parsed.timestamp : 28/Mar/2023:19:36:36 -0700
        |               └ create evt.Parsed.clientip : <my ip address>
        |               └ create evt.Parsed.http_user_agent : "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36"        |               └ create evt.Parsed.response : 404
        |               └ update evt.StrTime :  -> 28/Mar/2023:19:36:36 -0700
        |               └ create evt.Meta.http_path : /core/preview?fileId=30881&c=d8410df2495251c45a937c1e6c6e6a52&x=500&y=500&forceIcon=0&a=0
        |               └ create evt.Meta.http_status : 404
        |               └ create evt.Meta.log_type : http_access-log
        |               └ create evt.Meta.service : http
        |               └ create evt.Meta.source_ip : <my ip address>
        ├ s02-enrich
        |       ├ 🟢 crowdsecurity/dateparse-enrich (+1 ~1)
        |               ├ create evt.Enriched.MarshaledTime : 2023-03-28T19:36:36-07:00
        |               ├ update evt.MarshaledTime :  -> 2023-03-28T19:36:36-07:00
        |       ├ 🟢 crowdsecurity/geoip-enrich (+13)
        |               ├ create evt.Enriched.ASNNumber : 209
        |               ├ create evt.Enriched.ASNOrg : CENTURYLINK-US-LEGACY-QWEST
        |               ├ create evt.Enriched.SourceRange : 97.112.0.0/12
        |               ├ create evt.Enriched.Longitude : <my longitude>
        |               ├ create evt.Enriched.ASNumber : 209
        |               ├ create evt.Enriched.IsInEU : false
        |               ├ create evt.Enriched.IsoCode : US
        |               ├ create evt.Enriched.Latitude : <my latitude>
        |               ├ create evt.Meta.ASNNumber : 209
        |               ├ create evt.Meta.ASNOrg : CENTURYLINK-US-LEGACY-QWEST
        |               ├ create evt.Meta.IsInEU : false
        |               ├ create evt.Meta.IsoCode : US
        |               ├ create evt.Meta.SourceRange : 97.112.0.0/12
        |       ├ 🟢 crowdsecurity/http-logs (+8 ~1)
        |               ├ update evt.Parsed.request : /core/preview?fileId=30881&c=d8410df2495251c45a937c1e6c6e6a52&x=500&y=500&forceIcon=0&a=0 -> /core/preview
        |               ├ create evt.Parsed.file_frag : preview
        |               ├ create evt.Parsed.http_args : fileId=30881&c=d8410df2495251c45a937c1e6c6e6a52&x=500&y=500&forceIcon=0&a=0
        |               ├ create evt.Parsed.file_name : preview
        |               ├ create evt.Parsed.static_ressource : false
        |               ├ create evt.Parsed.file_dir : /core/
        |               ├ create evt.Parsed.file_ext :
        |               ├ create evt.Parsed.impact_completion : false
        |               ├ create evt.Meta.http_args_len : 75
        |       ├ 🔴 crowdsecurity/naxsi-logs
        |       ├ 🟢 crowdsecurity/nextcloud-whitelist (unchanged)
        |       └ 🟢 crowdsecurity/whitelists (unchanged)
        ├-------- parser success 🟢
        ├ Scenarios
                ├ 🟢 crowdsecurity/http-crawl-non_statics
                └ 🟢 crowdsecurity/http-probing

chadsell commented 1 year ago

Have you had a chance to look at this, yet? I've got my home IP whitelisted separately, so I can manage files for now... but it's not a static IP, so it's still not ideal. Plus I can't go through my files while away from home without triggering a ban.

LaurenceJJones commented 1 year ago

Could you ensure all of your parsers are up to date, I have tested it with all current parsers and it is working as intended.

╰─λ sudo cscli explain --log 'mydomain.* 1.2.3.4 - - [07/Oct/2022:00:01:18 +0200] "GET /core/preview?fileId=30857&c=ab392b1c22d4aa00c68300fc4b2998f0&x=500&y=500&forceIcon=0&a=0 HTTP/2.0" 404 20 "https://myapp.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:105.0) Gecko/20100101 Firefox/105.0"' --type apache2 --only-successful-parsers -v
line: mydomain.* 1.2.3.4 - - [07/Oct/2022:00:01:18 +0200] "GET /core/preview?fileId=30857&c=ab392b1c22d4aa00c68300fc4b2998f0&x=500&y=500&forceIcon=0&a=0 HTTP/2.0" 404 20 "https://myapp.com/" "Mozilla/5.0 (WindowsNT 10.0; Win64; x64; rv:105.0) Gecko/20100101 Firefox/105.0"
├ s00-raw
|       ├ 🟢 crowdsecurity/non-syslog (+5 ~8)
|               ├ update evt.ExpectMode : %!s(int=0) -> 1
|               ├ update evt.Stage :  -> s01-parse
|               ├ update evt.Line.Raw :  -> mydomain.* 1.2.3.4 - - [07/Oct/2022:00:01:18 +0200] "GET /core/preview?fileId=30857&c=ab392b1c22d4aa00c68300fc4b2998f0&x=500&y=500&forceIcon=0&a=0 HTTP/2.0" 40420 "https://myapp.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:105.0) Gecko/20100101 Firefox/105.0"
|               ├ update evt.Line.Src :  -> /tmp/cscli_explain1641109772/cscli_test_tmp.log
|               ├ update evt.Line.Time : 0001-01-01 00:00:00 +0000 UTC -> 2023-04-12 12:02:46.976241306 +0000 UTC
|               ├ create evt.Line.Labels.type : apache2
|               ├ update evt.Line.Process : %!s(bool=false) -> true
|               ├ update evt.Line.Module :  -> file
|               ├ create evt.Parsed.message : mydomain.* 1.2.3.4 - - [07/Oct/2022:00:01:18 +0200] "GET /core/preview?fileId=30857&c=ab392b1c22d4aa00c68300fc4b2998f0&x=500&y=500&forceIcon=0&a=0 HTTP/2.0" 404 20 "https://myapp.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:105.0) Gecko/20100101 Firefox/105.0"
|               ├ create evt.Parsed.program : apache2
|               ├ update evt.Time : 0001-01-01 00:00:00 +0000 UTC -> 2023-04-12 12:02:46.976276509 +0000 UTC
|               ├ create evt.Meta.datasource_path : /tmp/cscli_explain1641109772/cscli_test_tmp.log
|               ├ create evt.Meta.datasource_type : file
├ s01-parse
|       └ 🟢 crowdsecurity/apache2-logs (+21 ~2)
|               └ update evt.Stage : s01-parse -> s02-enrich
|               └ create evt.Parsed.bytes : 20
|               └ create evt.Parsed.ident : -
|               └ create evt.Parsed.target_fqdn :
|               └ create evt.Parsed.auth : -
|               └ create evt.Parsed.http_user_agent : Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:105.0) Gecko/20100101 Firefox/105.0
|               └ create evt.Parsed.httpversion : 2.0
|               └ create evt.Parsed.referrer : https://myapp.com/
|               └ create evt.Parsed.clientip : 1.2.3.4
|               └ create evt.Parsed.port :
|               └ create evt.Parsed.rawrequest :
|               └ create evt.Parsed.response : 404
|               └ create evt.Parsed.verb : GET
|               └ create evt.Parsed.request : /core/preview?fileId=30857&c=ab392b1c22d4aa00c68300fc4b2998f0&x=500&y=500&forceIcon=0&a=0
|               └ create evt.Parsed.timestamp : 07/Oct/2022:00:01:18 +0200
|               └ update evt.StrTime :  -> 07/Oct/2022:00:01:18 +0200
|               └ create evt.Meta.http_path : /core/preview?fileId=30857&c=ab392b1c22d4aa00c68300fc4b2998f0&x=500&y=500&forceIcon=0&a=0
|               └ create evt.Meta.http_verb : GET
|               └ create evt.Meta.source_ip : 1.2.3.4
|               └ create evt.Meta.http_status : 404
|               └ create evt.Meta.http_user_agent : Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:105.0) Gecko/20100101 Firefox/105.0
|               └ create evt.Meta.log_type : http_access-log
|               └ create evt.Meta.service : http
├ s02-enrich
|       ├ 🟢 crowdsecurity/dateparse-enrich (+2 ~2)
|               ├ create evt.Enriched.MarshaledTime : 2022-10-07T00:01:18+02:00
|               ├ update evt.Time : 2023-04-12 12:02:46.976276509 +0000 UTC -> 2022-10-07 00:01:18 +0200 +0200
|               ├ update evt.MarshaledTime :  -> 2022-10-07T00:01:18+02:00
|               ├ create evt.Meta.timestamp : 2022-10-07T00:01:18+02:00
|       ├ 🟢 crowdsecurity/geoip-enrich (+10)
|               ├ create evt.Enriched.IsoCode : FR
|               ├ create evt.Enriched.Latitude : 48.832300
|               ├ create evt.Enriched.Longitude : 2.407500
|               ├ create evt.Enriched.ASNNumber : 0
|               ├ create evt.Enriched.ASNOrg :
|               ├ create evt.Enriched.ASNumber : 0
|               ├ create evt.Enriched.IsInEU : true
|               ├ create evt.Meta.ASNNumber : 0
|               ├ create evt.Meta.IsoCode : FR
|               ├ create evt.Meta.IsInEU : true
|       ├ 🟢 crowdsecurity/http-logs (+8 ~1)
|               ├ update evt.Parsed.request : /core/preview?fileId=30857&c=ab392b1c22d4aa00c68300fc4b2998f0&x=500&y=500&forceIcon=0&a=0 -> /core/preview
|               ├ create evt.Parsed.file_dir : /core/
|               ├ create evt.Parsed.file_ext :
|               ├ create evt.Parsed.impact_completion : false
|               ├ create evt.Parsed.file_frag : preview
|               ├ create evt.Parsed.file_name : preview
|               ├ create evt.Parsed.http_args : fileId=30857&c=ab392b1c22d4aa00c68300fc4b2998f0&x=500&y=500&forceIcon=0&a=0
|               ├ create evt.Parsed.static_ressource : false
|               ├ create evt.Meta.http_args_len : 75
|       ├ 🟢 crowdsecurity/nextcloud-whitelist (~2 [whitelisted])
|               ├ update evt.Whitelisted : %!s(bool=false) -> true
|               ├ update evt.WhitelistReason :  -> Nextcloud Whitelist
|       └ 🟢 crowdsecurity/whitelists (unchanged)
└-------- parser success, ignored by whitelist (Nextcloud Whitelist) 🟢

Do these in order:

cscli hub update
cscli hub upgrade #any tainted scenarios or parsers will not be upgraded if apache2 is tainted you need to run
# cscli parsers upgrade crowdsecurity/apache2-logs --force

Breaking down your output to this expression - evt.Meta.http_status == '404' && evt.Meta.http_verb == 'GET' && evt.Parsed.request == '/core/preview' && evt.Parsed.http_args contains 'x=' && evt.Parsed.http_args contains 'y=' && evt.Parsed.http_args contains 'fileId=' #File preview often 404s while searching

Seems like you parser may be 3 versions outdated.

chadsell commented 1 year ago

I ran the update and upgrade, but everything was already up to date. I've also confirmed that my nextcloud-whitelist file matches the current one on github and contains the lines that should work to fix this issue. Nevertheless, I continue to see the problem.

This time to test, I connected to a personal VPN I host on a private VPS (and which isn't in my IP whitelist) and scrolled through a nexctcloud directory with a few document files that don't have previews. I didn't scroll enough to get banned--just enough to get a log line to parse. Here's the result:

line: <X.X.X.X> - - [12/Apr/2023:10:12:36 -0700] "GET /index.php/apps/files/api/v1/thumbnail/256/256/Documents/SMS/SMS%20received%20(7).docx HTTP/1.1" 404 2827 "-" "Mozilla/5.0 (Android) Nextcloud-android/3.24.2"
        ├ s00-raw
        |       ├ 🟢 crowdsecurity/non-syslog (first_parser)
        |       └ 🔴 crowdsecurity/syslog-logs
        ├ s01-parse
        |       └ 🟢 crowdsecurity/apache2-logs (+18 ~2)
        |               └ update evt.Stage : s01-parse -> s02-enrich
        |               └ create evt.Parsed.http_user_agent : "Mozilla/5.0 (Android) Nextcloud-android/3.24.2"
        |               └ create evt.Parsed.ident : -
        |               └ create evt.Parsed.request : /index.php/apps/files/api/v1/thumbnail/256/256/Documents/SMS/SMS%20received%20(7).docx
        |               └ create evt.Parsed.target_fqdn :
        |               └ create evt.Parsed.verb : GET
        |               └ create evt.Parsed.auth : -
        |               └ create evt.Parsed.timestamp : 12/Apr/2023:10:12:36 -0700
        |               └ create evt.Parsed.httpversion : 1.1
        |               └ create evt.Parsed.rawrequest :
        |               └ create evt.Parsed.referrer : "-"
        |               └ create evt.Parsed.response : 404
        |               └ create evt.Parsed.clientip : X.X.X.X
        |               └ create evt.Parsed.bytes : 2827
        |               └ update evt.StrTime :  -> 12/Apr/2023:10:12:36 -0700
        |               └ create evt.Meta.log_type : http_access-log
        |               └ create evt.Meta.service : http
        |               └ create evt.Meta.source_ip : X.X.X.X
        |               └ create evt.Meta.http_path : /index.php/apps/files/api/v1/thumbnail/256/256/Documents/SMS/SMS%20received%20(7).docx
        |               └ create evt.Meta.http_status : 404
        ├ s02-enrich
        |       ├ 🟢 crowdsecurity/dateparse-enrich (+1 ~1)
        |               ├ create evt.Enriched.MarshaledTime : 2023-04-12T10:12:36-07:00
        |               ├ update evt.MarshaledTime :  -> 2023-04-12T10:12:36-07:00
        |       ├ 🟢 crowdsecurity/geoip-enrich (+13)
        |               ├ create evt.Enriched.ASNOrg : ORACLE-BMC-31898
        |               ├ create evt.Enriched.IsInEU : false
        |               ├ create evt.Enriched.Latitude : 37.237900
        |               ├ create evt.Enriched.Longitude : -121.794600
        |               ├ create evt.Enriched.SourceRange : X.X.0.0/16
        |               ├ create evt.Enriched.ASNNumber : 31898
        |               ├ create evt.Enriched.ASNumber : 31898
        |               ├ create evt.Enriched.IsoCode : US
        |               ├ create evt.Meta.ASNNumber : 31898
        |               ├ create evt.Meta.ASNOrg : ORACLE-BMC-31898
        |               ├ create evt.Meta.IsInEU : false
        |               ├ create evt.Meta.IsoCode : US
        |               ├ create evt.Meta.SourceRange : X.X.0.0/16
        |       ├ 🟢 crowdsecurity/http-logs (+7)
        |               ├ create evt.Parsed.static_ressource : false
        |               ├ create evt.Parsed.file_ext : .docx
        |               ├ create evt.Parsed.file_frag : SMS%20received%20(7)
        |               ├ create evt.Parsed.file_name : SMS%20received%20(7).docx
        |               ├ create evt.Parsed.file_dir : /index.php/apps/files/api/v1/thumbnail/256/256/Documents/SMS/
        |               ├ create evt.Parsed.impact_completion : false
        |               ├ create evt.Meta.http_args_len : 0
        |       ├ 🔴 crowdsecurity/naxsi-logs
        |       ├ 🟢 crowdsecurity/nextcloud-whitelist (unchanged)
        |       └ 🟢 crowdsecurity/whitelists (unchanged)
        ├-------- parser success 🟢
        ├ Scenarios
                ├ 🟢 crowdsecurity/http-crawl-non_statics
                └ 🟢 crowdsecurity/http-probing

LaurenceJJones commented 1 year ago

It's not the whitelist that needs updating its apache2-logs parser that seems to be out of date

Hence why in my previous comment I said check the apache2 parser or force update it

Edit: I realised it wasn't that obvious as I thought I made it.

chadsell commented 1 year ago

I'm going to check the file manually, but I ran the forced update command that you posted for the apache2-logs parser before posting my last reply. Sorry that I didn't make that clear!

chadsell commented 1 year ago

My copy of apache2-logs.yaml does NOT match the one found here: https://github.com/crowdsecurity/hub/blob/master/parsers/s01-parse/crowdsecurity/apache2-logs.yaml this despite the fact that I ran "cscli parsers upgrade crowdsecurity/apache2-logs --force"

Is that the one that it's supposed to be?

chadsell commented 1 year ago

I ran the forced update again, and here's what it says:

INFO[12-04-2023 11:38:18] crowdsecurity/apache2-logs : up-to-date
WARN[12-04-2023 11:38:18] crowdsecurity/apache2-logs : overwrite
updated crowdsecurity/apache2-logs
INFO[12-04-2023 11:38:18] 📦 crowdsecurity/apache2-logs : updated
INFO[12-04-2023 11:38:18] Upgraded 1 items
INFO[12-04-2023 11:38:18] Run 'sudo systemctl reload crowdsec' for the new configuration to be effective.

So I reloaded crowdsec again and ran the same line as before through "explain":

sudo cscli explain -v --log 'X.X.X.X - - [12/Apr/2023:10:12:36 -0700] "GET /index.php/apps/files/api/v1/thumbnail/256/256/Documents/SMS/SMS%20received%20(7).docx HTTP/1.1" 404 2827 "-" "Mozilla/5.0 (Android) Nextcloud-android/3.24.2"' --type apache2
line: X.X.X.X - - [12/Apr/2023:10:12:36 -0700] "GET /index.php/apps/files/api/v1/thumbnail/256/256/Documents/SMS/SMS%20received%20(7).docx HTTP/1.1" 404 2827 "-" "Mozilla/5.0 (Android) Nextcloud-android/3.24.2"
        ├ s00-raw
        |       ├ 🟢 crowdsecurity/non-syslog (first_parser)
        |       └ 🔴 crowdsecurity/syslog-logs
        ├ s01-parse
        |       └ 🟢 crowdsecurity/apache2-logs (+18 ~2)
        |               └ update evt.Stage : s01-parse -> s02-enrich
        |               └ create evt.Parsed.clientip : X.X.X.X
        |               └ create evt.Parsed.referrer : "-"
        |               └ create evt.Parsed.ident : -
        |               └ create evt.Parsed.rawrequest :
        |               └ create evt.Parsed.auth : -
        |               └ create evt.Parsed.httpversion : 1.1
        |               └ create evt.Parsed.target_fqdn :
        |               └ create evt.Parsed.timestamp : 12/Apr/2023:10:12:36 -0700
        |               └ create evt.Parsed.verb : GET
        |               └ create evt.Parsed.bytes : 2827
        |               └ create evt.Parsed.http_user_agent : "Mozilla/5.0 (Android) Nextcloud-android/3.24.2"
        |               └ create evt.Parsed.request : /index.php/apps/files/api/v1/thumbnail/256/256/Documents/SMS/SMS%20received%20(7).docx
        |               └ create evt.Parsed.response : 404
        |               └ update evt.StrTime :  -> 12/Apr/2023:10:12:36 -0700
        |               └ create evt.Meta.http_path : /index.php/apps/files/api/v1/thumbnail/256/256/Documents/SMS/SMS%20received%20(7).docx
        |               └ create evt.Meta.http_status : 404
        |               └ create evt.Meta.log_type : http_access-log
        |               └ create evt.Meta.service : http
        |               └ create evt.Meta.source_ip : X.X.X.X
        ├ s02-enrich
        |       ├ 🟢 crowdsecurity/dateparse-enrich (+1 ~1)
        |               ├ create evt.Enriched.MarshaledTime : 2023-04-12T10:12:36-07:00
        |               ├ update evt.MarshaledTime :  -> 2023-04-12T10:12:36-07:00
        |       ├ 🟢 crowdsecurity/geoip-enrich (+13)
        |               ├ create evt.Enriched.ASNNumber : 31898
        |               ├ create evt.Enriched.IsoCode : US
        |               ├ create evt.Enriched.Longitude : -121.794600
        |               ├ create evt.Enriched.Latitude : 37.237900
        |               ├ create evt.Enriched.SourceRange : X.X.0.0/16
        |               ├ create evt.Enriched.ASNOrg : ORACLE-BMC-31898
        |               ├ create evt.Enriched.ASNumber : 31898
        |               ├ create evt.Enriched.IsInEU : false
        |               ├ create evt.Meta.IsoCode : US
        |               ├ create evt.Meta.ASNNumber : 31898
        |               ├ create evt.Meta.ASNOrg : ORACLE-BMC-31898
        |               ├ create evt.Meta.IsInEU : false
        |               ├ create evt.Meta.SourceRange : X.X.0.0/16
        |       ├ 🟢 crowdsecurity/http-logs (+7)
        |               ├ create evt.Parsed.file_ext : .docx
        |               ├ create evt.Parsed.impact_completion : false
        |               ├ create evt.Parsed.static_ressource : false
        |               ├ create evt.Parsed.file_frag : SMS%20received%20(7)
        |               ├ create evt.Parsed.file_dir : /index.php/apps/files/api/v1/thumbnail/256/256/Documents/SMS/
        |               ├ create evt.Parsed.file_name : SMS%20received%20(7).docx
        |               ├ create evt.Meta.http_args_len : 0
        |       ├ 🔴 crowdsecurity/naxsi-logs
        |       ├ 🟢 crowdsecurity/nextcloud-whitelist (unchanged)
        |       └ 🟢 crowdsecurity/whitelists (unchanged)
        ├-------- parser success 🟢
        ├ Scenarios
                ├ 🟢 crowdsecurity/http-crawl-non_statics
                └ 🟢 crowdsecurity/http-probing

LaurenceJJones commented 1 year ago

What crowdsec version you running?

cscli version

chadsell commented 1 year ago

Here's the output of the -version command:

2023/04/12 11:48:46 version: v1.4.6-debian-pragmatic-5f71037b40c498045e1b59923504469e2b8d0140
2023/04/12 11:48:46 Codename: alphaga
2023/04/12 11:48:46 BuildDate: 2023-02-09_14:32:24
2023/04/12 11:48:46 GoVersion: 1.19.2
2023/04/12 11:48:46 Platform: linux
2023/04/12 11:48:46 Constraint_parser: >= 1.0, <= 2.0
2023/04/12 11:48:46 Constraint_scenario: >= 1.0, < 3.0
2023/04/12 11:48:46 Constraint_api: v1
2023/04/12 11:48:46 Constraint_acquis: >= 1.0, < 2.0

LaurenceJJones commented 1 year ago

Hmm and the output of

ls -la /etc/crowdsec/parsers/s01-parse/

Plus output of

grep -i hub /etc/crowdsec/config.yaml

chadsell commented 1 year ago

ls -la /etc/crowdsec/parsers/s01-parse/

lrwxrwxrwx 1 root root 71 Mar 26 14:58 apache2-logs.yaml -> /var/lib/crowdsec/hub/parsers/s01-parse/crowdsecurity/apache2-logs.yaml
lrwxrwxrwx 1 root root 70 Mar 26 14:58 cowrie-logs.yaml -> /var/lib/crowdsec/hub/parsers/s01-parse/crowdsecurity/cowrie-logs.yaml
lrwxrwxrwx 1 root root 71 Mar 26 14:58 dovecot-logs.yaml -> /var/lib/crowdsec/hub/parsers/s01-parse/crowdsecurity/dovecot-logs.yaml
lrwxrwxrwx 1 root root 72 Mar 26 14:58 iptables-logs.yaml -> /var/lib/crowdsec/hub/parsers/s01-parse/crowdsecurity/iptables-logs.yaml
lrwxrwxrwx 1 root root 70 Mar 26 14:58 modsecurity.yaml -> /var/lib/crowdsec/hub/parsers/s01-parse/crowdsecurity/modsecurity.yaml
lrwxrwxrwx 1 root root 69 Mar 26 14:58 mysql-logs.yaml -> /var/lib/crowdsec/hub/parsers/s01-parse/crowdsecurity/mysql-logs.yaml
lrwxrwxrwx 1 root root 69 Mar 26 15:50 nextcloud-logs.yaml -> /etc/crowdsec/hub/parsers/s01-parse/crowdsecurity/nextcloud-logs.yaml
lrwxrwxrwx 1 root root 69 Mar 26 15:41 nginx-logs.yaml -> /var/lib/crowdsec/hub/parsers/s01-parse/crowdsecurity/nginx-logs.yaml
lrwxrwxrwx 1 root root 71 Mar 26 14:58 postfix-logs.yaml -> /var/lib/crowdsec/hub/parsers/s01-parse/crowdsecurity/postfix-logs.yaml
lrwxrwxrwx 1 root root 74 Mar 26 14:58 postscreen-logs.yaml -> /var/lib/crowdsec/hub/parsers/s01-parse/crowdsecurity/postscreen-logs.yaml
lrwxrwxrwx 1 root root 67 Mar 26 14:58 smb-logs.yaml -> /var/lib/crowdsec/hub/parsers/s01-parse/crowdsecurity/smb-logs.yaml
lrwxrwxrwx 1 root root 68 Mar 26 14:58 sshd-logs.yaml -> /var/lib/crowdsec/hub/parsers/s01-parse/crowdsecurity/sshd-logs.yaml
lrwxrwxrwx 1 root root 71 Mar 26 14:58 tcpdump-logs.yaml -> /var/lib/crowdsec/hub/parsers/s01-parse/crowdsecurity/tcpdump-logs.yaml
lrwxrwxrwx 1 root root 70 Mar 26 14:58 vsftpd-logs.yaml -> /var/lib/crowdsec/hub/parsers/s01-parse/crowdsecurity/vsftpd-logs.yaml

grep -i hub /etc/crowdsec/config.yaml:

  hub_dir: /etc/crowdsec/hub/
  index_path: /etc/crowdsec/hub/.index.json

LaurenceJJones commented 1 year ago

Hmm if you

cat /etc/crowdsec/hub/parsers/s01-parse/crowdsecurity/apache2-logs.yaml

Is that the latest version? I don't see why your symlinks are going to /var/lib/

You can even see your nextcloud whitelist is correct path. Is this install been around for a while? If so I would say you need to manually remove the parsers and then install again

LaurenceJJones commented 1 year ago

Think we figured out what happened you install via debian repo > install our repo -> installed newer crowdsec

Debian hub path is /var/lib/, however, main repo is /etc/hub/ so this means if you update you will stop getting updates to parsers. Please run these commands:

sudo cscli -oraw parsers list | grep -v ^name |cut -d',' -f1 > /tmp/crowdsec_p.lst
for i in $(cat /tmp/crowdsec_p.lst);do sudo cscli parsers remove "$i";done
for i in $(cat /tmp/crowdsec_p.lst);do sudo cscli parsers install "$i";done
sudo cscli -oraw scenarios list | grep -v ^name |cut -d',' -f1 > /tmp/crowdsec_s.lst
for i in $(cat /tmp/crowdsec_s.lst);do sudo cscli scenarios remove "$i";done
for i in $(cat /tmp/crowdsec_s.lst);do sudo cscli scenarios install "$i";done
sudo systemctl restart crowdsec
rm /tmp/crowdsec_{p,s}.lst

In short this will remove and install all parser and scenarios you have installed. (This will force the symlinks to be updated to the new location)

chadsell commented 1 year ago

Ah, that makes sense. I definitely installed that way, since the debian repo version that I installed at first wouldn't install the nextcloud collection.

I ran the commands, but the one to remove the parsers doesn't seem to work. Here's an example pair of lines from the output:

WARN[13-04-2023 08:19:29] crowdsecurity/apache2-logs (/etc/crowdsec/parsers/s01-parse/apache2-logs.yaml) isn't a symlink to /etc/crowdsec/hub/parsers/s01-parse/crowdsecurity/apache2-logs.yaml
FATA[13-04-2023 08:19:29] unable to disable crowdsecurity/apache2-logs : crowdsecurity/apache2-logs isn't managed by hub

The nextcloud parsers were removed properly, but all of the ones that came with the debian repo installation had these two error lines.

chadsell commented 1 year ago

I went ahead and nuked my install of crowdsec and reinstalled. (My nextcloud server is accessible via a cloudflare tunnel. I temporarily closed the tunnel while working so I wouldn't have to worry about security issues due to crowdsec being removed and re-setup)

I did:

sudo apt remove crowdsec
# then realized I probably needed 
sudo apt remove crowdsec --purge # to really make sure everything was a clean slate.
sudo rm -R /etc/crowdsec
sudo apt install crowdsec
sudo cscli hub update
sudo cscli collections install crowdsecurity/nextcloud
sudo systemctl reload crowdsec

and now when I run ls -la /etc/crowdsec/parsers/s01-parse/ I get

total 3
drwxr-xr-x 2 root root  5 Apr 13 09:15 .
drwxr-xr-x 5 root root  5 Apr 13 09:09 ..
lrwxrwxrwx 1 root root 67 Apr 13 09:09 apache2-logs.yaml -> /etc/crowdsec/hub/parsers/s01-parse/crowdsecurity/apache2-logs.yaml
lrwxrwxrwx 1 root root 69 Apr 13 09:15 nextcloud-logs.yaml -> /etc/crowdsec/hub/parsers/s01-parse/crowdsecurity/nextcloud-logs.yaml
lrwxrwxrwx 1 root root 64 Apr 13 09:09 sshd-logs.yaml -> /etc/crowdsec/hub/parsers/s01-parse/crowdsecurity/sshd-logs.yaml

So it looks like everything is pointing to the correct directories, now? Plus now I don't have all of the extra parsers/collections that came with the debian repo install and that don't really apply to my setup. Although now I need to double check that I definitely didn't really need all of those!

I've got some other things to work on while my tunnel is down, but once it's back up I'll test the 404 scenario again and make sure everything is operating as expected.

chadsell commented 1 year ago

OK, so that didn't work. I brought my tunnel back up and tested the "nextcloud 404" issue via my VPN/VPS IP. And again, it flagged

line: X.X.X.X - - [13/Apr/2023:09:59:07 -0700] "GET /index.php/apps/files/api/v1/thumbnail/256/256/Documents/ecobee-reports/Living%20Room.xlsx HTTP/1.1" 404 2827 "-" "Mozilla/5.0 (Android) Nextcloud-android/3.24.2"
        ├ s00-raw
        |       ├ 🟢 crowdsecurity/non-syslog (first_parser)
        |       └ 🔴 crowdsecurity/syslog-logs
        ├ s01-parse
        |       └ 🟢 crowdsecurity/apache2-logs (+21 ~2)
        |               └ update evt.Stage : s01-parse -> s02-enrich
        |               └ create evt.Parsed.response : 404
        |               └ create evt.Parsed.http_user_agent : Mozilla/5.0 (Android) Nextcloud-android/3.24.2
        |               └ create evt.Parsed.ident : -
        |               └ create evt.Parsed.port :
        |               └ create evt.Parsed.rawrequest :
        |               └ create evt.Parsed.request : /index.php/apps/files/api/v1/thumbnail/256/256/Documents/ecobee-reports/Living%20Room.xlsx
        |               └ create evt.Parsed.bytes : 2827
        |               └ create evt.Parsed.clientip : X.X.X.X
        |               └ create evt.Parsed.target_fqdn :
        |               └ create evt.Parsed.auth : -
        |               └ create evt.Parsed.timestamp : 13/Apr/2023:09:59:07 -0700
        |               └ create evt.Parsed.httpversion : 1.1
        |               └ create evt.Parsed.referrer : -
        |               └ create evt.Parsed.verb : GET
        |               └ update evt.StrTime :  -> 13/Apr/2023:09:59:07 -0700
        |               └ create evt.Meta.service : http
        |               └ create evt.Meta.source_ip : X.X.X.X
        |               └ create evt.Meta.http_user_agent : Mozilla/5.0 (Android) Nextcloud-android/3.24.2
        |               └ create evt.Meta.http_verb : GET
        |               └ create evt.Meta.http_path : /index.php/apps/files/api/v1/thumbnail/256/256/Documents/ecobee-reports/Living%20Room.xlsx
        |               └ create evt.Meta.http_status : 404
        |               └ create evt.Meta.log_type : http_access-log
        ├ s02-enrich
        |       ├ 🟢 crowdsecurity/dateparse-enrich (+2 ~1)
        |               ├ create evt.Enriched.MarshaledTime : 2023-04-13T09:59:07-07:00
        |               ├ update evt.MarshaledTime :  -> 2023-04-13T09:59:07-07:00
        |               ├ create evt.Meta.timestamp : 2023-04-13T09:59:07-07:00
        |       ├ 🟢 crowdsecurity/geoip-enrich (+13)
        |               ├ create evt.Enriched.ASNOrg : ORACLE-BMC-31898
        |               ├ create evt.Enriched.IsInEU : false
        |               ├ create evt.Enriched.ASNNumber : 31898
        |               ├ create evt.Enriched.ASNumber : 31898
        |               ├ create evt.Enriched.IsoCode : US
        |               ├ create evt.Enriched.Latitude : 37.237900
        |               ├ create evt.Enriched.Longitude : -121.794600
        |               ├ create evt.Enriched.SourceRange : X.X.0.0/16
        |               ├ create evt.Meta.ASNNumber : 31898
        |               ├ create evt.Meta.IsoCode : US
        |               ├ create evt.Meta.SourceRange : X.X.0.0/16
        |               ├ create evt.Meta.ASNOrg : ORACLE-BMC-31898
        |               ├ create evt.Meta.IsInEU : false
        |       ├ 🟢 crowdsecurity/http-logs (+7)
        |               ├ create evt.Parsed.file_ext : .xlsx
        |               ├ create evt.Parsed.static_ressource : false
        |               ├ create evt.Parsed.file_dir : /index.php/apps/files/api/v1/thumbnail/256/256/Documents/ecobee-reports/
        |               ├ create evt.Parsed.file_frag : Living%20Room
        |               ├ create evt.Parsed.file_name : Living%20Room.xlsx
        |               ├ create evt.Parsed.impact_completion : false
        |               ├ create evt.Meta.http_args_len : 0
        |       ├ 🟢 crowdsecurity/nextcloud-whitelist (unchanged)
        |       └ 🟢 crowdsecurity/whitelists (unchanged)
        ├-------- parser success 🟢
        ├ Scenarios
                ├ 🟢 crowdsecurity/http-crawl-non_statics
                └ 🟢 crowdsecurity/http-probing

I verified that the apache2-logs.yaml matches the current version on github. And now the symlinks are correct.

I noticed that there is again a directory at /var/lib/crowdsec. But no parsers in there, this time. Instead, there's another directory called "data" which contains the following files:

chad@nextcloudpi:/var/lib/crowdsec/data$ ls
GeoLite2-ASN.mmdb   bad_user_agents.regex.txt  crowdsec.db-wal          log4j2_cve_2021_44228.txt  thinkphp_cve_2018-20062.txt
GeoLite2-City.mmdb  crowdsec.db                http_path_traversal.txt  sensitive_data.txt         xss_probe_patterns.txt
backdoors.txt       crowdsec.db-shm            jira_cve_2021-26086.txt  sqli_probe_patterns.txt

Is this normal for the crowdsec repo install? Or am I still plagues by the ghost of the debian repo?

LaurenceJJones commented 1 year ago

That is normal we use /var/lib/crowdsec/data/ to host the data files. So the log line you showing us isnt whitelisted at the moment before you was saying about nextcloud pictures. This is /index.php/apps/files

chadsell commented 1 year ago

Ah, gotcha. I thought there was already a whitelist rule for non-image file previews/thumbnails, since many files aren't expected to have previews but nextcloud still tries to load one and throws a 404, anyway... but it looks like you're already on that, now!

I'm not sure I even have any photos left that don't have previews, at the moment. When I scroll through a photo directory while tailing nc-access.log, all I see are 200s, no 404s. I'll do a little digging around and maybe create a test directory that hasn't had previews generated yet, in order to test.

Thanks for your help with this!

LaurenceJJones commented 1 year ago

@chadsell I have pushed an update to the whitelist to catch that thumbnail 404 for files. You can run cscli hub update and cscli hub upgrade

chadsell commented 1 year ago

Sweet, thanks! That was fast! I updated and now that same line results with "parser failure", which I guess is the "right" kind of failure, in this case.

...
        |       ├ 🟢 crowdsecurity/nextcloud-whitelist (~2 [whitelisted])
        |               ├ update evt.Whitelisted : %!s(bool=false) -> true
        |               ├ update evt.WhitelistReason :  -> Nextcloud Whitelist
        |       └ 🟢 crowdsecurity/whitelists (unchanged)
        └-------- parser failure 🔴

LaurenceJJones commented 1 year ago

In the latest version of crowdsec (1.5) coming soon the message is way more clear.

|       ├ 🟢 crowdsecurity/nextcloud-whitelist (~2 [whitelisted])
|               ├ update evt.Whitelisted : %!s(bool=false) -> true
|               ├ update evt.WhitelistReason :  -> Nextcloud Whitelist
|       └ 🟢 crowdsecurity/whitelists (unchanged)
└-------- parser success, ignored by whitelist (Nextcloud Whitelist) 🟢