Closed laurentgoudet closed 8 months ago
github-actions[bot]
commented
8 months ago @laurentgoudet: Thanks for opening an issue, it is currently awaiting triage.
In the meantime, you can:
LaurenceJJones
commented
8 months ago Could you provide the configuration file? I know you had a previous issue so it would be interesting to know if you increase the routines including bucket routines.
Edit: It also might help for us to know what type of applications you are proxying via nginx, because there may be some applications that causes a lot of log lines (web socket, long/short polling) these log lines will cause shorter life buckets to stay alive
buixor
commented
8 months ago Hello @laurentgoudet !
(thanks again for the very detailed issue)
Looking at what you said, specifically:
The weird thing is that cs_bucket_underflowed_total reports those are underflowed, but cs_buckets keeps increasing.
And looking at the code, the cs_buckets metrics gets decreased when the routine exits while the underflow one gets increased a bit before. What might look plausible is that the routine gets stuck trying to send the underflow information back (ie. leaky.AllOut is congested), however, this sounds quite unusual.
How many parser/buckets/output routines do you have? (output_routines would be the one in charge of consuming the leaky.AllOut channel). I suppose for this to happen, you might either need to produce a massive amount of alerts, and/or have postoverflows that do DNS resolutions or something expensive (that would slow down the ability to empty this queue).
edit: @LaurenceJJones told me that it seems from your other discussion that you are using reverse dns postoverflows, and unfortunately this mechanism doesn't have any cache and will be super duper slow - especially if you generate a lot of events. Can you try to disable it for now? It sounds like a valid suspect.
laurentgoudet
commented
8 months ago Could you provide the configuration file? I know you had a previous issue so it would be interesting to know if you increase the routines including bucket routines.
Edit: It also might help for us to know what type of applications you are proxying via nginx, because there may be some applications that causes a lot of log lines (web socket, long/short polling) these log lines will cause shorter life buckets to stay alive
Sure. I haven't done a lot of customization yet, so the whole config is basically (environment-specific certs are omitted):
crowdsec_agent:
image: 'crowdsecurity/crowdsec'
image_tag: 'v1.5.5'
environment:
LEVEL_INFO: true
COLLECTIONS: 'crowdsecurity/nginx crowdsecurity/whitelist-good-actors'
PARSERS: 'crowdsecurity/geoip-enrich crowdsecurity/dateparse-enrich'
DISABLE_PARSERS: 'crowdsecurity/cri-logs crowdsecurity/docker-logs'
DISABLE_SCENARIOS: 'crowdsecurity/http-bad-user-agent'
DISABLE_LOCAL_API: true
LOCAL_API_URL: "https://127.0.0.1:7671"
USE_TLS: true
CACERT_FILE: /etc/crowdsec/ca_bundle.pem
CLIENT_CERT_FILE: /etc/crowdsec/agent.pem
CLIENT_KEY_FILE: /etc/crowdsec/agent-key.pem
files:
/etc/crowdsec/config.yaml.local:
format: 'yaml'
content:
common:
log_media: 'file'
log_dir: /var/log/
log_max_size: 10
log_max_age: 7
log_max_files: 3
compress_logs: true
/etc/crowdsec/acquis.yaml:
format: 'yaml'
content:
source: 'file'
filenames:
- '/logs/nginx/access.log'
labels:
type: 'nginx'
/staging/etc/crowdsec/parsers/s02-enrich/flnltd-internal-ips-whitelist.yaml:
format: 'yaml'
content:
name: crowdsecurity/whitelists
description: Whitelist events from internal IP addresses
whitelist:
reason: internal ip ranges
expression:
- |
evt.Meta.source_ip in [
<redacted, list in internal NAT gateways>,
] && evt.Parsed.http_user_agent != 'crowdsec'
/staging/etc/crowdsec/parsers/s02-enrich/flnltd-ignore-403s.yaml:
format: 'yaml'
content:
name: crowdsecurity/whitelists
description: Ignore 403s
whitelist:
reason: ignore 403s as we use that status code for blocked sources
expression:
- evt.Parsed.status == '403'
/staging/etc/crowdsec/scenarios/flnltd-http-bruteforce-login.yaml:
format: 'yaml'
content:
type: leaky
name: flnltd/http-bruteforce-login
description: "Detect advanced brute-force patterns on the login endpoint"
filter: "evt.Parsed.request == '<redacted>' && evt.Parsed.status == '401'"
groupby: evt.Meta.SourceRange
capacity: 10
leakspeed: "1m"
blackhole: 10m
labels:
service: http
type: bruteforce
remediation: true
scope:
type: Range
expression: evt.Meta.SourceRange
/staging/etc/crowdsec/parsers/s02-enrich/flnltd-gptbot-whitelist.yaml:
format: 'yaml'
content:
name: crowdsecurity/whitelists
description: "Whitelist OpenAI GPTBot"
whitelist:
reason: "OpenAI GPTbot triggers crowdsecurity/http-probing when crawling the site"
expression:
- "any(File('gptbot-ranges.txt'), { len(#) > 0 && IpInRange(evt.Meta.source_ip ,#)})"
data:
- source_url: https://openai.com/gptbot-ranges.txt
dest_file: gptbot-ranges.txt
type: string
# This is needed as for now the data external source above is only used when the parser is installed from the hub
# https://docs.crowdsec.net/docs/next/parsers/format/#data
/var/lib/crowdsec/data/gptbot-ranges.txt:
format: 'raw'
content: |
52.230.152.0/24
52.233.106.0/24
volumes:
- guest: '/logs/nginx:ro'
host: '/mnt/logs/nginx'
- guest: '/var/log'
host: '/mnt/logs'
ports:
# Prometheus metrics
- host: 6060
guest: 6060How many parser/buckets/output routines do you have? (
output_routineswould be the one in charge of consuming theleaky.AllOutchannel). I suppose for this to happen, you might either need to produce a massive amount of alerts, and/or have postoverflows that do DNS resolutions or something expensive (that would slow down the ability to empty this queue).
I have not customized output_routines, so it'd be the default value.
edit: @LaurenceJJones told me that it seems from your other discussion that you are using reverse dns postoverflows, and unfortunately this mechanism doesn't have any cache and will be super duper slow - especially if you generate a lot of events. Can you try to disable it for now? It sounds like a valid suspect.
Yes I'm using crowdsecurity/whitelist-good-actors with uses rdns. I could switch to whitelisting public IP ranges though (e.g. https://developers.google.com/static/search/apis/ipranges/googlebot.json, https://www.bing.com/toolbox/bingbot.json..), since I can understand how running RDNS queries at the edge is not ideal.
Having said that looking at overflows during the incident their number were still very low, i.e. if I am reading that thing right it peaked at 1 overflow every 10s (per host):
If I assume that cs_bucket_overflowed_total contains the overflows later discarded by crowdsecurity/whitelist-good-actors (since it's called "post" overflow), then the number of reverse dns queries seem really minimal, i.e. I am not sure if it's the good suspect?
The other issue I am facing with the deployment is https://github.com/crowdsecurity/crowdsec/issues/2669 so possibly both are somehow related, although not sure how.
laurentgoudet
commented
8 months ago Nothing in the CrowdSec logs during the event/when expired buckets started not to be cleaned up anymore. Also the attack triggered ~2k alerts, which does not seem that many to me.
Actually spoke too soon, by bad. Looking back at the CrowdSec logs, it seems that the issue comes from my custom flnltd/http-bruteforce-login scenario which does:
scope:
type: Range
expression: evt.Meta.SourceRangeFor some reason - probably when the entry is missing from the GeoLite2 DB - the source range can be empty, as per the Range performed 'flnltd/http-bruteforce-login' (11 events over 40.366500614s) at 2023-12-21 23:29:25.593189181 +0000 UTC logs below, which align with our External LB going out of memory:
When that happens, the agents seems be retrying to send the alert to the LAPI a large number of times (100000?), which I am guessing causes the output goroutines to be stuck, as per the stuck for 148.510521ms sending event to 2355c624f9a2271765356bf0b9c1726d81215572 (sigclosed:1 failed_sent:99998 attempts:100000) logs below:
I guess the 100000 retry figure might be a bit extreme? In the meantime I can probably work around the issue with filter: evt.Meta.SourceRange - I couldn't pin down which IP triggers that though.
buixor
commented
8 months ago Seems to be a good suspect. Will try to reproduce on our side and keep you posted :)
laurentgoudet
commented
8 months ago In the meantime I can probably work around the issue with
filter: evt.Meta.SourceRange
I've deployed that workaround on a canary host & CrowdSec has been running fine so far, although most of the bad actors still seem to be on holiday 🏖️.
I'll keep monitoring it but that GH issue can probably be closed now, as the root cause is tracked in https://github.com/crowdsecurity/crowdsec/issues/2687.
What happened?
Our edge load balancers ran out of memory, due to the CrowdSec process memory usage growing unbounded:
Upon looking into why, it seems that the
crowdsecurity/http-crawl-non_staticsbuckets count started to creep up:The bucket creation rate increased a bit at the time due to an ongoing attack:
The attack was a brute force attack against our login endpoint, so the IPs were not being reused and
crowdsecurity/http-crawl-non_staticshas a short leak speed (0.5s), so the underflow rate closely tracks the creation rate:Despite that it looks like as if the underflowed buckets were not being cleaned up as the total count just keeps increasing through the roof.
The issue was not limited to that specific scenario - all buckets started to not being cleaned up anymore, including the ones that were not triggered by the attack:
In short, it feels like CrowdSec was "stuck" cleaning up underflowed buckets. The weird thing is that
cs_bucket_underflowed_totalreports those are underflowed, butcs_bucketskeeps increasing.Nothing in the CrowdSec logs during the event/when expired buckets started not to be cleaned up anymore. Also the attack triggered ~2k alerts, which does not seem that many to me.
The same issue happened 3 times during the last 2 days so we had to turn off CrowdSec for now. Any idea of what might be going on/how to debug it further?
What did you expect to happen?
CrowdSec to survive attacks
How can we reproduce it (as minimally and precisely as possible)?
Not sure what triggers the behavior
Anything else we need to know?
No response
Crowdsec version
OS version
Enabled collections and parsers
Acquisition config
Config show
Prometheus metrics
Related custom configs versions (if applicable) : notification plugins, custom scenarios, parsers etc.
No response