Global override for parameters defined in scenario yaml files

[adam-ah]

What would you like to be added?

Completely new feature not currently available

For small / home servers, the usual usual capacity (5) with the usual leakspeed (10s) is rather stressful.

I would like to be able to globally override this (with minimums, if possible).

I want to be able to add to config.yaml something like this (best)

max_capacity = 2
min_leakspeed = 2s

this would only tighten the rules, so if someone already defined a blackhole of 120m, it would keep that.

Alternatively (if heaps easier as in, can get done by tomorrow), I want to be able to add to config.yaml:

capacity = 2
leakspeed = 2s

This would override all scenarios and would make these values instead of the default, rather lax limits.

Many thanks!

Why is this needed?

The default values might be good for collecting crowd-sourced IP information, by my servers are consistently attacked from IPs not on the list and getting hammered by these IPs sometimes. Reacting faster would help my tiny Lightsail instance not to get overwhelmed.

I'm not sure how to add /kind enhancement Better?

[github-actions[bot]]

@adam-ah: Thanks for opening an issue, it is currently awaiting triage.

In the meantime, you can:

1. Check Crowdsec Documentation to see if your issue can be self resolved.
2. You can also join our Discord.
3. Check Releases to make sure your agent is on the latest version.

Details

I am a bot created to help the [crowdsecurity](https://github.com/crowdsecurity) developers manage community feedback and contributions. You can check out my [manifest file](https://github.com/crowdsecurity/crowdsec/blob/master/.github/governance.yml) to understand my behavior and what I can do. If you want to use this for your project, you can check out the [BirthdayResearch/oss-governance-bot](https://github.com/BirthdayResearch/oss-governance-bot) repository.

[github-actions[bot]]

@adam-ah: There are no 'kind' label on this issue. You need a 'kind' label to start the triage process.

• /kind feature
• /kind enhancement
• /kind bug
• /kind packaging

Details

I am a bot created to help the [crowdsecurity](https://github.com/crowdsecurity) developers manage community feedback and contributions. You can check out my [manifest file](https://github.com/crowdsecurity/crowdsec/blob/master/.github/governance.yml) to understand my behavior and what I can do. If you want to use this for your project, you can check out the [BirthdayResearch/oss-governance-bot](https://github.com/BirthdayResearch/oss-governance-bot) repository.

[adam-ah]

/kind enhancement

[LaurenceJJones]

I think you are confusing black_hole property and decision property?

The blackhole property on scenario is purely to stop re triggering the scenario if the behaviour can be retriggered easily such as bad user agent https://docs.crowdsec.net/docs/next/scenarios/format#blackhole

The blackhole property is not how long the IP is banned for this is purely driven by your profiles.yaml

The other properties are purely dependent on the scenario so having a mass override can potentially cause alot of false positives dependant on the filter

[adam-ah]

I might be confusing them, @LaurenceJJones , time to fix to docos and/or rename the variable I guess? Can we please have the overrides for the other ones then?

[LaurenceJJones]

Can we please have the overrides for the other ones then?

The other properties are purely dependent on the scenario so having a mass override can potentially cause alot of false positives dependent on your setup, as en example http crawl non statics has a capacity of 40 because we want to detect crawlers that hit alot of endpoints

But if you override this to be 5 and you load a webpage that loads a bunch of static assets could easily trigger it

[LaurenceJJones]

time to fix to docos and/or rename the variable I guess?

Renaming would be a no go as it will undo 2 years of work, the documentation seems to inform you exactly what the property does.

We do provide a whole documentation about profiles and how it drives decisions.

[adam-ah]

documentation seems to inform you exactly what the property does.

some documentation somewhere might inform someone, I agree. For a new user, the documentation is overly complicated, scattered, and 99% irrelevant, despite someone(s) putting in tremendous effort to make them.

This is the only information I ever needed, which is not even part of the official doco https://www.crowdsec.net/blog/how-to-secure-your-raspberry-pi-with-crowdsec

We do provide a whole documentation about profiles and how it drives decisions.

As above. If in doubt, just check the repetitive questions on discourse, where all answers are pointing to some doco somewhere. It's a perfect case of lose-lose.

Renaming would be a no go as it will undo 2 years of work

I really hope no one actually invested 2 years into naming something as blackhole, hoping it would be obvious to all users what that might mean. The leaky bucket algorithm most definitely does not have a black hole concept.

But if you override this [...]

This is not how fail2ban works, and, therefore, it is a nonargument / surprising behaviour.

In fact, in fail2ban I can see on a single page what my rules are and set a default ban time and how aggressively I want to enforce them.

In crowdsec these rules are spread across two dozen files (times two, because there is a hub/scenarios and a scenarios folder as well. I'm sure some doco somewhere explains why this is essential)

I appreciate the concern of over-banning attacks, but users would appreciate even more mitigating attacks that are coming in 24/7. Crowdsec on my instance has not initiated an actual ban yet, despite tons of attempts and alerts, rendering crowdsec essentially useless for me (might be useful for you collecting data about these attempts, but not for me protecting the server).

If you are happy crowsec being useless for default installations, then the current state is perfect!

If you are thinking this is a me problem: https://www.reddit.com/r/CrowdSec/comments/153fd9x/default_threshold/ https://discourse.crowdsec.net/t/request-for-comments-parsers-scenarios-customization-in-the-crowdsec-agent/129 https://discourse.crowdsec.net/t/sshd-detection-not-working/431?page=2

[LaurenceJJones]

ome documentation somewhere might inform someone, I agree. For a new user, the documentation is overly complicated, scattered, and 99% irrelevant, despite someone(s) putting in tremendous effort to make them.

This is the only information I ever needed, which is not even part of the official doco https://www.crowdsec.net/blog/how-to-secure-your-raspberry-pi-with-crowdsec

As above. If in doubt, just check the repetitive questions on discourse, where all answers are pointing to some doco somewhere. It's a perfect case of lose-lose.

We agree, the documentation in its current form is more of a manual of properties not how a user would use it in "real life" so we have a started a process internally to improve on this.

I really hope no one actually invested 2 years into naming something as blackhole, hoping it would be obvious to all users what that might mean. The leaky bucket algorithm most definitely does not have a black hole concept.

Its not what I meant, its more that we have users adding scenarios to the hub for the past 2 years and if we rename we would have to go back and redo this (I know its a simple sed for the hub) but users local scenarios would break if they dont read changelogs.

In crowdsec these rules are spread across two dozen files (times two, because there is a hub/scenarios and a scenarios folder as well. I'm sure some doco somewhere explains why this is essential)

Not really the files within scenarios are just symlinks to hub/scenarios so editing one will edit the actual scenario file.

I appreciate the concern of over-banning attacks, but users would appreciate even more mitigating attacks that are coming in 24/7. Crowdsec on my instance has not initiated an actual ban yet, despite tons of attempts and alerts, rendering crowdsec essentially useless for me

Okay, then I can help you with that. Which scenarios / collections are you using? you can edit the scenario files and change the properties you outlined above to have less events before its poured.

We know that this will class the file as tainted and running cscli hub command will avoid updating this file due to your local changes. We are going to be working on a revision to cwhub (the lib we use between crowdsec and cscli) to allow these modifications without breaking updates to the files.

(might be useful for you collecting data about these attempts, but not for me protecting the server).

No that's not how the system works, if you get no alerts locally then we get no alerts. Hence we can edit the scenarios locally for your own threat model and can improve your protection.

If you are thinking this is a me problem

Never said this, as above I outlined what we know and how we will improve on the system.

[adam-ah]

Thanks for giving me background info on docs and upcoming work, @LaurenceJJones. Ping me if you want me to have a look at an updated doc from a naive user perspective.

Given the constraints you mentioned, I would propose allowing easy global overrides of rules from profiles.yaml, in line with fail2ban conventions (I'm still not sure why we need a config.yaml.local as well, but I guess these are all suggestions for another ticket).

overrides:
 http-sensitive-files:
  - type: trigger
  - capacity: 0
 another-plugin:
  - blah: meh

(I'm a bit puzzled why we have trigger if I must set capacity to 0 anyway: a bucket with capacity 0 is by definition a a trigger. But again, this could be the concern of another ticket)

Such an override would be easy to implement, would eliminate the need for the complicated tainted handling, and would be rather easy to eyeball what's happening in the system.

[LaurenceJJones]

Ping me if you want me to have a look at an updated doc from a naive user perspective.

I will do, since we work with the product daily we become blind to the pitfalls in the documentation.

(I'm a bit puzzled why we have trigger if I must set capacity to 0 anyway: a bucket with capacity 0 is by definition a a trigger. But again, this could be the concern of another ticket)

I believe it doesn't matter what you set capacity it will default to -1 but like you said can be an improvement if it cant be omitted

Given the constraints you mentioned, I would propose allowing easy global overrides of rules from profiles.yaml, in line with fail2ban conventions (I'm still not sure why we need a config.yaml.local as well, but I guess these are all suggestions for another ticket).

Once we have got the WAAP (Waf) project and cwhub rewrite out and stable, we can dive into this as you saw our CTO @buixor has mentioned we need an easier way to override than tainting file. If this is apart of the rewrite I dont know but I will keep you in the loop.

[adam-ah]

I believe it doesn't matter what you set capacity it will default to -1 but like you said can be an improvement if it cant be omitted

It bombs out (does not start) if I set it to trigger but capacity is set to non-0. Bug? Feature? I'm not sure:

time="14-10-2023 08:50:21" level=fatal msg="reload handler failure: unable to init crowdsec: while loading scenarios: scenario loading failed: loading of crowdsecurity/WHATEVER failed : invalid bucket from /etc/crowdsec/scenariosWHATEVER.yaml : trigger bucket must have 0 capacity"