search

elastic / kibana

Your window into the Elastic Stack
https://www.elastic.co/products/kibana
Other
19.82k stars 8.2k forks source link

[Security Solution] Detection Rules' max_signals parameter is not respected beyond 1000 #157313

Open rylnd opened 1 year ago

rylnd commented 1 year ago

Describe the bug:

Kibana/Elasticsearch Stack version: 8.8

Steps to reproduce:

  1. Create a rule with max_signals set to 10,000
  2. Execute the rule such that it would create >1000 alerts
  3. Observe the rule fail

Current behavior: Up to 1000 alerts are generated, and the rule has an execution error. max_signals is not respected, and the default alerting value of 1000 is used to cap/validate the number of signals.

Expected behavior: Up to 10,000 alerts are generated. If that number is exceeded, a warning is created on the rule.

Screenshots (if relevant):

Screenshot 2023-05-10 at 1 27 38 PM

Related Issues

While not confirmed, I believe that the work done for https://github.com/elastic/kibana/issues/153611 introduced this behavior.

Slak conversation

https://elastic.slack.com/archives/CN7AD3605/p1683741078780019

elasticmachine commented 1 year ago

Pinging @elastic/security-solution (Team: SecuritySolution)

elasticmachine commented 1 year ago

Pinging @elastic/security-detections-response (Team:Detections and Resp)

e40pud commented 1 year ago

We added docs about this behaviour https://github.com/elastic/security-docs/pull/3293

@yctercero I guess we can consider options to fix this in 8.9?

yctercero commented 1 year ago

It'd be great to get to this one 8.9 if possible. I'm not sure how many users are setting their max signals that high - maybe we have the telemetry @peluja1012 ? But would still be great to understand.

yctercero commented 4 months ago

@banderror I think your team is dealing with this in the current prebuilt rule customization work - is that accurate?

banderror commented 4 months ago

@yctercero Yes and no.

We worked on https://github.com/elastic/kibana/issues/173593 and fixed https://github.com/elastic/kibana/issues/164234. We also noticed the issue described in this ticket and had some discussion with PM and TRaDE. Summary of the discussion can be found here. Immediate action items we've already addressed were:

  • We agreed that it would be worth documenting that those rules that have max_signals > xpack.alerting.rules.run.alerts.max will be generating at most xpack.alerting.rules.run.alerts.max alerts per execution (1000 by default).
    • @joepeeples documented this in the public docs (PR).
    • @Mikaayenson updated setup guides of those 18 rules with that information (PR).

No more work related to this is planned within the rule customization epic.

But I agree that it's an issue and non-friendly and obscure UX. I'm just unsure that it should be handled as described in the expected behavior section in this ticket. The xpack.alerting.rules.run.alerts.max setting seems to be a hard cap by design, because it's a global circuit breaker. We could handle this issue by changing the UX of installing and enabling rules. If we do it like that, the issue will be on the Rule Management side.

cc @approksiu

elasticmachine commented 4 months ago

Pinging @elastic/security-detection-rule-management (Team:Detection Rule Management)

yctercero commented 4 months ago

Thanks for the quick response @banderror !

We could handle this issue by changing the UX of installing and enabling rules. If we do it like that, the issue will be on the Rule Management side.

By this, do you mean just updating the UI to add validation so users can't set anything above xpack.alerting.rules.run.alerts.max and updating the prebuilt rules to be below that threshold? I know it could be a moving target since it's a setting but perhaps setting the prebuilt rules to the default xpack.alerting.rules.run.alerts.max is what I imagine?

banderror commented 4 months ago

By this, do you mean just updating the UI to add validation so users can't set anything above xpack.alerting.rules.run.alerts.max and updating the prebuilt rules to be below that threshold?

@yctercero No, we already have validation and it's a warning in this case (user can still create or update the rule). Also, updating the 18 prebuilt rules that have max_signals == 10000 is not an option.

The idea is that the app should give a heads up and explain to the user that if they want to use certain prebuilt rules, they will need/want to increase the global setting to allow these rules to promote as many external alerts as possible. I think how users increase this global setting is a separate question. Maybe, we can make it easier for users via Kibana Advanced Settings.

But it's all speculation from my side. Let's wait for input from @approksiu when she has a chance to think about the UX for this feature.