search

elastic / kibana

Your window into the Elastic Stack
https://www.elastic.co/products/kibana
Other
19.82k stars 8.2k forks source link

[Security Solution] [Detections] [Platform] Expose `_tier` in the Security Solution #130517

Open dhurley14 opened 2 years ago

dhurley14 commented 2 years ago

Currently (versions <= 8.1), if users want to segment data in the security solution between the different data tiers they have to rely on index aliases. By exposing the _tier field in queries executed in the security solution, we can better provide users with finer-grained controls when searching for alerts and associated source events from hot,warm,cold,frozen nodes.

This issue will serve as a reference as work progresses towards this goal.

TODO:

  1. [ ] Update the UI for rules to accept tiers from which to include in the given rules execution
  2. [ ] Update sourcerer to include a tier selection for queries executed against a given data view.
elasticmachine commented 2 years ago

Pinging @elastic/security-solution (Team: SecuritySolution)

yctercero commented 1 year ago

Note that @banderror mentioned this issue here https://github.com/elastic/kibana/issues/158495

Closed that one as duplicate.

s-abdelwahhab commented 1 year ago

Any updates about this please?

yctercero commented 1 year ago

Hi @s-abdelwahhab ! When we were initially discussing this issue we were trying to figure out if adding a _tier filter could help avoid EQL query rules from querying frozen tiers during the pre-search phase. It was determined that such a filter, even if exposed, would not be applied in that instance.

If you're able to, would love to understand your use case here.

marshallmain commented 11 months ago

Customer in https://discuss.elastic.co/t/coordinating-nodes-high-circuit-breaker-tripped-counts/344161/11 also suggested an option to filter on _tier to deal with queries hitting the frozen tier due to future timestamps. In the thread, a number of prebuilt new terms and EQL sequence rules time out due to frozen tier nodes and future timestamps. Disabling fallback to @timestamp fixes the issue, but a filter on _tier also improves performance without having to disable fallback.