search

aws / event-ruler

Event Ruler is a Java library that allows matching many thousands of Events per second to any number of expressive and sophisticated rules.
Apache License 2.0
556 stars 58 forks source link

Performance Query #120

Closed sridhard closed 8 months ago

sridhard commented 8 months ago

Hi,

As per the docs below are suggested when designing events and rules:

  1. Shorten number of fields inside rules, the less key in the rules, the short path to find them out.
  2. Shorten number of fields inside event, the less key inside event, the less attempts will be required to find out rules.

As per my knowledge, the rules are matched with the input event using a state machine. So the number of keys inside the rule matters for performance. But how does the number of fields inside a event matter for performance.

Suppose we have a rule with 2 keys. If the input event has 10 keys or 100 keys, the matching is done only for the keys inside rule correct? In this case it does 2 matches irrespective of number of keys in event correct?

timbray commented 8 months ago

It's complicated. The process of matching has two phases - the incoming event has to be "flattened" - turned into a list of fields, i.e. key/value pairs, and sorted. Then the flattened form is run over the machine. Typically, the flattening takes >50% of the execution time.

Only fields that appear in one or more Rules in the Machine are included in the flattened output, the rest are bypassed quite efficiently. However, running over all the data that will not be used in matching is not free.

So the matching time is related to the number of fields that are mentioned in one or more rules, and the total size of the event.

sridhard commented 8 months ago

got it. Thanks