TL;DR: This issue serves as a placeholder for investigating our usage of the lodash/fp/set and @elastic/safer-lodash-set/fp to ensure we are not experiencing bottlenecks in other areas of our application where these functions are being used.
When doing some performance testing I found a bottleneck in our code where we merge the _source and fields data.
When we merge the data from the _source and fields properties to build the document to insert into the alerts as data index, we iterate over every field (10k) for every event found.
When a rule has a lot of fields, the rules would time out and fail to complete execution. I did some digging and found that using the set written in the fp-style was causing the bottleneck. In the fp-style version the object we are adding the property to is immutable so we copy the data from the source object along with the new property into a new object. Doing this 10k+ times for every event can slow down rule execution dramatically when users have a lot of data in the fields section of the resulting search from the rule (For instance if a rule is querying an index with a lot of runtime fields defined on that index's mapping).
This has been an issue since the code was introduced in 8.0. Essentially if a customer has an index with a large amount of properties in the field response, the rules will crawl to a halt and time out.
TL;DR: This issue serves as a placeholder for investigating our usage of the
lodash/fp/set
and@elastic/safer-lodash-set/fp
to ensure we are not experiencing bottlenecks in other areas of our application where these functions are being used.When doing some performance testing I found a bottleneck in our code where we merge the
_source
andfields
data.When we merge the data from the _source and fields properties to build the document to insert into the alerts as data index, we iterate over every field (10k) for every event found.
elastic/kibana@52a28ba/x-pack/plugins/security_solution/server/lib/detection_engine/signals/source_fields_merging/strategies/merge_missing_fields_with_source.ts#L28-L50
When a rule has a lot of fields, the rules would time out and fail to complete execution. I did some digging and found that using the
set
written in the fp-style was causing the bottleneck. In the fp-style version the object we are adding the property to is immutable so we copy the data from the source object along with the new property into a new object. Doing this 10k+ times for every event can slow down rule execution dramatically when users have a lot of data in thefields
section of the resulting search from the rule (For instance if a rule is querying an index with a lot of runtime fields defined on that index's mapping).elastic/kibana@52a28ba/x-pack/plugins/security_solution/server/lib/detection_engine/signals/source_fields_merging/strategies/merge_missing_fields_with_source.ts#L47
This has been an issue since the code was introduced in 8.0. Essentially if a customer has an index with a large amount of properties in the field response, the rules will crawl to a halt and time out.