elastic / kibana

Your window into the Elastic Stack
https://www.elastic.co/products/kibana
Other
19.76k stars 8.16k forks source link

Investigate more performant SavedObjects client/repository #194435

Open ymao1 opened 2 hours ago

ymao1 commented 2 hours ago

In this PR: https://github.com/elastic/kibana/pull/193341, Response Ops is switching to perform partial updates on the rule saved object directly using an ES client instead of via the SavedObjectsClient. As part of the effort to scale the throughput of alerting rules to 10x, we identified several areas of performance bottleneck, one of which is using the SavedObjectsClient for the partial update because the SO client performs two ES requests for each update. This introduces some risk (outlined in the PR summary) because we're bypassing a lot of the built in audit logging, migration handling, encryption handling that the SavedObjects client provides. Response Ops decided this was a trade-off worth making for the performance gains provided by using the ES client directly to partially updated an allow-listed set of fields that are only updated during rule execution by the system user and are not included as AAD fields on the encrypted saved object.

From @rudolf:

At this point we're not aware of other performance use cases which need to avoid extra roundtrips at all costs.
I also think that Core should not try to solve every possible use case, but rather make it easy and as safe as possible for plugins to drop down one level of abstraction to enable unique requirements.
Perhaps ideally we'd expose something like the ApiExecutionContext so that you could e.g. still use encryption but not choose everything else

In this particular case, you actually probably don't want anything in saved objects, encryption, authorization, audit logging will all make the call slower
elasticmachine commented 2 hours ago

Pinging @elastic/kibana-core (Team:Core)