Repeated requests are being made to load the config saved-object when using search source client

[mikecote]

After running some performance tests for Kibana Alerting, I noticed in the APM traces that whenever we create a search source client (await dataPlugin.search.searchSource.asScoped), it performs four sets of requests to the following endpoints:

• POST /_security/user/has_privileges
• GET /.kibana_8.16.0/_doc/config%3A8.16.0

The alerting framework use to create a search source client and data view object before running each alerting rule, regardless if it was used or not. Whenever we run at a scale of 32,000 alerting rule runs per minute, it correlated to 128k requests to has_privileges and 128k requests to config each minute for the search source service and ~32k for data view service. We’ve now moved to lazy load these components but figured I’d share the findings for the search source client. It may still be a high num of requests for users running alerting rules that use search source / data views.

Preferably the optimizations work well when there are a large number of search source and data view instances created.

APM trace:

[elasticmachine]

Pinging @elastic/kibana-data-discovery (Team:DataDiscovery)

[lukasolson]

Related to https://github.com/elastic/kibana/issues/84923.