search

elastic / kibana

Your window into the Elastic Stack
https://www.elastic.co/products/kibana
Other
19.67k stars 8.23k forks source link

[DataViews] Add caching for the request of fields #169622

Closed kertal closed 10 months ago

kertal commented 1 year ago

In Kibana, currently data views are cached by instance. This made sense back in the days when it was sure, there was just one DataView / index pattern. Each used index pattern is requesting all fields when a data view is created.

With recent changes, and especially with the introduction of ES|QL, ad hoc data views were introduce, they are not persisted, created on the fly, and when created, they are fetching fields.

With this change there can be multiple data views using the same index pattern. This can cause redundant requests to Elasticsearch, and redundant caching of the same fields used by different DataView instances. These redundant requests can delay e.g. the rendering of a Dashboard in a significant way. If there's a dashboard having multiple visualizations provided by ES|QL or just using ad-hoc data view when being created, on initial load (when e.g. users get a link to this dashboard, or the open a Dashboard in Kibana with un-cached data views), all fields of the required data views are being loaded before the dashboard is rendered. While this is just a short delay when the requests are fast and small, this can cause a significant delay when this is not the case. Here's an example:

With 4 visualizations on a dashboard, all using the same index pattern, and with a slowed network (simulating 3G), in this case it takes 4 seconds until the fields are requested and the fetch for data and aggregated data can start. When navigating to this dashboard again, it's rendering 4s faster. When reloading the dashboard in the browser, it again takes 4s, since the cache is lost.

Cursor_und_Elastic

Therefore there should be a way of caching implemented, so redundant requests to Elasticsearch for fields should be omitted. In the given case the initial loading time would be reduced from 4 to 2 seconds.

Measure success

This is a dashboard containing 4 ES|QL based visualization, targeting the same index pattern. When initially loading the dashboard, this triggers 5 requests for the same fields. When reloading the dashboard in the browser, the same 5 requests are triggered again. esql_dashboard.ndjson.zip

A successful caching strategy should reduce the requests to

  1. A single request on initial loading of the dashboard
  2. No request when the dashboard is reloading by the browser

For the given dashboard, when there are simulated bad network conditions, this reduces the initial loading time from 10s to 2s, and from 10s to 0s if the page is refreshed in the browser. The same would apply for large or slow field requests e.g. in CCS.

elasticmachine commented 1 year ago

Pinging @elastic/kibana-data-discovery (Team:DataDiscovery)

kertal commented 1 year ago

Linking POC research work