Open thomasneirynck opened 10 months ago
Pinging @elastic/kibana-data-discovery (Team:DataDiscovery)
Pinging @elastic/kibana-presentation (Team:Presentation)
Query-consolidation is an unexplored area, and could benefit from the existing architecture.
A few notes, in case it might help:
bfetch
batches client requests, I think, by default, up until 25 requests or until 10ms elapse, whichever is first.Content-Encoding: chunked
HTTP header and ability to listen for new chunks in the browser using some less known XHR request APIs.bfetch
response stream can also compress each message, in which case it then encodes each line as Base64 text (instead of JSON).
thx @vadimkibana!
wrt synthetics use-case, @dominiqueclarke just informed this usage was removed in 8.10.
Consider turning bsearch off in just Serverless https://github.com/elastic/kibana/issues/181938
With https://github.com/elastic/kibana/pull/179663, we have been collecting more telemetry on the overhead of bsearch, specifically the custom encoding part into the line-delimited base64 format.
75 percentile sits under 50-60 ms per bsearch
call.
Given that a single dashboard will have typically 5-6 bsearch calls to fetch data for all charts, we can expect 10s to 100s of milliseconds spent on a single dashboard, just re-encoding the data per single time2data
cycle.
Message size and encoding time scales linearly (duh).
At the end of the long tail (+95percentile), we find evidence of really large responses (in the order of megabytes)
Removing time spent re-encoding data in bsearch
should be a broad but shallow improvement to overall time. We should expect it to compound positively as well, given the single-threaded nature of nodejs. While we have no metrics on that, given the evidence of large data-responses, removal of this encoding should also reduce memory pressure on the kibana-server at runtime.
Overall, removal will help work towards "thinning" the kibana server footprint, and should yield measurable improvements to time2data (providing kibana-server supports http2 parallelization).
qq: is the research part done? can this be closed?
The kibana Server internal
bsearch
API is foundational to the functioning of dashboards.Architecture
bsearch
collects various Elasticsearch aggregation request in the browser on a debounce schedulee.g. rough schematic
Purpose
_async_search
calls . This enables queries from the Dashboards to be run as background sessions.Areas for improvement
bsearch
resolves key constraints, but also introduces new ones. Primarily, it increased pressure on Kibana Server.bsearch
is primarily work-around for http1 limitations. http2 support of Kibana Server/Cloud infra would clear this hurdle (https://github.com/elastic/kibana/issues/7104)maps/mvt
endpoints).bsearch
to be aware of the semantics of the requests. ie. this would only really work with aggregations).Goals
Consider:
bsearch
bsearch
can be "smarter" in its collection of queries. The vast majority ofbsearch
calls from Dashboards are aggregations and could be more efficiently run with a singlemsearch
orsearch
query that combines the aggs in a single definition.