search

elastic / kibana

Your window into the Elastic Stack
https://www.elastic.co/products/kibana
Other
19.64k stars 8.23k forks source link

[data.plugin] Use CBOR instead of JSON as transfer format #199392

Open thomasneirynck opened 6 days ago

thomasneirynck commented 6 days ago

Describe the feature:

Use cbor instead of JSON as transfer format of the data-plugin (e.g. ese and esql search-strategies)

Describe a specific use case for the feature:

Earlier investigation has shown that using cbor provides a marginal benefit over using JSON (5% size reduction, faster decoding on client).

Changing the transfer format is now possible due to https://github.com/elastic/kibana/pull/193060. In that PR, data-service now uses metadata about async-searches from the http-headers rather than decoding it from the body. It would allow leaving the cbor returned from ES intact and only decode it on client.

Overall, this would be a small performance benefit in isolation for a single search. It would have broad overall impact on all applications using the data-plugin (Discover, Dashboard) since that benefit would apply to all searches issued by those apps (DSL/ES|QL in Lens and Discover).

elasticmachine commented 6 days ago

Pinging @elastic/kibana-data-discovery (Team:DataDiscovery)

kertal commented 3 days ago

One question, would't CBOR instead of JSON introduce another barrier in debugging problems? Since CBOR is encoded, I guess it's harder do debug when we get HAR reporting proplems. We then would need decoders to have a look at the responses raw data? Does a marginal benefit in terms of speed and performance a higher cost in maintenance of dev & support side?

thomasneirynck commented 5 hours ago

@kertal

One question, would't CBOR instead of JSON introduce another barrier in debugging problems?

Yes, 100% agreed. That would be a drawback.

It would come down to make a trade-off between the gain (if any) and a decrease of debugability. We do not have a good idea of the gain at this point.