search

opensearch-project / OpenSearch

🔎 Open source distributed and RESTful search engine.
https://opensearch.org/docs/latest/opensearch/index/
Apache License 2.0
9.7k stars 1.8k forks source link

[Feature Request] Compressing HTTP requests with LZ4 and other compression algorithms #14860

Open adityamohan93 opened 3 months ago

adityamohan93 commented 3 months ago

Is your feature request related to a problem? Please describe

OpenSearch service supports only gzip for compressing HTTP requests. My team runs low latency systems and for Redis, we have seen LZ4 provides lower compression/decompression time at higher throughput and lower latency for over the wire transfers compared to gzip and deflate.

Describe the solution you'd like

OpenSearch be modified to support LZ4 also for HTTP requests. I see it's used as default codec for storing index, so we should also consider it for compression for request/response.

Related component

Search:Performance

Describe alternatives you've considered

No response

Additional context

No response

mch2 commented 3 months ago

Related - https://github.com/opensearch-project/OpenSearch/issues/10684. Curious what the perf comparison would be here with protobuf?

adityamohan93 commented 2 months ago

@mch2 Protobuf and LZ4 are different things. I have seen LZ4 compression used on top of Protobuf in my org and it improves the results. Compression works better when there are duplicate fields and data in the object being transferred which is always the case with OpenSearch response which contains a bunch of documents with same field names and in some cases have similar or same field values. Protobuf is a binary replacement for JSON formatting while LZ4 is a faster compression replacement for GZIP. Whether OpenSearch adds support for Protobuf/gRPC or not, the decision of adding LZ4 support runs in parallel and would benefit regardless.