Enable transport compression between nodes to reduce DTS costs

[dakrone]

Summary

In order to reduce DTS costs for cross-zone data transfer, we should investigate whether we should enable transport compression between nodes by default.

We could consider using a lightweight compression algorithm between the coordinating node and the primary shard, so that bulk requests are compressed. We would want to investigate using a lightweight compression algorithm such as LZ4, which is light on CPU usage so that we don't have a deleterious impact on indexing throughput.

7.14

• [X] Add option to only compress requests/responses primarily composed of source documents (#74587)
  • transport.compress can now be set to indexing_data as an experimental option
• [X] Add fast LZ4 as a transport compression option (#74587)
  • transport.compression_scheme can be set to lz4 (experimental) or deflate (default)

7.15

• [x] Ensure decompression works properly with content aggregation
• [x] Enable compression on more indexing_data operations (resync, transport bulk action) (#76321)
• [x] Reintroduce remote cluster specific compression scheme settings
• [x] Make nodes to respond with the same compression scheme they received (#76372)
• [x] Default cluster.remote.*.transport.compression_scheme to deflate if compression is explicitly enabled on remote cluster (#76580)
• [x] Update documentation to reflect the fact that this work is not experimental (#76441)

7.16

• [x] Remove unnecessary checksumming when LZ4 compressing (#76666)
  • We use deflate (not GZIP) for transport compression which does not checksum. We are not reading checksum on LZ4 uncompress.
• [x] Implement performance improvements using Java9 (https://github.com/elastic/elasticsearch/pull/77153)
• [ ] Ensure licensing is correct on forked code (https://github.com/elastic/elasticsearch/pull/80165)

Maybe Later:

• [ ] Transition to LZ4 frame format opposed to custom lz4-java block format

[elasticmachine]

Pinging @elastic/es-distributed (Team:Distributed)

[Tim-Brooks]

This will be closed by #80165.

[Tim-Brooks]

Closing as the final required task is tracked.