Reduce DTS costs for cross zone data transfer within Elasticsearch

[dakrone]

DTS, or Data Transfer & Storage, is a high cost for users either running on ESS or their own cloud deployment. Users use forced awareness to maintain copies of indices on multiple availability zones for high availability. Elasticsearch currently does no special handling of data to reduce the amount of data transferred between zones.

This meta issue links to issues with potential ideas for mitigating the cost of DTS.

Sources of inter-AZ data transfer

Some of these sources are not directly transferring data, however, their execution leads to data being transferred between availability zones

• Regular relocation
• Shrinking an index
• Forcemerge
• ILM migration (migrating between tiers)
• Rollups
• ILM allocation (through the "allocate" action)
• Rollover
• Changing the number of replicas

Separate issues

• Replicate or relocate data via snapshot (#73496)
• Enable transport compression between nodes (#73497)
• Make shard balancing aware of forced awareness attributes (#73498)
• Combine ILM shrink and force merge (#73499)
• Shrink an index from a snapshot (#73500)
• Experiment with CBOR or other smaller/compressed shipping formats (#8481)

[elasticmachine]

Pinging @elastic/es-distributed (Team:Distributed)

[elasticmachine]

Pinging @elastic/es-core-features (Team:Core/Features)

[dakrone]

https://github.com/elastic/elasticsearch/issues/73971 may also be related, dealing with DTS costs from async search.

[seang-es]

https://github.com/elastic/elasticsearch/issues/62194 related

[DaveCTurner]

I think the ability to relocate shards via snapshots has reduced or eliminated much of the DTS costs mentioned above. I'm therefore removing this from the distrib team area.

[elasticsearchmachine]

Pinging @elastic/es-data-management (Team:Data Management)