Add an interface to allow users to compress values being written to shuffle

apache / beam

Apache Beam is a unified programming model for Batch and Streaming data processing.

https://beam.apache.org/

Apache License 2.0

7.81k stars 4.23k forks source link

Add an interface to allow users to compress values being written to shuffle #21692

Open damccorm opened 2 years ago

damccorm commented 2 years ago

Frequently values being shuffled are large and compressible, while users can compress them on their own by using a coder that compresses the data, it would be nice to be able to do so globally for all values.

Imported from Jira BEAM-14534. Original Jira may contain additional context. Reported by: SteveNiemitz.

damccorm commented 2 years ago

Unable to assign user @SteveNiemitz. If able, self-assign, otherwise tag @damccorm so that he can assign you. Because of GitHub's spam prevention system, your activity is required to enable assignment in this repo.