search

uber / RemoteShuffleService

Remote shuffle service for Apache Spark to store shuffle data on remote servers.
Other
321 stars 100 forks source link

Support zstd compression for shuffle data #91

Closed hiboyang closed 1 year ago

hiboyang commented 1 year ago

Issue https://github.com/uber/RemoteShuffleService/issues/90 mentioned zstd has much better compression rate.

Add a conf spark.shuffle.rss.compress to allow lz4 and zstd compression.

mabansal commented 1 year ago

@hiboyang can you also please add few tests

YutingWang98 commented 1 year ago

Sure, will add test to it, thanks

hiboyang commented 1 year ago

@hiboyang can you also please add few tests

@mabansal added test for zstd compression