uber / RemoteShuffleService

Remote shuffle service for Apache Spark to store shuffle data on remote servers.
Other
323 stars 101 forks source link

close writers during end of stage using onStageCompleted #75

Open tuckerh99 opened 2 years ago

tuckerh99 commented 2 years ago

Improvement on when to close writers. While implementing an s3 storage engine, noticed that we want to close the writers sooner than later to ensure shuffle files are stored on s3 by the end of a stage.

CLAassistant commented 2 years ago

CLA assistant check
All committers have signed the CLA.

hiboyang commented 2 years ago

@tuckerh99, thanks for the PR! Curious whether you run your Spark production workload with shuffle files storing on S3? If so, how is the performance?

cpd85 commented 2 years ago

@hiboyang hi we are testing this out actively right now. in the first iteration it seems the performance impact is very high due to the nature of shuffle data. Basically if we wait to merge the shuffle files until the stage is completed, then we need to add in some delay before starting the next stage while the shuffle data is uploaded to S3.

leletan commented 1 year ago

Hi @tuckerh99 how are things going on the later iterations? Any luck?