MemVerge / splash

Splash, a flexible Spark shuffle manager that supports user-defined storage backends for shuffle data storage and exchange
Apache License 2.0
127 stars 29 forks source link

Write spill instead of partition data file #28

Closed jealous closed 5 years ago

jealous commented 5 years ago

For the bypass merge sort shuffle code path, use the make spill file API instead of the make shuffle data file API provided by the storage plugin to avoid the file already exists error.

codecov[bot] commented 5 years ago

Codecov Report

Merging #28 into master will decrease coverage by 0.03%. The diff coverage is 50%.

Impacted file tree graph

@@             Coverage Diff              @@
##             master      #28      +/-   ##
============================================
- Coverage     75.78%   75.74%   -0.04%     
- Complexity      435      437       +2     
============================================
  Files            30       30              
  Lines          2044     2045       +1     
  Branches        332      332              
============================================
  Hits           1549     1549              
  Misses          275      275              
- Partials        220      221       +1
Impacted Files Coverage Δ Complexity Δ
...k/shuffle/SplashBypassMergeSortShuffleWriter.scala 78.31% <50%> (-0.96%) 22 <2> (+2)

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 6145ded...e828a30. Read the comment docs.

jealous commented 5 years ago

Work on another solution.