MemVerge / splash

Splash, a flexible Spark shuffle manager that supports user-defined storage backends for shuffle data storage and exchange
Apache License 2.0
127 stars 29 forks source link

Remove the file exists check. #27

Closed jealous closed 5 years ago

jealous commented 5 years ago

Remove multiple files exists check in the shuffle manager. Instead, try the operation and catch the exception if the file does not exist. We hope this could improve the performance on happy paths.

This commit also fixes the warning messages in the build. This closes #16

codecov[bot] commented 5 years ago

Codecov Report

Merging #27 into master will increase coverage by 0.45%. The diff coverage is 48.27%.

Impacted file tree graph

@@             Coverage Diff              @@
##             master      #27      +/-   ##
============================================
+ Coverage     75.33%   75.78%   +0.45%     
  Complexity      435      435              
============================================
  Files            30       30              
  Lines          2035     2044       +9     
  Branches        332      332              
============================================
+ Hits           1533     1549      +16     
+ Misses          277      275       -2     
+ Partials        225      220       -5
Impacted Files Coverage Δ Complexity Δ
...apache/spark/shuffle/sort/SplashUnsafeSorter.scala 80.79% <ø> (-0.13%) 37 <0> (ø)
.../org/apache/spark/shuffle/SplashObjectWriter.scala 83.9% <100%> (+1.14%) 30 <0> (ø) :arrow_down:
...k/shuffle/SplashBypassMergeSortShuffleWriter.scala 79.26% <20%> (-1.75%) 20 <0> (ø)
...che/spark/shuffle/SplashShuffleBlockResolver.scala 75.56% <50%> (-1.82%) 34 <1> (-1)
.../main/java/com/memverge/splash/TmpShuffleFile.java 77.27% <55.55%> (+7.7%) 2 <0> (-1) :arrow_down:
...memverge/splash/shared/SharedFSTmpShuffleFile.java 74.02% <0%> (+2.59%) 15% <0%> (+1%) :arrow_up:
...om/memverge/splash/shared/SharedFSShuffleFile.java 87.5% <0%> (+12.5%) 9% <0%> (ø) :arrow_down:
.../apache/spark/shuffle/local/LocalShuffleFile.scala 60% <0%> (+17.14%) 14% <0%> (+1%) :arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 00261f5...243b4c2. Read the comment docs.