Closed animeshtrivedi closed 6 years ago
In the code https://github.com/zrlio/spark-io/blob/master/src/main/scala/org/apache/spark/storage/CrailStore.scala - the check to delete stages are based upon shuffleCache.contains - which is wrong. See here : https://docs.oracle.com/javase/7/docs/api/java/util/concurrent/ConcurrentHashMap.html#contains(java.lang.Object)
shuffleCache.contains
Instead of this, call shuffleCache.containsKey(...) which works nicely ! :)
shuffleCache.containsKey(...)
I wonder why this ever worked before (for testing delete shuffle stages). Spark 2.2 does call to delete shuffle stages.
Fixed in : 8c30b6f
animeshtrivedi
commented
6 years ago animeshtrivedi
commented
6 years ago
In the code https://github.com/zrlio/spark-io/blob/master/src/main/scala/org/apache/spark/storage/CrailStore.scala - the check to delete stages are based upon
shuffleCache.contains- which is wrong. See here : https://docs.oracle.com/javase/7/docs/api/java/util/concurrent/ConcurrentHashMap.html#contains(java.lang.Object)Instead of this, call
shuffleCache.containsKey(...)which works nicely ! :)I wonder why this ever worked before (for testing delete shuffle stages). Spark 2.2 does call to delete shuffle stages.