search

zrlio / crail-spark-io

Fast I/O plugins for Spark
Apache License 2.0
41 stars 14 forks source link

FIX: delete old shuffle stages which are indexed on shuffleID #7

Closed animeshtrivedi closed 6 years ago

animeshtrivedi commented 6 years ago

The old code uses, "contains" which is the same as "containsValue" however the shuffleID is the key, which should be lookedup using "containsKey". See here for more details:

https://github.com/zrlio/crail-spark-io/issues/4

Signed-off-by: Animesh Trivedi atrivedi@apache.org