Create empty shuffle data file failed when the file already exists.

MemVerge / splash

Splash, a flexible Spark shuffle manager that supports user-defined storage backends for shuffle data storage and exchange

Apache License 2.0

127 stars 29 forks source link

Create empty shuffle data file failed when the file already exists. #41

Closed jealous closed 5 years ago

jealous commented 5 years ago

Consider this scenario, a previous task failed and didn't clean up the temp file, the re-run task somehow create the temp file at the same location. This operation will fail due to the FileAlreadyExistsException. We need to delete and retry creation in this scenario.