search

Intel-bigdata / Spark-PMoF

Spark Shuffle Optimization with RDMA+AEP
Apache License 2.0
30 stars 22 forks source link

Delete fsdax files independently #96

Closed kelvin-qin closed 3 years ago

kelvin-qin commented 3 years ago

When running PMOF jobs with large volumes of data, it is common that FSDAX files cannot be deleted,thus affecting the next Job running. For example, when running a 2TB Terasort test, the FSDAX file cannot always be deleted. The potential problem might be in cleaning up the pool, but FSDAX does not need to clean up the pool and can directly delete files using POSIX operations. Therefore, it is recommended to separate the fsdax file deletion operation from devdax.

kelvin-qin commented 3 years ago

See PR#98

kelvin-qin commented 3 years ago

98