We use Rss in our adhoc cluster with preemption enabled, which mean some tasks may fail due to resource being preempted while writing shuffle data.
I am wondering if the shuffle writer has written only with part of the shuffle data when it is preempted, and the server doesn't mark it as corrupted since the stream has been closed properly. Will this cause any shuffle data inconsistency in the partition file? Thanks.
We use Rss in our adhoc cluster with preemption enabled, which mean some tasks may fail due to resource being preempted while writing shuffle data.
I am wondering if the shuffle writer has written only with part of the shuffle data when it is preempted, and the server doesn't mark it as corrupted since the stream has been closed properly. Will this cause any shuffle data inconsistency in the partition file? Thanks.