Closed mythrocks closed 1 week ago
I'll take a look @mythrocks
Sorry this took forever. I don't think we need to make this change. We don't instantiate our own IndexShuffleBlockResolver
, instead we use whatever was there already for the MT shuffle path, so we'd be OK there.
For the UCX path, we don't support dynamic allocation/migration or anything like that. There is plenty broken if we were told to migrate data from executor a to executor b today.
This concerns the shuffle cleanup fix introduced in SPARK-46957. Per description:
This might need evaluation by someone familiar with Spark Shuffle. It appears to be a missed condition where shuffle files might not be cleaned up.