[Object Spilling] Use subdirectories to avoid large top level inodes for file spilling

[rkooo567]

What is the problem?

For cases like 100TB shuffle, this could be problematic. Spark is doing the same thing for spilling.

cc @clarkzinzow

Reproduction (REQUIRED)

Please provide a short code snippet (less than 50 lines if possible) that can be copy-pasted to reproduce the issue. The snippet should have no external library dependencies (i.e., use fake or mock data / environments):

If the code snippet cannot be run by itself, the issue will be closed with "needs-repro-script".

• [ ] I have verified my script runs in a clean environment and reproduces the issue.
• [ ] I have verified the issue also occurs with the latest wheels.

[wuisawesome]

Btw we can also consider just using a different file system for this (iirc Facebook haystack used XFS to solve this before?)