[VL] Allow velox spill to multiple directories in micro benchmark

apache / incubator-gluten

Gluten is a middle layer responsible for offloading JVM-based SQL engines' execution to native engines.

https://gluten.apache.org/

Apache License 2.0

1.22k stars 438 forks source link

[VL] Allow velox spill to multiple directories in micro benchmark #7969

Open marin-ma opened 1 week ago

marin-ma commented 1 week ago

Description

Currently the spill from Velox pipeline in micro benchmark is hard-coded to the /tmp dir. To better simulate the spilling of real workloads, use GLUTEN_SPARK_LOCAL_DIRS env to create and set velox spill directory