zhztheplayer
commented
13 hours ago @zhztheplayer Is there any specific consideration on memory management side?
There should be no hard limitation from Spark's / Gluten's memory management system against using multiple threads in the same Spark task. There were only a few of lock issues that have been resolved already. The only rule is we'd report the allocations to the task's memory manager, and this is something against the approach running a background execution thread pool for all executor's tasks.
We can set task.cores to more than 1 then set to Velox's backend threads.
Velox's top-level threading model should be designed for Presto's push model. Which doesn't allow Spark's use case that creates an iterator over Velox's task. The parallelism we can easily adopt from Velox is the in-pipeline background thread pools, for example, parallel join build or scan pre-fetch. But again using the thread pools will require us to handle the memory allocation carefully speaking of Spark's memory manager model.
Having said that, there could be a hacky way to combine Spark's pull model with Velox's push model, which requires creating a queue in between them as a data broker. We can do some researches and PoCs to see if this way really brings performance advantages, if it does, we could consider doing refactors on our memory management code and other arch code to switch to it. But it's a big deal and is something more or less violates Spark's design, so we may need to think more before coding.
Yohahaha
zhli1142015
Description
Velox is initially designed to use multiple threads in each context. To fit spark's working model, we limit 1 thread per context. It's needed by fallback.
Since Velox backend can fully offload lots of queries, it will be interesting to investigate multiple threads per context. We can set task.cores to more than 1 then set to Velox's backend threads. Then use larger partition and see how the performance looks like.
@zhztheplayer Is there any specific consideration on memory management side?