search

apache / incubator-gluten

Gluten is a middle layer responsible for offloading JVM-based SQL engines' execution to native engines.
https://gluten.apache.org/
Apache License 2.0
1.22k stars 438 forks source link

[CORE][SPARK-4.0] Only require one config spark.plugins to enable gluten #5915

Open ulysses-you opened 5 months ago

ulysses-you commented 5 months ago

Description

For now, there are at least three required configs to enable gluten:

Less required configs is always better. Fortunately, We can set spark.memory.offHeap.enabled and spark.shuffle.manager inside GlutenDriverPlugin since Spark 4.0.0, so only spark.plugins is required.

FelixYBW commented 5 months ago

We may set shuffle manager default as local shuffle manager. We may overwrite offheap.enabled=true once offheap.size is set.

acvictor commented 5 months ago

@FelixYBW as long as ColumnarShuffleManager is in the classpath, do you see any issues with setting default shuffle manager to ColumnarShuffleManager?

FelixYBW commented 5 months ago

@FelixYBW as long as ColumnarShuffleManager is in the classpath, do you see any issues with setting default shuffle manager to ColumnarShuffleManager?

No. go ahead to submit a PR.

acvictor commented 5 months ago

@FelixYBW as long as ColumnarShuffleManager is in the classpath, do you see any issues with setting default shuffle manager to ColumnarShuffleManager?

No. go ahead to submit a PR.

Where would this be set? I think in Spark 3.x it must be set as part of session builder?