fix: need default value for getSizeAsMb(EXECUTOR_MEMORY.key) - Githubissues

apache / datafusion-comet

Apache DataFusion Comet Spark Accelerator

https://datafusion.apache.org/comet

Apache License 2.0

823 stars 163 forks source link

fix: need default value for getSizeAsMb(EXECUTOR_MEMORY.key) #1046

Closed neyama closed 3 weeks ago

neyama commented 3 weeks ago

Which issue does this PR close?

Closes #1045 .

Rationale for this change

Provide a default value 1g to getSizeAsMb(EXECUTOR_MEMORY.key), which is the default value of spark.executor.memory in Apache Spark 3.5.3.

What changes are included in this PR?

Define private val EXECUTOR_MEMORY_DEFAULT in the class
Call getSizeAsMb(EXECUTOR_MEMORY.key) with EXECUTOR_MEMORY_DEFAULT

How are these changes tested?

Tested with the datafusion-benchmarks as described in the issue #1045 .

viirya commented 3 weeks ago

Thanks @neyama @andygrove