search

apache / datafusion-comet

Apache DataFusion Comet Spark Accelerator
https://datafusion.apache.org/comet
Apache License 2.0
825 stars 164 forks source link

feat: Use global tokio runtime per executor #1104

Open andygrove opened 4 days ago

andygrove commented 4 days ago

Which issue does this PR close?

N/A

Rationale for this change

This was an experiment related to researching https://github.com/apache/datafusion-comet/issues/1098. I wondered if creating a new tokio runtime per task was causing overhead. It does not seem to matter too much but this PR does result in a small performance improvement for my benchmarking (2% faster for TPC-H @ 100 GB).

What changes are included in this PR?

Create one global tokion runtime for the process.

How are these changes tested?

kazuyukitanimura commented 4 days ago

Wondering if the performance improvement comes from different numbers of threads? Are the default numbers are the same?