apache / datafusion

Apache DataFusion SQL Query Engine
https://datafusion.apache.org/
Apache License 2.0
6.33k stars 1.2k forks source link

Add example for using a separate threadpool for CPU bound work #13424

Open alamb opened 1 week ago

alamb commented 1 week ago

TODOs:

Which issue does this PR close?

Rationale for this change

I added documentation that explains the problem here:

But now we need to show people how to fix it

I am reminded when trying to do this how non obvious it is

What changes are included in this PR?

Add a well commented example of how to use mutiple runtimes

The DedicatedExecutor code is orginally from

  1. InfluxDB 3.0 (todo link), largely written by @tustvold and @crepererum
  2. Largely based on https://github.com/datafusion-contrib/datafusion-dft/pull/247 from @matthewmturner

The XXX object store code is also based on work from @matthewmturner in

Are these changes tested?

By CI

Are there any user-facing changes?