search

apache / datafusion

Apache DataFusion SQL Query Engine
https://datafusion.apache.org/
Apache License 2.0
5.92k stars 1.12k forks source link

Document DataFusion Threading (and how to separate IO and CPU bound work) #12393

Open tustvold opened 1 week ago

tustvold commented 1 week ago

Is your feature request related to a problem or challenge?

DataFusion performs CPU bound work within async closures. This causes issues if running IO on the same async runtime, as the cooperative nature of such schedulers allows the CPU bound work to starve servicing of IO. This leads to errors such as https://github.com/apache/arrow-rs/issues/5882.

Describe the solution you'd like

I think at the very least this needs to be better documented, I couldn't find any mention of this in the DataFusion documentation following a cursory search.

I also think more holistic approach would be valuable to this, as it stands the use of async within DataFusion acts as a massive footgun that encourages users to intermix IO and CPU work in a way that is at best inefficient, but this can be tracked as a separate follow on task.

Describe alternatives you've considered

No response

Additional context

No response

alamb commented 1 week ago

I recommend two things:

  1. Write a blog with background and explanation of why using two threadpools is important with DataFusion and examples of how to do it
  2. Add additional documentation (ideally linking to the blog) with a summary and linking to the blog with content.