Nov 20. 2024: This week in DataFusion

[alamb]

Introduction

• Follow on to https://github.com/apache/datafusion/issues/13265 This ticket is a weekly-ish summary of interesting things happening in DataFusion. Note this is not a complete list (it is what I remember / can find). Please feel free to leave comments on this ticket about things that I may have missed or you think should get wider attention by the community

Loosely inspired by https://this-week-in-rust.org/

Highlights

• 🚀 DataFusion 43 currently has the fastest results on ClickBench (blog coming soon)
• 📝 Completed https://github.com/apache/datafusion/issues/12740. Thanks to @Omega359 and @jonathanc-n for driving the project to completion
• ⚒️ Completed https://github.com/apache/datafusion/issues/8709 -- thanks @jcsherin @jatin510 for helping push it over the line

DataFusion Related Reading List

• Comparing approaches to User Defined Functions in Apache DataFusion using Python from @timsaucer
• New Concepts, Readings, Events[¶](https://datafusion.apache.org/user-guide/concepts-readings-events.html#concepts-readings-events) page

Recent Releases

• [x] https://github.com/apache/datafusion/issues/13166
• [x] https://github.com/apache/datafusion/issues/12470 (thanks @andygrove)
• [x] https://github.com/apache/datafusion-sqlparser-rs/issues/1423 (huge kudos to @iffyio for all the reviews)

Highlights from last week(s):

(I am sorry if I missed you -- please add a note to this ticket with anything you would like to highlight)

Performance

• @Rachelint completed https://github.com/apache/datafusion/pull/12996 for another boost on ClickBench performance
• @jonathanc-n https://github.com/apache/datafusion/pull/13457

🐛 🔨

• @findepi fixed LIKE in various places https://github.com/apache/datafusion/pull/13288, https://github.com/apache/datafusion/pull/13260
• Also fixed https://github.com/apache/datafusion/pull/13434 and https://github.com/apache/datafusion/pull/13426
• @findepi also fixed https://github.com/apache/datafusion/pull/13388

Features

• @goldmedal added https://github.com/apache/datafusion/pull/13255
• Also @goldmedal added https://github.com/apache/datafusion/pull/13341
• Custom type planner (also from @goldmedal ): https://github.com/apache/datafusion/pull/13294
• @jonahgao completed support for PREPARE and EXECUTE, in several PRs such as https://github.com/apache/datafusion/pull/13242 and https://github.com/apache/datafusion/pull/13311

Logical Types

• @jayzhan211 is reworking signatures with https://github.com/apache/datafusion/pull/13240 and https://github.com/apache/datafusion/pull/13354

SortMergeJoin correctness and stability fixes(@comphead):

• https://github.com/apache/datafusion/pull/13369
• https://github.com/apache/datafusion/pull/13398

Aggregation testing coverage

• @jonathanc-n keep improving with PR after PR

Unparser (Plan --> String)

• @blaginin such as https://github.com/apache/datafusion/pull/13326
• @goldmedal https://github.com/apache/datafusion/pull/13418
• @Sevenannn https://github.com/apache/datafusion/pull/13428 and https://github.com/apache/datafusion/pull/13427
• @phillipleblanc in https://github.com/apache/datafusion/pull/13462

Others

• @alamb started documenting how to use multiple threadpools, see https://github.com/apache/datafusion/pull/13423

Major Projects / Discussions under way

• https://github.com/apache/datafusion/issues/13504 with @timsaucer and @Omega359

• 13499 to try and get delta-rs updated

Looking to get more involved? Try code review!

DataFusion has a long history of community members contributing in all aspects of the project. Reviewing PRs is an especially great way to get introduced to the project, help the community and grow your own knowledge -- researching and understanding the code enough to review PRs also often inspires additional ideas for improvements.

We have docs about reviews. TLDR is: look for test coverage, if the change is understandable and well documented, and if the code can be improved. When you think the PR looks good to merge, try @ mentioning one of the committers.

Help wanted

Please feel leave your own comments on this ticket if you are looking for help

Community

• Weekly Call
• Slack/Discord: info links

Upcoming meetups:

• 2024 Dec 18 Chicago: https://lu.ma/eq5myc5i @adriangb @timsaucer
• https://github.com/apache/datafusion/discussions/12988
• 2025 Jan 15 Boston

Background:

Previous update:

• https://github.com/apache/datafusion/issues/13167

Andrew's Focus Areas:

• 12393

[alamb]

New blog posts:

• DataFusion Blog: Comparing approaches to User Defined Functions in Apache DataFusion using Python

• DataFusion Blog: Apache DataFusion Comet 0.4.0 Release

• Building Databases over a Weekend -- @ameyc ❤️ (thanks to @devanbenz for this link)

[alamb]

• Apache DataFusion is now the fastest single node engine for querying Apache Parquet files