Tracking Upgrade to Datafusion 37

[Michael-J-Ward]

I sketched out some of the upgrade in #662 and wanted to share what I encountered.

Some major changes in between datafusion 36 and 37.

Needs Completing

• [x] 1) Porting functions.rs BuiltInFunctons to datafusion-functions
• [x] 2) Replacing SectionContext::tables or updating the tests to not rely on it
• [x] 3) new trait methods for ExecutionPlan and ExecutionPlanProperties for DatasetExec need implementing

1) Porting BuiltinFunctions

See this epic https://github.com/apache/datafusion/issues/9285.

That means much of datafusion-python::functions.rs needs an update. An incomplete first pass is https://github.com/apache/datafusion-python/pull/662/commits/83249fe9a4e809bb6065671bcaee543065d2d356

2) SectionContext::tables

It was removed in https://github.com/apache/datafusion/pull/9627

Many of the tests rely on this method. Does each test get updated to a new API or do we implement our own version to start and then update the tests until it's not necessary?

3) New Trait Methods

I just marked these as TODO to get the upgrade to compile. See https://github.com/apache/datafusion-python/pull/662/commits/8cd4fd43c57a8b3fb6733a85da3aac068b73dcd1

[Michael-J-Ward]

Current Status

The initial steps are completed and python -m pytest now passing in #662 (modulo 1, discussed below).

TODO

Besides the failing test

• [ ] Debug last failing test from porting BuiltinFunctions: https://github.com/apache/datafusion-python/pull/662/commits/ecff35793e918a3dbdaf9220670d1ca5892b48d3
• [ ] Decide if re-implementing SessionContext::tables is what we want to do
• [ ] (maybe?) Remove allow(deprecated) from udf.rs
• [ ] (maybe?) Ensure full test coverage of ported functions.rs