EPIC: Add all `SessionContext` and `DataFrame` methods to Python API

[andygrove]

The Python bindings currently only expose a subset of functionality, and we want to expose as much as possible.

Here is a list of all available rust methods. Note that there may be reasons why we don't want to expose some of these.

SessionContext

• [x] new
• [x] with_config
• [x] register_csv
• [x] register_parquet
• [x] sql
• [x] catalog
• [x] table_exist
• [x] table
• [x] tables
• [x] register_table
• [x] register_udf
• [x] register_udaf
• [x] https://github.com/apache/arrow-datafusion-python/issues/28
• [ ] register_variable
• [ ] read_table
• [x] https://github.com/apache/arrow-datafusion-python/pull/56
• [x] https://github.com/apache/arrow-datafusion-python/pull/57
• [x] https://github.com/apache/arrow-datafusion-python/pull/58
• [ ] register_listing_table
• [ ] register_json
• [ ] register_avro
• [ ] register_catalog
• [ ] deregister_table
• [x] session_id
• [ ] state

DataFrame

• [ ] new
• [x] select
• [x] select_columns
• [x] with_column
• [x] with_column_renamed
• [x] explain
• [x] collect
• [x] show / show_limit
• [x] schema
• [x] filter
• [x] aggregate
• [x] sort
• [x] join
• [x] limit
• [x] https://github.com/apache/arrow-datafusion-python/issues/27
• [x] https://github.com/apache/arrow-datafusion-python/issues/25
• [x] https://github.com/apache/arrow-datafusion-python/issues/29
• [x] https://github.com/apache/arrow-datafusion-python/issues/30
• [x] https://github.com/apache/arrow-datafusion-python/issues/26
• [x] https://github.com/apache/arrow-datafusion-python/issues/12
• [x] collect_partitioned
• [ ] to_logical_plan
• [ ] create_physical_plan
• [ ] registry

[francis-du]

@andygrove Hi Andy. Could you update this task list? If you have a time.

[nkarpov]

Noticing that join_on (supporting arbitrary expressions instead of keys) is not present in the list above.

I can't find a way to use the existing binding to join on an arbitrary expression

Should the Rust join_on have its own wrapper too? or should we have the existing join wrapper choose appropriately?

[ami-navon]

+1 register_json( )