apache / datafusion-ballista

Apache DataFusion Ballista Distributed Query Engine

https://datafusion.apache.org/ballista

Apache License 2.0

1.39k stars 181 forks source link

[Python] Add `read_csv` and `read_parquet` methods #976

Closed andygrove closed 4 months ago

andygrove commented 4 months ago

Which issue does this PR close?

N/A

Depends on https://github.com/apache/arrow-datafusion-python/pull/582

Rationale for this change

This is the first step to re-enabling the full DataFrame API from Python, but without duplicating DataFusion's PyDataFrame this time.

What changes are included in this PR?

Add read_csv and read_parquet to PySessionContext
Refactor code into separate files
Add new tests that demonstrate using these methods as well as using DataFrame API transformations

Are there any user-facing changes?