[FEATURE] Make LocalExecutionEngine respect partition numbers

fugue-project / fugue

A unified interface for distributed computing. Fugue executes SQL, Python, Pandas, and Polars code on Spark, Dask and Ray without any rewrites.

https://fugue-tutorials.readthedocs.io/

Apache License 2.0

1.92k stars 94 forks source link

[FEATURE] Make LocalExecutionEngine respect partition numbers #497

Closed goodwanghan closed 11 months ago

goodwanghan commented 11 months ago

LocalExecutionEngine didn't respect partition numbers, no matter what number set, it will have only one partition (when partition by is not set). This is not convenient for a local test to simulate the distributed execution. So we will make LocalExecutionEngine respect the partition number.