search

fugue-project / fugue

A unified interface for distributed computing. Fugue executes SQL, Python, Pandas, and Polars code on Spark, Dask and Ray without any rewrites.
https://fugue-tutorials.readthedocs.io/
Apache License 2.0
1.98k stars 94 forks source link

[FEATURE] Remove Engine composition #457

Open goodwanghan opened 1 year ago

goodwanghan commented 1 year ago

We are able to combine an execution engine with a sql engine for example

native+duckdb

would use native execution engine + duckdb sql engine.

But now

  1. different sql engines can be used inside any FugueWorkflow (CONNECT in Fugue SQL)
  2. ExecutionEngine now consists of multiple components: SQLEngine, MapEngine, etc. The current expression using + only works to overwrite SQLEngine which no longer makes sense
  3. This feature is rarely used

So in order to keep things simple, we should remove this feature. We may add back when there is a better design.