voltrondata / spark-substrait-gateway

Implements a gateway that speaks the SparkConnect protocol and drives a backend using Substrait (over ADBC Flight SQL).
Apache License 2.0
15 stars 8 forks source link

Support createDataFrame() #48

Closed EpsilonPrime closed 1 month ago

EpsilonPrime commented 1 month ago

The currently included backends (DuckDB and Datafusion) do not support Substrait virtual tables which this feature requires. Acero does but it is not currently enabled.

We either need to work with the backends to get this feature supported or perform a rewrite where we convert the virtual table into a named table (we do something similar for Datafusion to convert a read of local files into a named table).