search

voltrondata / spark-substrait-gateway

Implements a gateway that speaks the SparkConnect protocol and drives a backend using Substrait (over ADBC Flight SQL).
Apache License 2.0
15 stars 8 forks source link

Support "CreateDataFrameViewCommand" on in-memory DataFrames #59

Closed pthatte1-bb closed 4 weeks ago

pthatte1-bb commented 1 month ago

In-memory DataFrames can be created successfully, but cannot be registered as tables.

Snippet of supported feature:

spark_session.createDataFrame([(1, 'Alice'), (2, 'Bob')], ['age', 'name']).show()

Snippet of requested feature:

spark_session.createDataFrame([(1, 'Alice'), (2, 'Bob')], ['age', 'name']).createOrReplaceTempView("temp1")
EpsilonPrime commented 1 month ago

I've started work on this one.