search

apache / datafusion-ballista

Apache DataFusion Ballista Distributed Query Engine
https://datafusion.apache.org/ballista
Apache License 2.0
1.39k stars 181 forks source link

Is it possible to support Spark Connect? #964

Open Enochack opened 5 months ago

Enochack commented 5 months ago

Is your feature request related to a problem or challenge? Please describe what you are trying to do. As Spark Connect was introduced since Spark 3.4, it could also be a chance for query engines like Ballista. If Ballista could support the Spark Connect protocol, it will be much more smooth for Spark users to migrate their codebase. I know it's a very chanlleging task to complete. Just brainstorm.

Describe the solution you'd like The Spark Connect client passes an unresolved logical plan to the server. So we will need to transform the logical plan into an optimized Ballista/Datafusion plan.

Describe alternatives you've considered

Additional context

andygrove commented 4 months ago

Yes, it could be supported, at least to some degree. I looked at the docs, and it looks like Spark Connect is essentially a gRPC service that accepts protobuf describing Spark logical plans and then executes the plans and sends results back.

This seems achievable for anything that Spark can represent in a logical plan that Ballista also supports. I haven't read enough of the documentation to understand how UDFs would work.