sjrusso8 / spark-connect-rs

Apache Spark Connect Client for Rust
https://docs.rs/spark-connect-rs
Apache License 2.0
52 stars 11 forks source link

Epic: Core Client Kernel #63

Open sjrusso8 opened 2 weeks ago

sjrusso8 commented 2 weeks ago

Description

Lots of other programming languages can access gRPC and Arrow, which means many different languages will be recreating the core client logic to handle the client requests and response handling.

The idea is that there could one kernel client that all other programming languages use, and then the specific language is left to implement the specific of the core spark objects.

What does this involve?

  1. Isolate the existing client.rs into core and move all other rust specific implementations into rust
  2. Remove any dependencies in client.rs that are only for the rust library
  3. Update error handling in client.rs to create a new ConnectClientError error type, (currently leverages SparkError)
  4. lots of more research and analysis into the feasibility of this kernel :)