[FEAT] add daft connect - Githubissues

Eventual-Inc / Daft

Distributed data engine for Python/SQL designed for the cloud, powered by Rust

https://getdaft.io

Apache License 2.0

2.39k stars 170 forks source link

[FEAT] add daft connect #3182

Closed andrewgazelka closed 3 weeks ago

andrewgazelka commented 3 weeks ago

This change adds a new Spark Connect server implementation in Rust, enabling Daft to communicate with Spark Connect clients. Key additions include:

Session management for handling client connections and query execution
Command handling for executing plans, analyzing plans, and managing configs
Data conversion utilities for translating between Spark Connect and Daft types
Support for range, showString, and other basic Spark operations
Config and user context management infrastructure
Full protobuf-based RPC service implementation

The commit also includes a small change to Utf8Array to use i64 as the default offset type.

andrewgazelka commented 3 weeks ago

closing this as per @samster25 and my discussion. See #3236 for related PR.