sjrusso8 / spark-connect-rs

Apache Spark Connect Client for Rust
https://docs.rs/spark-connect-rs
Apache License 2.0
52 stars 11 forks source link

Implement File Format Reader/Writer #54

Open sjrusso8 opened 2 weeks ago

sjrusso8 commented 2 weeks ago

Description

Implement the initial methods to read and write .csv, .json, .orc, .parquet, and .text.

Consider creating ConfigOpts trait for each of those file options and have a custom struct represent the options for each of those file types.

Example with CSV

Create the Options and modify the opts object. The object is passed into the method.


let mut opts = CsvOptions::new()

opts.header = true;
opts.delimiter = b'|';

let df = spark.read().csv(path, opts)

Example of what the function signature might look like


impl DataFrameReader {
 ....
     pub fn csv<C: ConfigOpts>(path: &str, opts: Some(C))
}