sjrusso8 / spark-connect-rs

Apache Spark Connect Client for Rust
https://docs.rs/spark-connect-rs
Apache License 2.0
52 stars 11 forks source link

feat(data types): implement data types #41

Closed sjrusso8 closed 1 month ago

sjrusso8 commented 1 month ago

Description

Implement data types based on the Spark Data Types

Not Implemented

Usage

As a schema option

let schema = StructType::new(vec![
    StructField {
        name: "name",
        data_type: DataType::String,
        nullable: false,
        metadata: None,
    },
    StructField {
        name: "age",
        data_type: DataType::Short,
        nullable: true,
        metadata: None,
    },
]);

let df = spark
    .clone()
    .read()
    .format("json")
    .schema(schema)
    .load(path)?;

As a column cast

let df = df
    .select([
        F::col("name").cast(DataType::String).alias("name_str"),
        F::col("age").cast(DataType::Integer).alias("age_int"),
    ]);