sjrusso8 / spark-connect-rs

Apache Spark Connect Client for Rust

https://docs.rs/spark-connect-rs

Apache License 2.0

76 stars 13 forks source link

feat(types) & bug(plan): add new datatypes & fix plan id bug #31

Closed sjrusso8 closed 4 months ago

sjrusso8 commented 4 months ago

Description

New DataTypes & Literals:

update types with Timestamp, TimestampNtz & &[u8] as Binary
update expressions to handle Timestamp and TimestampNtz correctly
- use the chrono crate correctly and not manually parsing dates to unix epoch
update expressions to create a binary literal from &[u8]

Fix plan id & column issue:

Plan ID was not incrementing correctly, changed to a static AtomicI64 counter. Implementation is much simpler.
Issue with plan_id being hardcoded to Some(1) on column expressions. Results in an error. Changed to None and error is resolved

Other:

Changed the rust Cargo.toml to only include tokio as a dev dependency
Updated README