sjrusso8 / spark-connect-rs

Apache Spark Connect Client for Rust
https://docs.rs/spark-connect-rs
Apache License 2.0
54 stars 11 forks source link

Help getting spark-connect-rs running locally #14

Closed MrPowers closed 2 months ago

MrPowers commented 2 months ago

Commands I ran (I am a total n00b):

[dependencies]
spark-connect-rs = "0.0.1-beta.3"
tokio = { version = "1", features = ["full"] }

Here is the error message:

   Compiling spark-connect-gist v0.1.0 (/Users/matthew.powers/Documents/code/my_apps/spark-connect-gist)
    Finished dev [unoptimized + debuginfo] target(s) in 1.00s
     Running `target/debug/spark-connect-gist`
Error: tonic::transport::Error(Transport, hyper::Error(Connect, ConnectError("tcp connect error", Os { code: 61, kind: ConnectionRefused, message: "Connection refused" })))
MrPowers commented 2 months ago

Whoops, this is actually working, I forgot to start the Spark Connect session with the command: ./sbin/start-connect-server.sh --packages org.apache.spark:spark-connect_2.12:3.5.1

Are you OK if I add these setup instructions to the README?

sjrusso8 commented 2 months ago

So I just ran through the steps on a fresh ubuntu image, and here is what I had to do.

spark-connect-rs = "0.0.1-beta.3"
tokio = "1.37.0"

Ensure spark connect is up and running.


I'll update the top level readme with the example from the gist. The current one leverages a file path that might not exist for all users depending on how they are running their spark cluster.

sjrusso8 commented 2 months ago

I see you also updated the spark-connect-go readme. I'll mirror that style