search

HCADatalab / powderkeg

Live-coding the cluster!
Eclipse Public License 1.0
159 stars 23 forks source link

how to connect to a spark yarn cluster? #8

Closed clojurians-org closed 7 years ago

clojurians-org commented 7 years ago

i can't find the way to connect to yarn cluster by your api? can you offer the tiny example code inside README.md?

cgrand commented 7 years ago

Have you tried ./spark-submit --master spark:... --class powderkeg.repl your.jar? (your.jar may just be an uberjar resulting from an empty new project with powderkeg as a dep)

clojurians-org commented 7 years ago

okay, i see. my master is yarn.

i do the same thing in groovy before.
the script package the jar for itself in very limited way for spark 2.0 and run without jar. https://github.com/clojurians-org/groovy-example/blob/master/spark_etl_prt.groovy

so i want to switch to cider repl for clojure in emacs directly. i'll explore it more. thanks for your clue.

cgrand commented 7 years ago

Spark 2.0 is untested.

I'd like a way to get all the yarn-related configs and envs without having to spark-submit (and sorry for the spark:... master url I just copy-pasted from the README without replacing it with yarn).

You should be interested in ./spark-submit --master yarn --class powderkeg.repl your.jar your.ns/main-fn app-arg0 ...

Source is here https://github.com/HCADatalab/powderkeg/blob/master/src/main/clojure/powderkeg/repl.clj