[Request] provide example of Spark Yarn cluster conectivity (EMR)

vericast / spylon-kernel

Jupyter kernel for scala and spark

Other

187 stars 38 forks source link

[Request] provide example of Spark Yarn cluster conectivity (EMR) #49

Open fcastillao opened 5 years ago

fcastillao commented 5 years ago

please provide some examples of yarn cluster connectivity

use case: want to connect to an EMR cluster running spark on YARN mode, running on a public vpc/subnet and allowing ssh (port 22) connections, can open other ports when necessary.

thanks

Lihengwannafly commented 5 years ago

%%init_spark
launcher.num_executors = 18
launcher.executor_cores = 6
launcher.driver_memory = '30g'
launcher.master = "yarn"
launcher.jars = ["file:///root/spark-tensorflow-connector_2.11-1.14.0.jar"]

You can try this.