search

jupyter-incubator / sparkmagic

Jupyter magics and kernels for working with remote Spark clusters
Other
1.31k stars 444 forks source link

Connecting EC2 instance to EMR Livy enabled spark cluster #704

Open ggittu opened 3 years ago

ggittu commented 3 years ago

@devstein We are experimenting the connection from EC2 instance (where Jupiter is setup with sparkmagic) to EMR spark cluster(Livy enabled). I am following this blog post https://towardsdatascience.com/every-data-scientist-can-need-some-spark-magic-853da0b81006. But unfortunately its not explained well the connecting part from the Jupyter book to the EMR cluster.

Is there any documentation I can follow for finishing the setup ?

clayms commented 3 years ago

@ggittu Are you not able to Add an End Point and then Create a Session?

Try the following to connect to your EMR cluster

  1. Add an Endpoint (replacing 10.20.40.50 with your EMR master-node-ip):

image

  1. Create the Session

image

  1. Check the Session

image