search

jupyter-incubator / sparkmagic

Jupyter magics and kernels for working with remote Spark clusters
Other
1.33k stars 447 forks source link

YARN Application ID as None on starting spark Application #422

Closed mrunmayeejog closed 6 years ago

mrunmayeejog commented 6 years ago

I have jupyer notebook running on one node. I have installed sparkmagic and separately installed livy using livy-0.4.0-incubating-bin.zip on the same node. I have configured livy to yarn-client to remotely connect to spark. I am able to create spark context sc however the following table gets printed along with it which specifies following values. Is there any other configuartion parameters missing?

ID | YARN Application ID | Kind | State | Spark UI | Driver log | Current session 0 | None | spark | idle |   |   | ✔

aggFTW commented 6 years ago

Not sure if livy sends app id with yarn-client or only with yarn-cluster. Could you use yarn-cluster mode? Otherwise, I would post a question to their forums.

mrunmayeejog commented 6 years ago

Thanks for replying. With yarn-cluster it specifies these values. Functionality wise I did not face any issue in execution of notebooks was just curious as these values are missing with yarn-client. Also in my current deployment multiple notebook execution using yarn-cluster leads to failure in cell execution. In application logs found issue was with memory limits. Where as did not face this issue with yarn-client. As per my knowledge the only difference in these two modes is where the driver application ( initiating spark context) runs.

aggFTW commented 6 years ago

Correct, only difference is where driver application is. I'm closing this then. If you would like to get the application id with yarn client, please file an incident on the livy repo.