Closed at15 closed 7 years ago
Spark jps
output when doing wordcount
slave
2096 NodeManager
5520 CoarseGrainedExecutorBackend
2002 DataNode
5788 Jps
2302 Worker
master
2784 ResourceManager
8848 CoarseGrainedExecutorBackend
3248 Master
2913 NodeManager
2291 NameNode
3395 Worker
8756 SparkSubmit
2631 SecondaryNameNode
9195 Jps
2446 DataNode
Hadoop jps
master
2784 ResourceManager
2913 NodeManager
3395 Worker
2631 SecondaryNameNode
2446 DataNode
3248 Master
9809 Jps
2291 NameNode
9719 YarnChild
9720 YarnChild
9721 YarnChild
9722 YarnChild
9467 RunJar
9723 YarnChild
9724 YarnChild
9725 YarnChild
9726 YarnChild
slave
2096 NodeManager
6097 YarnChild
6113 YarnChild
2002 DataNode
5960 MRAppMaster
6152 YarnChild
6361 Jps
2302 Worker
6126 YarnChild
6142 YarnChild
6127 YarnChild
MRAppMaster https://pravinchavan.wordpress.com/2013/04/24/hadoop-next-gen-yarn-architecture/
This is a big issue I didn't think about, and now too late to solve?
Real solution
Fallback
NOTE spark's executor is a new java process, I have to get the pid of the new java process, might use
jps
?What is Yarn