Yarn use container - Githubissues

at15 / hadoop-spark-perf

(Deprecated) Hadoop and Spark CPU performance Benchmark

MIT License

0 stars 0 forks source link

Yarn use container #8

Closed at15 closed 7 years ago

at15 commented 7 years ago

This is a big issue I didn't think about, and now too late to solve?

Real solution

[ ] monitoring container is possible, though I don't know how Yarn make use of container
[ ] find the container
[ ] use perf inside container

Fallback

[ ] only monitor the node using perf, i.e. namenode, datanode, the is actually the characteristic of framework part, not the job part .... I could focus on that and leave the hard part later
[ ] see how Spark work
- [ ] with Yarn
- [ ] without Yarn
  - http://spark.apache.org/docs/latest/cluster-overview.html
  - simple cluster setup without mesos and yarn http://spark.apache.org/docs/latest/spark-standalone.html

NOTE spark's executor is a new java process, I have to get the pid of the new java process, might use jps?

What is Yarn

http://www.ibm.com/developerworks/library/bd-hadoopyarn/

at15 commented 7 years ago

Spark jps output when doing wordcount

slave

2096 NodeManager
5520 CoarseGrainedExecutorBackend
2002 DataNode
5788 Jps
2302 Worker

master

2784 ResourceManager
8848 CoarseGrainedExecutorBackend
3248 Master
2913 NodeManager
2291 NameNode
3395 Worker
8756 SparkSubmit
2631 SecondaryNameNode
9195 Jps
2446 DataNode

at15 commented 7 years ago

Hadoop jps

master

2784 ResourceManager
2913 NodeManager
3395 Worker
2631 SecondaryNameNode
2446 DataNode
3248 Master
9809 Jps
2291 NameNode
9719 YarnChild
9720 YarnChild
9721 YarnChild
9722 YarnChild
9467 RunJar
9723 YarnChild
9724 YarnChild
9725 YarnChild
9726 YarnChild

slave

2096 NodeManager
6097 YarnChild
6113 YarnChild
2002 DataNode
5960 MRAppMaster
6152 YarnChild
6361 Jps
2302 Worker
6126 YarnChild
6142 YarnChild
6127 YarnChild

MRAppMaster https://pravinchavan.wordpress.com/2013/04/24/hadoop-next-gen-yarn-architecture/