cdh中yarn调度spark，container资源倾斜，container集中于一台或几台机器

ddxygq / ddxygq.github.io

柯广的博客

http://www.ikeguang.com/

0 stars 0 forks source link

cdh中yarn调度spark，container资源倾斜，container集中于一台或几台机器 | 柯广的博客 #139

Open ddxygq opened 4 years ago

ddxygq commented 4 years ago

http://www.ikeguang.com/2020/07/08/yarn-spark-container/

在cdh 6.0.1版本中，提交spark任务，发现yarn分配的container集中于几台节点，其它节点没有分配。这显然会导致个别机器负载过高，从而影响集群整个性能。原因yarn.scheduler.fair.assignmultiple: Whether to allow multiple container assignments in one heartbeat. Defaults t

ddxygq commented 3 years ago

哈哈

ddxygq commented 3 years ago

嘿嘿

ddxygq commented 3 years ago

cdh中yarn调度spark，container资源倾斜，container集中于一台或几台机器

ddxygq commented 3 years ago

test