search

GoogleCloudDataproc / bdutil

[DEPRECATED] Script used to manage Hadoop and Spark instances on Google Compute Engine
https://cloud.google.com/dataproc
Apache License 2.0
109 stars 94 forks source link

ambari deployment broken since mid-Nov due to upstream config change #70

Closed dennishuo closed 8 years ago

dennishuo commented 8 years ago

Currently, "./bdutil -e ambari deploy" fails about 10 minutes into the Ambari installation step with something like:

Thu Dec 10 10:21:11 PST 2015: Waiting on async 'ssh' jobs to finish. Might take a while...
Thu Dec 10 10:32:05 PST 2015: Exited 1 : gcloud --project=project --quiet --verbosity=info compute ssh dhuo-bdutil-m --command=sudo su -l -c "cd ${PWD} && ./install-ambari-components.sh" 2>>install-ambari-components_deploy.stderr 1>>install-ambari-components_deploy.stdout --ssh-flag=-tt --ssh-flag=-oServerAliveInterval=60 --ssh-flag=-oServerAliveCountMax=3 --ssh-flag=-oConnectTimeout=30 --zone=us-east1-c

Inside the debuginfo.txt that bdutil prints out, you may find something like:

dhuo-bdutil-m:  ==> install-ambari-components_deploy.stderr <==
dhuo-bdutil-m:  ambari_wait status: INPROGRESS
dhuo-bdutil-m:  ambari_wait status: INPROGRESS
dhuo-bdutil-m:  ambari_wait status: INPROGRESS
dhuo-bdutil-m:  ambari_wait status: INPROGRESS
dhuo-bdutil-m:  ambari_wait status: INPROGRESS
dhuo-bdutil-m:  ambari_wait status: INPROGRESS
dhuo-bdutil-m:  ambari_wait status: INPROGRESS
dhuo-bdutil-m:  ambari_wait status: INPROGRESS
dhuo-bdutil-m:  ambari_wait status: PENDING
dhuo-bdutil-m:  ambari_wait status: INPROGRESS
dhuo-bdutil-m:  ambari_wait status: PENDING
dhuo-bdutil-m:  ambari_wait status: INPROGRESS
dhuo-bdutil-m:  ambari_wait status: INPROGRESS
dhuo-bdutil-m:  ambari_wait status: INPROGRESS
dhuo-bdutil-m:  ambari_wait status: INPROGRESS
dhuo-bdutil-m:  ambari_wait status: PENDING
dhuo-bdutil-m:  ambari_wait status: INPROGRESS
dhuo-bdutil-m:  ambari_wait status: INPROGRESS
dhuo-bdutil-m:  ambari_wait status: PENDING
dhuo-bdutil-m:  ambari_wait status: INPROGRESS
dhuo-bdutil-m:  ambari_wait status: INPROGRESS
dhuo-bdutil-m:  ambari_wait status: PENDING
dhuo-bdutil-m:  ambari_wait status: INPROGRESS
dhuo-bdutil-m:  ambari_wait status: INPROGRESS
dhuo-bdutil-m:  ambari_wait status: INPROGRESS
dhuo-bdutil-m:  ambari_wait status: INPROGRESS
dhuo-bdutil-m:  ambari_wait status: INPROGRESS
dhuo-bdutil-m:  ambari_wait status: INPROGRESS
dhuo-bdutil-m:  ambari_wait status: INPROGRESS
dhuo-bdutil-m:  Ambari operiation failed with status: FAILED

Logging into the Ambari GUI and clicking on failed operations shows the ResourceManager failed to come up; digging up ResourceManager startup logs shows something like:

2015-12-10 18:31:59,431 INFO  service.AbstractService (AbstractService.java:noteFailure(272)) - Service org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler failed in state INITED; cause: java.lang.IllegalArgumentException: Illegal capacity of -1.0 for node-label=default in queue=root, valid capacity should in range of [0, 100].
java.lang.IllegalArgumentException: Illegal capacity of -1.0 for node-label=default in queue=root, valid capacity should in range of [0, 100].
  at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.internalGetLabeledQueueCapacity(CapacitySchedulerConfiguration.java:465)
  at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.getLabeledQueueCapacity(CapacitySchedulerConfiguration.java:477)
  at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CSQueueUtils.loadCapacitiesByLabelsFromConf(CSQueueUtils.java:143)
  at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CSQueueUtils.loadUpdateAndCheckCapacities(CSQueueUtils.java:122)
  at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.setupConfigurableCapacities(AbstractCSQueue.java:99)
  at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.setupQueueConfigs(AbstractCSQueue.java:242)
  at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.setupQueueConfigs(ParentQueue.java:109)
  at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.<init>(ParentQueue.java:100)
  at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.parseQueue(CapacityScheduler.java:589)
  at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initializeQueues(CapacityScheduler.java:465)
  at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initScheduler(CapacityScheduler.java:297)
  at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.serviceInit(CapacityScheduler.java:326)
  at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
  at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
  at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceInit(ResourceManager.java:576)
  at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
  at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:1016)
  at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:269)
  at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
  at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1237)
2015-12-10 18:31:59,433 INFO  service.AbstractService (AbstractService.java:noteFailure(272)) - Service RMActiveServices failed in state INITED; cause: java.lang.IllegalArgumentException: Illegal capacity of -1.0 for node-label=default in queue=root, valid capacity should in range of [0, 100].
java.lang.IllegalArgumentException: Illegal capacity of -1.0 for node-label=default in queue=root, valid capacity should in range of [0, 100].
  at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.internalGetLabeledQueueCapacity(CapacitySchedulerConfiguration.java:465)
  at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.getLabeledQueueCapacity(CapacitySchedulerConfiguration.java:477)
  at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CSQueueUtils.loadCapacitiesByLabelsFromConf(CSQueueUtils.java:143)
  at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CSQueueUtils.loadUpdateAndCheckCapacities(CSQueueUtils.java:122)
  at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.setupConfigurableCapacities(AbstractCSQueue.java:99)
  at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.setupQueueConfigs(AbstractCSQueue.java:242)
  at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.setupQueueConfigs(ParentQueue.java:109)
  at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.<init>(ParentQueue.java:100)
  at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.parseQueue(CapacityScheduler.java:589)
  at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initializeQueues(CapacityScheduler.java:465)
  at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initScheduler(CapacityScheduler.java:297)
  at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.serviceInit(CapacityScheduler.java:326)
  at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
  at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
  at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceInit(ResourceManager.java:576)
  at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
  at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:1016)
  at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:269)
  at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
  at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1237)
dennishuo commented 8 years ago

Release tests on November 12th showed the Ambari deployment working, so this points at a regression upstream that rolled out sometime after November 12th.

@pmkc dug up https://issues.apache.org/jira/browse/AMBARI-8064 as the likely culprit, which adds, among other things, the following:

+  <property>
+    <name>yarn.scheduler.capacity.root.accessible-node-labels.default.capacity</name>
+    <value>-1</value>
+    <description></description>
+  </property>
+  <property>
+    <name>yarn.scheduler.capacity.root.accessible-node-labels.default.maximum-capacity</name>
+    <value>-1</value>
+    <description></description>
+  </property>

We've confirmed these indeed appear in /etc/hadoop/conf/capacity-scheduler.xml on the partially-deployed clusters. The timelines also match to identify that as the culprit; it was rolled into Ambari 1.7.0 for its HDP 2.2 deployment (which is what bdutil deploys by default at the moment), going live sometime mid-November. Also relevant is the removal of the problematic config entries here: https://issues.apache.org/jira/browse/AMBARI-13232 which targets Ambari 2.1.2, which unfortunately is not yet used by bdutil.

Workaround should be straightforward, to override the problematic config entries inside the configuration.json packaged with bdutil.

dennishuo commented 8 years ago

https://github.com/GoogleCloudPlatform/bdutil/pull/71 pending