brooklyncentral / brooklyn

This project has moved and is now part of the ASF
https://github.com/apache/incubator-brooklyn
72 stars 27 forks source link

hung installing java (with apt-get) in AWS during OpenGamma demo #807

Open aledsage opened 11 years ago

aledsage commented 11 years ago

When trying to run the ActiveMQ entity in aws-ec2 us-east-1, running Ubuntu 13.04 on a t1.micro. (I know, t1.micro is crazy small - but this failed before even getting to ActiveMQ)...

It hung when trying to run apt-get install -y --allow-unauthenticated openjdk-6-jdk.

The problem may be related to: http://askubuntu.com/questions/163200/e-dpkg-was-interrupted-run-sudo-dpkg-configure-a

After Duncan had terminated his ssh session (disconnecting from network), I logged in and looked at the box.


The brooklyn web-console showed:

SSH executing, INSTALL_OPENJDK ActiveMQBrokerImpl{id=icoFdEEn, openWirePort=61616}

Task[start [EFFECTOR, ActiveMQBrokerImpl{id=icoFdEEn, openWirePort=61616}]; ntSxRUhV]
Submitted by Task[invoke [BasicStartableImpl{id=UPWpGLed}]; QhHiJSOd]

Waiting (timed) on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4220fc87
At: net.schmizz.concurrent.Promise.tryRetrieve(Promise.java:168)
    net.schmizz.concurrent.Promise.retrieve(Promise.java:137)
    net.schmizz.concurrent.Event.await(Event.java:103)
    net.schmizz.sshj.connection.channel.AbstractChannel.join(AbstractChannel.java:282)
    brooklyn.util.internal.ssh.sshj.SshjTool$ShellAction.create(SshjTool.java:739)
    brooklyn.util.internal.ssh.sshj.SshjTool$ShellAction.create(SshjTool.java:1)
    brooklyn.util.internal.ssh.sshj.SshjTool.acquire(SshjTool.java:390)
    brooklyn.util.internal.ssh.sshj.SshjTool.execScript(SshjTool.java:326)
    brooklyn.location.basic.SshMachineLocation$8.exec(SshMachineLocation.java:442)
    brooklyn.location.basic.SshMachineLocation$10.apply(SshMachineLocation.java:506)
    brooklyn.location.basic.SshMachineLocation$10.apply(SshMachineLocation.java:1)
    brooklyn.util.pool.BasicPool.exec(BasicPool.java:129)
    brooklyn.location.basic.SshMachineLocation.execSsh(SshMachineLocation.java:285)
    brooklyn.location.basic.SshMachineLocation.execWithLogging(SshMachineLocation.java:504)
    brooklyn.location.basic.SshMachineLocation.execScript(SshMachineLocation.java:440)
    brooklyn.entity.basic.AbstractSoftwareProcessSshDriver.execute(AbstractSoftwareProcessSshDriver.java:195)
    brooklyn.entity.basic.lifecycle.ScriptHelper.execute(ScriptHelper.java:212)
    brooklyn.entity.java.JavaSoftwareProcessSshDriver.installJava(JavaSoftwareProcessSshDriver.java:295)
    brooklyn.entity.java.JavaSoftwareProcessSshDriver.start(JavaSoftwareProcessSshDriver.java:323)
    brooklyn.entity.basic.SoftwareProcessImpl.callStartHooks(SoftwareProcessImpl.java:181)
    brooklyn.entity.basic.SoftwareProcessImpl.startInLocation(SoftwareProcessImpl.java:397)
    brooklyn.entity.basic.SoftwareProcessImpl.startInLocation(SoftwareProcessImpl.java:343)
    brooklyn.entity.basic.SoftwareProcessImpl.startInLocation(SoftwareProcessImpl.java:293)
    brooklyn.entity.basic.SoftwareProcessImpl.startInLocation(SoftwareProcessImpl.java:288)
    brooklyn.entity.basic.SoftwareProcessImpl.start(SoftwareProcessImpl.java:273)
    brooklyn.util.GroovyJavaMethods.invokeMethodOnMetaClass(GroovyJavaMethods.groovy:125)
    brooklyn.management.internal.AbstractManagementContext.invokeEffectorMethodLocal(AbstractManagementContext.java:188)
    brooklyn.management.internal.AbstractManagementContext.invokeEffectorMethodSync(AbstractManagementContext.java:215)
    brooklyn.management.internal.EffectorUtils.invokeEffector(EffectorUtils.java:218)
    brooklyn.management.internal.EffectorUtils$invokeEffector.call(Unknown Source)
    brooklyn.management.internal.EffectorUtils$invokeEffector.call(Unknown Source)
    brooklyn.entity.basic.AbstractEntity.invokeMethod(AbstractEntity.groovy:1018)
    brooklyn.entity.basic.MethodEffector.call(MethodEffector.java:83)
    brooklyn.management.internal.AbstractManagementContext$2.call(AbstractManagementContext.java:180)
    brooklyn.util.task.BasicExecutionManager$3.call(BasicExecutionManager.java:360)

duncan@ip-10-202-43-188:~$ sudo dpkg --get-selections | grep jdk

openjdk-6-jdk:i386                              install
openjdk-6-jre:i386                              install
openjdk-6-jre-headless:i386                     install
openjdk-6-jre-lib                               install

sudo apt-get check

E: dpkg was interrupted, you must manually run 'sudo dpkg --configure -a' to correct the problem. 

duncan@ip-10-202-43-188:~$ sudo dpkg --configure -a

Processing triggers for libc-bin ...
ldconfig deferred processing now taking place
Setting up openjdk-6-jre-headless:i386 (6b27-1.12.5-1ubuntu1) ...
Connection to ec2-184-72-215-247.compute-1.amazonaws.com closed.

duncan@ip-10-202-43-188:~$ sudo dpkg --configure -a

Setting up openjdk-6-jre-headless:i386 (6b27-1.12.5-1ubuntu1) ...
Error occurred during initialization of VM
Could not reserve enough space for object heap
ignoring dump failure
Connection to ec2-184-72-215-247.compute-1.amazonaws.com closed.

duncan@ip-10-202-43-188:~$ java -version

Error occurred during initialization of VM
Could not reserve enough space for object heap
duncan@ip-10-202-43-188:~$ java -d64
This Java instance does not support a 64-bit JVM.
Please install the desired version.

duncan@ip-10-202-43-188:~$ sudo apt-get install -f

E: dpkg was interrupted, you must manually run 'sudo dpkg --configure -a' to correct the problem. 

duncan@ip-10-202-43-188:~$ sudo dpkg --configure -a

Setting up openjdk-6-jre-headless:i386 (6b27-1.12.5-1ubuntu1) ...
Error occurred during initialization of VM
Could not reserve enough space for object heap
ignoring dump failure
Error occurred during initialization of VM
Could not reserve enough space for object heap
ignoring dump failure
Setting up openjdk-6-jre:i386 (6b27-1.12.5-1ubuntu1) ...
update-alternatives: using /usr/lib/jvm/java-6-openjdk-i386/jre/bin/policytool to provide /usr/bin/policytool (policytool) in auto mode
Setting up openjdk-6-jre-lib (6b27-1.12.5-1ubuntu1) ...
Setting up icedtea-6-jre-cacao:i386 (6b27-1.12.5-1ubuntu1) ...
Setting up icedtea-netx:i386 (1.3.2-1ubuntu1) ...
update-alternatives: using /usr/lib/jvm/java-6-openjdk-i386/jre/bin/javaws to provide /usr/bin/javaws (javaws) in auto mode
update-alternatives: using /usr/lib/jvm/java-6-openjdk-i386/jre/bin/itweb-settings to provide /usr/bin/itweb-settings (itweb-settings) in auto mode
update-alternatives: using /usr/lib/jvm/java-7-openjdk-i386/jre/bin/javaws to provide /usr/bin/javaws (javaws) in auto mode
update-alternatives: using /usr/lib/jvm/java-7-openjdk-i386/jre/bin/itweb-settings to provide /usr/bin/itweb-settings (itweb-settings) in auto mode
Setting up openjdk-6-jdk:i386 (6b27-1.12.5-1ubuntu1) ...
update-alternatives: using /usr/lib/jvm/java-6-openjdk-i386/bin/appletviewer to provide /usr/bin/appletviewer (appletviewer) in auto mode
update-alternatives: using /usr/lib/jvm/java-6-openjdk-i386/bin/extcheck to provide /usr/bin/extcheck (extcheck) in auto mode
update-alternatives: using /usr/lib/jvm/java-6-openjdk-i386/bin/idlj to provide /usr/bin/idlj (idlj) in auto mode
update-alternatives: using /usr/lib/jvm/java-6-openjdk-i386/bin/jar to provide /usr/bin/jar (jar) in auto mode
update-alternatives: using /usr/lib/jvm/java-6-openjdk-i386/bin/jarsigner to provide /usr/bin/jarsigner (jarsigner) in auto mode
update-alternatives: using /usr/lib/jvm/java-6-openjdk-i386/bin/javac to provide /usr/bin/javac (javac) in auto mode
update-alternatives: using /usr/lib/jvm/java-6-openjdk-i386/bin/javadoc to provide /usr/bin/javadoc (javadoc) in auto mode
update-alternatives: using /usr/lib/jvm/java-6-openjdk-i386/bin/javah to provide /usr/bin/javah (javah) in auto mode
update-alternatives: using /usr/lib/jvm/java-6-openjdk-i386/bin/javap to provide /usr/bin/javap (javap) in auto mode
update-alternatives: using /usr/lib/jvm/java-6-openjdk-i386/bin/jconsole to provide /usr/bin/jconsole (jconsole) in auto mode
update-alternatives: using /usr/lib/jvm/java-6-openjdk-i386/bin/jdb to provide /usr/bin/jdb (jdb) in auto mode
update-alternatives: using /usr/lib/jvm/java-6-openjdk-i386/bin/jhat to provide /usr/bin/jhat (jhat) in auto mode
update-alternatives: using /usr/lib/jvm/java-6-openjdk-i386/bin/jinfo to provide /usr/bin/jinfo (jinfo) in auto mode
update-alternatives: using /usr/lib/jvm/java-6-openjdk-i386/bin/jmap to provide /usr/bin/jmap (jmap) in auto mode
update-alternatives: using /usr/lib/jvm/java-6-openjdk-i386/bin/jps to provide /usr/bin/jps (jps) in auto mode
update-alternatives: using /usr/lib/jvm/java-6-openjdk-i386/bin/jrunscript to provide /usr/bin/jrunscript (jrunscript) in auto mode
update-alternatives: using /usr/lib/jvm/java-6-openjdk-i386/bin/jsadebugd to provide /usr/bin/jsadebugd (jsadebugd) in auto mode
update-alternatives: using /usr/lib/jvm/java-6-openjdk-i386/bin/jstack to provide /usr/bin/jstack (jstack) in auto mode
update-alternatives: using /usr/lib/jvm/java-6-openjdk-i386/bin/jstat to provide /usr/bin/jstat (jstat) in auto mode
update-alternatives: using /usr/lib/jvm/java-6-openjdk-i386/bin/jstatd to provide /usr/bin/jstatd (jstatd) in auto mode
update-alternatives: using /usr/lib/jvm/java-6-openjdk-i386/bin/native2ascii to provide /usr/bin/native2ascii (native2ascii) in auto mode
update-alternatives: using /usr/lib/jvm/java-6-openjdk-i386/bin/rmic to provide /usr/bin/rmic (rmic) in auto mode
update-alternatives: using /usr/lib/jvm/java-6-openjdk-i386/bin/schemagen to provide /usr/bin/schemagen (schemagen) in auto mode
update-alternatives: using /usr/lib/jvm/java-6-openjdk-i386/bin/serialver to provide /usr/bin/serialver (serialver) in auto mode
update-alternatives: using /usr/lib/jvm/java-6-openjdk-i386/bin/wsgen to provide /usr/bin/wsgen (wsgen) in auto mode
update-alternatives: using /usr/lib/jvm/java-6-openjdk-i386/bin/wsimport to provide /usr/bin/wsimport (wsimport) in auto mode
update-alternatives: using /usr/lib/jvm/java-6-openjdk-i386/bin/xjc to provide /usr/bin/xjc (xjc) in auto mode
Setting up icedtea-6-jre-jamvm:i386 (6b27-1.12.5-1ubuntu1) ...
Setting up libatk-wrapper-java (0.30.4-0ubuntu4) ...
Setting up ca-certificates-java (20121112+nmu2) ...
Error occurred during initialization of VM
Could not reserve enough space for object heap
Connection to ec2-184-72-215-247.compute-1.amazonaws.com closed.
aledsage commented 11 years ago

So questions are:

Problem is that the thread stack shows it was still executing the apt-get install. We wouldn't know that it had failed - perhaps it was just being very slow, and the ssh connection disconnecting caused it to get into the "dpkg was interrupted" state.

So if we lose connection during such an operation, do we have enough retry logic (we have some at the SshjTool level), should we check for this error state (and try remediating action), and/or should we discard this VM and retry with a new VM?

Short term, we should ensure OpenGamma demo runs on something bigger than t1.micro in AWS.