search

rhtconsulting / puppet-jboss_admin

Puppet module for runtime configuration of a JBoss container
MIT License
15 stars 12 forks source link

Add retry to onlyif and unless statements #91

Closed tstibbs closed 8 years ago

tstibbs commented 8 years ago

I often notice that after creating a datasource, jboss decides that a reload is required. Cleanup.pp calls :reload, but then immediately tries to check if a restart is required. However, as the server is currently reloading, jboss-cli.sh fails to connect. The result is that puppet fails, and the output is a mangled version of the stack trace below. It would seem sensible to make the Restart Server ${name} resource set the same timeouts etc that the Check Server Up After ${name} resource has. Note this is slightly less trivial than it sounds, because the onlyif does not have the retry logic that the main command has.

Stack trace returned that causes the whoami call to fail (probably not too important but including here in case it helps anyone):

org.jboss.as.cli.CliInitializationException: Failed to connect to the controller
        at org.jboss.as.cli.impl.CliLauncher.initCommandContext(CliLauncher.java:280)
        at org.jboss.as.cli.impl.CliLauncher.main(CliLauncher.java:240)
        at org.jboss.as.cli.CommandLineMain.main(CommandLineMain.java:34)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.jboss.modules.Module.run(Module.java:270)
        at org.jboss.modules.Main.main(Main.java:411)
Caused by: org.jboss.as.cli.CommandLineException: The controller is not available at localhost:9999
        at org.jboss.as.cli.impl.CommandContextImpl.tryConnection(CommandContextImpl.java:951)
        at org.jboss.as.cli.impl.CommandContextImpl.connectController(CommandContextImpl.java:790)
        at org.jboss.as.cli.impl.CommandContextImpl.connectController(CommandContextImpl.java:766)
        at org.jboss.as.cli.impl.CliLauncher.initCommandContext(CliLauncher.java:278)
        ... 8 more
Caused by: java.io.IOException: java.net.ConnectException: JBAS012144: Could not connect to remote://localhost:9999. The connection timed out
        at org.jboss.as.controller.client.impl.AbstractModelControllerClient.executeForResult(AbstractModelControllerClient.java:129)
        at org.jboss.as.controller.client.impl.AbstractModelControllerClient.execute(AbstractModelControllerClient.java:71)
        at org.jboss.as.cli.impl.CommandContextImpl.tryConnection(CommandContextImpl.java:929)
        ... 11 more
Caused by: java.net.ConnectException: JBAS012144: Could not connect to remote://localhost:9999. The connection timed out
        at org.jboss.as.protocol.ProtocolConnectionUtils.connectSync(ProtocolConnectionUtils.java:131)
        at org.jboss.as.protocol.ProtocolConnectionManager$EstablishingConnection.connect(ProtocolConnectionManager.java:256)
        at org.jboss.as.protocol.ProtocolConnectionManager.connect(ProtocolConnectionManager.java:70)
        at org.jboss.as.protocol.mgmt.FutureManagementChannel$Establishing.getChannel(FutureManagementChannel.java:176)
        at org.jboss.as.cli.impl.CLIModelControllerClient.getOrCreateChannel(CLIModelControllerClient.java:155)
        at org.jboss.as.cli.impl.CLIModelControllerClient$2.getChannel(CLIModelControllerClient.java:117)
        at org.jboss.as.protocol.mgmt.ManagementChannelHandler.executeRequest(ManagementChannelHandler.java:115)
        at org.jboss.as.protocol.mgmt.ManagementChannelHandler.executeRequest(ManagementChannelHandler.java:90)
        at org.jboss.as.controller.client.impl.AbstractModelControllerClient.executeRequest(AbstractModelControllerClient.java:236)
        at org.jboss.as.controller.client.impl.AbstractModelControllerClient.execute(AbstractModelControllerClient.java:141)
        at org.jboss.as.controller.client.impl.AbstractModelControllerClient.executeForResult(AbstractModelControllerClient.java:127)
        ... 13 more
itewk commented 8 years ago

@tstibbs I have been seeing "ghost" CLI connection refused messages without much rime or reason on a non consitent basis for a while now but have not had the time to track down root cause. The connection refuses I normally see are at random times and I can only surmise have to do with JBoss just happening to be to busy at that time. This looks like another good description of a similar problem.

I agree the retry logic should be added to the onlyif and unless commands. If it is okay with you I am going to rename this ticket to reflect that action item.

Thank you for the detailed bug report.

Blue Skies, ~Ian

tstibbs commented 8 years ago

Yes, I've also been wondering if this is actually a jboss bug, but hopefully the changes I've suggested will help mitigate it anyway. Thanks.

itewk commented 8 years ago

@tstibbs I did speak with Red Hat JBoss Support on this topic and there are not any known issues with CLI connection issues. They did give some instructions for looking into the specific deployment I was working with though that helped to midigate issues somewhat. I believe in my case there is resource consumption issues with the application running on JBoss EAP causing the CLI not to have enough resources to respond to connection requests, but that is just a theory.

This enhancement will definitely be useful. If you feel up to the challenge feel free to submit a patch.

itewk commented 8 years ago

approved.

itewk commented 8 years ago

merged into master.