[REGRESSION]Failed to receive messages after changing master broker and restarting cluster

[glassfishrobot]

build: GF4.0 promoted build27

I am doing regression test by running existing SQE jms test against GF4.0 b27, I was noticed a test failure in appserver-sqe/pe/jms/jmscluster/test2. This is a regression bug since the same test passed against GF v3.1.2. Here is the test description:

The test does the following

• creates a Glassfish Cluster (gfjmscluster1) with three instances (gfjmsinstance1, gfjmsinstance2, gfjmsinstance3) with MQ Conventional Cluster with a Master Broker.
• deploys an MDB that listens on a Queue Destination.
• send a message to the MDB
• Verify MDB replies the msg
• Change the Master Broker in the cluster to a different Broker
• Verify that the Master Broker Change
• send a message to MDB
• Verify that the MDB receives the msg
• Once again Change the Master Broker in the cluster to a different Broker
• Stop/Restart the cluster
• Verify the Master Broker Change
• send a message to MDB
• Verify that the MDB receives the msg

Here are the steps to reproduce the bug:

1. Install GF4.0, start domain domain1 2. Checkout SQE workspace cvs co appserver-sqe/bootstrap.xml (CVSROOT: :pserver:@sunsw.us.oracle.com:/m/jws cd appserver-sqe ant -f bootstrap.xml co-jms 3.set env. variables S1AS_HOME , for example: /export/sonia/v4/glassfish3/glassfish SPS_HOME , for example: /export/sonia/appserver-sqe ANT_HOME , for example: /export/sonia/ant-1.7.1 JAVA_HOME , for example: /export/sonia/jdk1.7.0 4. cd appserver-sqe/pe/jms/jmscluster/test2, run "ant clean build setup deploy run", one test failed:

[exec] Start the Test : jmscluster-test2-AppClient3 [exec] Mar 14, 2012 12:50:50 PM org.hibernate.validator.util.Version [exec] INFO: Hibernate Validator 4.1.0.Final [exec] Mar 14, 2012 12:50:50 PM org.hibernate.validator.engine.resolver.DefaultTraversableResolver detectJPA [exec] INFO: Instantiated an instance of org.hibernate.validator.engine.resolver.JPATraversableResolver. [exec] Mar 14, 2012 12:50:50 PM com.sun.messaging.jms.ra.ResourceAdapter start [exec] INFO: MQJMSRA_RA1101: GlassFish MQ JMS Resource Adapter: Version: 4.6 (Build 3-c) Compile: Fri May 20 13:52:27 PDT 2011 [exec] Mar 14, 2012 12:50:50 PM com.sun.messaging.jms.ra.ResourceAdapter start [exec] INFO: MQJMSRA_RA1101: GlassFish MQ JMS Resource Adapter starting: broker is REMOTE, connection mode is TCP [exec] Mar 14, 2012 12:50:50 PM com.sun.messaging.jms.ra.ResourceAdapter start [exec] INFO: MQJMSRA_RA1101: GlassFish MQ JMS Resource Adapter Started:REMOTE [exec] jmscluster-test2-AppClient3 setup completed [exec] NULL Message Received [exec] Generating report at /export/sonia/appserver-sqe/test_results.xml [exec] [exec] - jmscluster-test2-AppClient3: FAIL - [exec] ----------------------------------------- [exec] Total PASS: 0 [exec] Total FAIL: 1 [exec] Total DNR: 0 [exec] ----------------------------------------- [exec] Mar 14, 2012 12:50:59 PM com.sun.messaging.jms.ra.ResourceAdapter stop [exec] INFO: MQJMSRA_RA1101: GlassFish MQ JMS Resource Adapter stopping... [exec] Mar 14, 2012 12:50:59 PM com.sun.messaging.jms.ra.ResourceAdapter stop [exec] INFO: MQJMSRA_RA1101: GlassFish MQ JMS Resource Adapter stopped. [exec] Mar 14, 2012 12:50:59 PM com.sun.enterprise.connectors.service.ResourceAdapterAdminServiceImpl sendStopToResourceAdapter [exec] INFO: RAR7094: jmsra shutdown successful.

During the test execution, I was noticed that it took very long time to restart cluster though it was successful started. I saw some exceptions in the cluster instance server.log (more exceptions in the attached server.log.instance1 file): [#|2012-03-14T12:50:07.180-0700|SEVERE|44.0|javax.enterprise.system.std.com.sun.enterprise.server.logging|_ThreadID=1;_ThreadName=Thread-2;|com.sun.messaging.jmq.jmsclient.ConnectException: [C4003]: Error occurred on connection creation [localhost:1117,mq://localhost:1117/,mq://localhost:3337/,mq://localhost:2227/]. at com.sun.messaging.jmq.jmsclient.ConnectionInitiator.createConnectionNew(ConnectionInitiator.java:316) at com.sun.messaging.jmq.jmsclient.ConnectionInitiator.createConnection(ConnectionInitiator.java:214) at com.sun.messaging.jmq.jmsclient.ConnectionInitiator.createConnection(ConnectionInitiator.java:164) at com.sun.messaging.jmq.jmsclient.ProtocolHandler.init(ProtocolHandler.java:843) at com.sun.messaging.jmq.jmsclient.ProtocolHandler.(ProtocolHandler.java:1562) at com.sun.messaging.jmq.jmsclient.ConnectionImpl.openConnection(ConnectionImpl.java:2383) at com.sun.messaging.jmq.jmsclient.ConnectionImpl.init(ConnectionImpl.java:1064) at com.sun.messaging.jmq.jmsclient.ConnectionImpl.(ConnectionImpl.java:442) at com.sun.messaging.jmq.jmsclient.UnifiedConnectionImpl.(UnifiedConnectionImpl.java:66) at com.sun.messaging.jmq.jmsclient.XAConnectionImpl.(XAConnectionImpl.java:64) at com.sun.messaging.XAConnectionFactory.createXAConnection(XAConnectionFactory.java:97) at com.sun.messaging.XAConnectionFactory.createXAConnection(XAConnectionFactory.java:75) at com.sun.messaging.jms.ra.EndpointConsumer.createRemoteMessageConsumer(EndpointConsumer.java:456) at com.sun.messaging.jms.ra.EndpointConsumer._init(EndpointConsumer.java:308) at com.sun.messaging.jms.ra.EndpointConsumer.(EndpointConsumer.java:185) at com.sun.messaging.jms.ra.ResourceAdapter.endpointActivation(ResourceAdapter.java:478) at com.sun.enterprise.connectors.inbound.ConnectorMessageBeanClient.setup(ConnectorMessageBeanClient.java:225) at com.sun.ejb.containers.MessageBeanContainer.(MessageBeanContainer.java:208) at com.sun.ejb.containers.ContainerFactoryImpl.createContainer(ContainerFactoryImpl.java:113) at org.glassfish.ejb.startup.EjbApplication.loadContainers(EjbApplication.java:228) at org.glassfish.ejb.startup.EjbDeployer.load(EjbDeployer.java:294) at org.glassfish.ejb.startup.EjbDeployer.load(EjbDeployer.java:102) at org.glassfish.internal.data.ModuleInfo.load(ModuleInfo.java:186) at org.glassfish.internal.data.ApplicationInfo.load(ApplicationInfo.java:264) at com.sun.enterprise.v3.server.ApplicationLifecycle.deploy(ApplicationLifecycle.java:508) at com.sun.enterprise.v3.server.ApplicationLoaderService.processApplication(ApplicationLoaderService.java:388) at com.sun.enterprise.v3.server.ApplicationLoaderService.postConstruct(ApplicationLoaderService.java:224) at com.sun.hk2.component.AbstractCreatorImpl.inject(AbstractCreatorImpl.java:132) at com.sun.hk2.component.ConstructorCreator.initialize(ConstructorCreator.java:117) at com.sun.hk2.component.AbstractCreatorImpl.get(AbstractCreatorImpl.java:84) at com.sun.hk2.component.SingletonInhabitant.get(SingletonInhabitant.java:67) at com.sun.hk2.component.EventPublishingInhabitant.get(EventPublishingInhabitant.java:141) at com.sun.hk2.component.AbstractInhabitantImpl.get(AbstractInhabitantImpl.java:135) at com.sun.enterprise.v3.server.StartupRunLevelBridge.activate(StartupRunLevelBridge.java:93) at com.sun.enterprise.v3.server.RunLevelBridge.postConstruct(RunLevelBridge.java:110) at com.sun.enterprise.v3.server.StartupRunLevelBridge.postConstruct(StartupRunLevelBridge.java:65) at com.sun.hk2.component.AbstractCreatorImpl.inject(AbstractCreatorImpl.java:132) at com.sun.hk2.component.ConstructorCreator.initialize(ConstructorCreator.java:117) at com.sun.hk2.component.AbstractCreatorImpl.get(AbstractCreatorImpl.java:84) at com.sun.hk2.component.SingletonInhabitant.get(SingletonInhabitant.java:67) at com.sun.hk2.component.EventPublishingInhabitant.get(EventPublishingInhabitant.java:141) at com.sun.hk2.component.EventPublishingInhabitant.get(EventPublishingInhabitant.java:141) at com.sun.hk2.component.RunLevelInhabitant.get(RunLevelInhabitant.java:110) at com.sun.hk2.component.AbstractInhabitantImpl.get(AbstractInhabitantImpl.java:135) at com.sun.enterprise.v3.server.AppServerStartup$StartupInhabitantActivator.activate(AppServerStartup.java:526) at org.jvnet.hk2.component.internal.runlevel.DefaultRunLevelService$Worker.activateRunLevel(DefaultRunLevelService.java:1106) at org.jvnet.hk2.component.internal.runlevel.DefaultRunLevelService$Worker.upActiveRecorder(DefaultRunLevelService.java:1060) at org.jvnet.hk2.component.internal.runlevel.DefaultRunLevelService$Worker.run(DefaultRunLevelService.java:1026) at org.jvnet.hk2.component.internal.runlevel.DefaultRunLevelService$SyncProceedToOp.proceedTo(DefaultRunLevelService.java:1256) at org.jvnet.hk2.component.internal.runlevel.DefaultRunLevelService.proceedTo(DefaultRunLevelService.java:797) at org.jvnet.hk2.component.internal.runlevel.DefaultRunLevelService.proceedTo(DefaultRunLevelService.java:759) at com.sun.enterprise.v3.server.AppServerStartup.proceedTo(AppServerStartup.java:360) at com.sun.enterprise.v3.server.AppServerStartup.run(AppServerStartup.java:254) at com.sun.enterprise.v3.server.AppServerStartup.doStart(AppServerStartup.java:172) at com.sun.enterprise.v3.server.AppServerStartup.start(AppServerStartup.java:163) at com.sun.enterprise.glassfish.bootstrap.GlassFishImpl.start(GlassFishImpl.java:79) at com.sun.enterprise.glassfish.bootstrap.GlassFishDecorator.start(GlassFishDecorator.java:63) at com.sun.enterprise.glassfish.bootstrap.osgi.OSGiGlassFishImpl.start(OSGiGlassFishImpl.java:71) at com.sun.enterprise.glassfish.bootstrap.GlassFishMain$Launcher.launch(GlassFishMain.java:117) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at com.sun.enterprise.glassfish.bootstrap.GlassFishMain.main(GlassFishMain.java:97) at com.sun.enterprise.glassfish.bootstrap.ASMain.main(ASMain.java:55)

1. 1. 1. 1. 1. 1. Connect exception for : mq://localhost:1117/jms com.sun.messaging.jms.JMSException: [C4003]: Error occurred on connection creation [localhost:1117]. - cause: java.net.ConnectException: Connection refused at com.sun.messaging.jmq.jmsclient.ExceptionHandler.throwConnectionException(ExceptionHandler.java:280) at com.sun.messaging.jmq.jmsclient.ExceptionHandler.handleConnectException(ExceptionHandler.java:226) at com.sun.messaging.jmq.jmsclient.PortMapperClient.readBrokerPorts(PortMapperClient.java:247) at com.sun.messaging.jmq.jmsclient.PortMapperClient.init(PortMapperClient.java:156) at com.sun.messaging.jmq.jmsclient.PortMapperClient.(PortMapperClient.java:98) at com.sun.messaging.jmq.jmsclient.protocol.tcp.TCPConnectionHandler.(TCPConnectionHandler.java:171) at com.sun.messaging.jmq.jmsclient.protocol.tcp.TCPStreamHandler.openConnection(TCPStreamHandler.java:141) at com.sun.messaging.jmq.jmsclient.ConnectionInitiator.createConnection(ConnectionInitiator.java:785) at com.sun.messaging.jmq.jmsclient.ConnectionInitiator.createConnectionNew(ConnectionInitiator.java:260) at com.sun.messaging.jmq.jmsclient.ConnectionInitiator.createConnection(ConnectionInitiator.java:214) at com.sun.messaging.jmq.jmsclient.ConnectionInitiator.createConnection(ConnectionInitiator.java:164) at com.sun.messaging.jmq.jmsclient.ProtocolHandler.init(ProtocolHandler.java:843) at com.sun.messaging.jmq.jmsclient.ProtocolHandler.(ProtocolHandler.java:1562) at com.sun.messaging.jmq.jmsclient.ConnectionImpl.openConnection(ConnectionImpl.java:2383) at com.sun.messaging.jmq.jmsclient.ConnectionImpl.init(ConnectionImpl.java:1064) at com.sun.messaging.jmq.jmsclient.ConnectionImpl.(ConnectionImpl.java:442) at com.sun.messaging.jmq.jmsclient.UnifiedConnectionImpl.(UnifiedConnectionImpl.java:66) at com.sun.messaging.jmq.jmsclient.XAConnectionImpl.(XAConnectionImpl.java:64) at com.sun.messaging.XAConnectionFactory.createXAConnection(XAConnectionFactory.java:97) at com.sun.messaging.XAConnectionFactory.createXAConnection(XAConnectionFactory.java:75) at com.sun.messaging.jms.ra.E|#]

[#|2012-03-14T12:50:07.184-0700|SEVERE|44.0|javax.enterprise.system.std.com.sun.enterprise.server.logging|_ThreadID=1;_ThreadName=Thread-2;|ndpointConsumer.createRemoteMessageConsumer(EndpointConsumer.java:456) at com.sun.messaging.jms.ra.EndpointConsumer._init(EndpointConsumer.java:308) at com.sun.messaging.jms.ra.EndpointConsumer.(EndpointConsumer.java:185) at com.sun.messaging.jms.ra.ResourceAdapter.endpointActivation(ResourceAdapter.java:478) at com.sun.enterprise.connectors.inbound.ConnectorMessageBeanClient.setup(ConnectorMessageBeanClient.java:225) at com.sun.ejb.containers.MessageBeanContainer.(MessageBeanContainer.java:208) at com.sun.ejb.containers.ContainerFactoryImpl.createContainer(ContainerFactoryImpl.java:113) at org.glassfish.ejb.startup.EjbApplication.loadContainers(EjbApplication.java:228) at org.glassfish.ejb.startup.EjbDeployer.load(EjbDeployer.java:294) at org.glassfish.ejb.startup.EjbDeployer.load(EjbDeployer.java:102) at org.glassfish.internal.data.ModuleInfo.load(ModuleInfo.java:186) at org.glassfish.internal.data.ApplicationInfo.load(ApplicationInfo.java:264) at com.sun.enterprise.v3.server.ApplicationLifecycle.deploy(ApplicationLifecycle.java:508) at com.sun.enterprise.v3.server.ApplicationLoaderService.processApplication(ApplicationLoaderService.java:388) at com.sun.enterprise.v3.server.ApplicationLoaderService.postConstruct(ApplicationLoaderService.java:224) at com.sun.hk2.component.AbstractCreatorImpl.inject(AbstractCreatorImpl.java:132) at com.sun.hk2.component.ConstructorCreator.initialize(ConstructorCreator.java:117) at com.sun.hk2.component.AbstractCreatorImpl.get(AbstractCreatorImpl.java:84) at com.sun.hk2.component.SingletonInhabitant.get(SingletonInhabitant.java:67) at com.sun.hk2.component.EventPublishingInhabitant.get(EventPublishingInhabitant.java:141) at com.sun.hk2.component.AbstractInhabitantImpl.get(AbstractInhabitantImpl.java:135) at com.sun.enterprise.v3.server.StartupRunLevelBridge.activate(StartupRunLevelBridge.java:93) at com.sun.enterprise.v3.server.RunLevelBridge.postConstruct(RunLevelBridge.java:110) at com.sun.enterprise.v3.server.StartupRunLevelBridge.postConstruct(StartupRunLevelBridge.java:65) at com.sun.hk2.component.AbstractCreatorImpl.inject(AbstractCreatorImpl.java:132) at com.sun.hk2.component.ConstructorCreator.initialize(ConstructorCreator.java:117) at com.sun.hk2.component.AbstractCreatorImpl.get(AbstractCreatorImpl.java:84) at com.sun.hk2.component.SingletonInhabitant.get(SingletonInhabitant.java:67) at com.sun.hk2.component.EventPublishingInhabitant.get(EventPublishingInhabitant.java:141) at com.sun.hk2.component.EventPublishingInhabitant.get(EventPublishingInhabitant.java:141) at com.sun.hk2.component.RunLevelInhabitant.get(RunLevelInhabitant.java:110) at com.sun.hk2.component.AbstractInhabitantImpl.get(AbstractInhabitantImpl.java:135) at com.sun.enterprise.v3.server.AppServerStartup$StartupInhabitantActivator.activate(AppServerStartup.java:526) at org.jvnet.hk2.component.internal.runlevel.DefaultRunLevelService$Worker.activateRunLevel(DefaultRunLevelService.java:1106) at org.jvnet.hk2.component.internal.runlevel.DefaultRunLevelService$Worker.upActiveRecorder(DefaultRunLevelService.java:1060) at org.jvnet.hk2.component.internal.runlevel.DefaultRunLevelService$Worker.run(DefaultRunLevelService.java:1026) at org.jvnet.hk2.component.internal.runlevel.DefaultRunLevelService$SyncProceedToOp.proceedTo(DefaultRunLevelService.java:1256) at org.jvnet.hk2.component.internal.runlevel.DefaultRunLevelService.proceedTo(DefaultRunLevelService.java:797) at org.jvnet.hk2.component.internal.runlevel.DefaultRunLevelService.proceedTo(DefaultRunLevelService.java:759) at com.sun.enterprise.v3.server.AppServerStartup.proceedTo(AppServerStartup.java:360) at com.sun.enterprise.v3.server.AppServerStartup.run(AppServerStartup.java:254) at com.sun.enterprise.v3.server.AppServerStartup.doStart(AppServerStartup.java:172) at com.sun.enterprise.v3.server.AppServerStartup.start(AppServerStartup.java:163) at com.sun.enterprise.glassfish.bootstrap.GlassFishImpl.start(GlassFishImpl.java:79) at com.sun.enterprise.glassfish.bootstrap.GlassFishDecorator.start(GlassFishDecorator.java:63) at com.sun.enterprise.glassfish.bootstrap.osgi.OSGiGlassFishImpl.start(OSGiGlassFishImpl.java:71) at com.sun.enterprise.glassfish.bootstrap.GlassFishMain$Launcher.launch(GlassFishMain.java:117) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at com.sun.enterprise.glassfish.bootstrap.GlassFishMain.main(GlassFishMain.java:97) at com.sun.enterprise.glassfish.bootstrap.ASMain.main(ASMain.java:55) Caused by: java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:337) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:198) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:180) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391) at java.net.Socket.connect(Socket.java:578) at java.net.Socket.connect(Socket.java:527) at java.net.Socket.(Socket.java:424) at java.net.Socket.(Socket.java:207) at com.sun.messaging.jmq.jmsclient.PortMapperClient.makeSocketWithTimeout(PortMapperClient.java:274) at com.sun.messaging.jmq.jmsclient.PortMapperClient.readBrokerPorts(PortMapperClient.java:226) ... 70 more

1. 1. 1. 1. 1. 1. Connect exception for : mq://localhost:1117/jms

---

[#|2012-03-14T12:50:07.667-0700|SEVERE|44.0|javax.enterprise.system.container.ejb.mdb.com.sun.ejb.containers|_ThreadID=1;_ThreadName=Thread-2;|MDB00017: [MessageQueueBean]: Exception in creating message-driven bean container: [java.lang.Exception]|#]

[#|2012-03-14T12:50:07.668-0700|SEVERE|44.0|javax.enterprise.system.container.ejb.mdb.com.sun.ejb.containers|_ThreadID=1;_ThreadName=Thread-2;|java.lang.Exception java.lang.Exception at com.sun.enterprise.connectors.inbound.ConnectorMessageBeanClient.setup(ConnectorMessageBeanClient.java:233) at com.sun.ejb.containers.MessageBeanContainer.(MessageBeanContainer.java:208) at com.sun.ejb.containers.ContainerFactoryImpl.createContainer(ContainerFactoryImpl.java:113) at org.glassfish.ejb.startup.EjbApplication.loadContainers(EjbApplication.java:228) at org.glassfish.ejb.startup.EjbDeployer.load(EjbDeployer.java:294) at org.glassfish.ejb.startup.EjbDeployer.load(EjbDeployer.java:102) at org.glassfish.internal.data.ModuleInfo.load(ModuleInfo.java:186) at org.glassfish.internal.data.ApplicationInfo.load(ApplicationInfo.java:264) at com.sun.enterprise.v3.server.ApplicationLifecycle.deploy(ApplicationLifecycle.java:508) at com.sun.enterprise.v3.server.ApplicationLoaderService.processApplication(ApplicationLoaderService.java:388) at com.sun.enterprise.v3.server.ApplicationLoaderService.postConstruct(ApplicationLoaderService.java:224) at com.sun.hk2.component.AbstractCreatorImpl.inject(AbstractCreatorImpl.java:132) at com.sun.hk2.component.ConstructorCreator.initialize(ConstructorCreator.java:117) at com.sun.hk2.component.AbstractCreatorImpl.get(AbstractCreatorImpl.java:84) at com.sun.hk2.component.SingletonInhabitant.get(SingletonInhabitant.java:67) at com.sun.hk2.component.EventPublishingInhabitant.get(EventPublishingInhabitant.java:141) at com.sun.hk2.component.AbstractInhabitantImpl.get(AbstractInhabitantImpl.java:135) at com.sun.enterprise.v3.server.StartupRunLevelBridge.activate(StartupRunLevelBridge.java:93) at com.sun.enterprise.v3.server.RunLevelBridge.postConstruct(RunLevelBridge.java:110) at com.sun.enterprise.v3.server.StartupRunLevelBridge.postConstruct(StartupRunLevelBridge.java:65) at com.sun.hk2.component.AbstractCreatorImpl.inject(AbstractCreatorImpl.java:132) at com.sun.hk2.component.ConstructorCreator.initialize(ConstructorCreator.java:117) at com.sun.hk2.component.AbstractCreatorImpl.get(AbstractCreatorImpl.java:84) at com.sun.hk2.component.SingletonInhabitant.get(SingletonInhabitant.java:67) at com.sun.hk2.component.EventPublishingInhabitant.get(EventPublishingInhabitant.java:141) at com.sun.hk2.component.EventPublishingInhabitant.get(EventPublishingInhabitant.java:141) at com.sun.hk2.component.RunLevelInhabitant.get(RunLevelInhabitant.java:110) at com.sun.hk2.component.AbstractInhabitantImpl.get(AbstractInhabitantImpl.java:135) at com.sun.enterprise.v3.server.AppServerStartup$StartupInhabitantActivator.activate(AppServerStartup.java:526) at org.jvnet.hk2.component.internal.runlevel.DefaultRunLevelService$Worker.activateRunLevel(DefaultRunLevelService.java:1106) at org.jvnet.hk2.component.internal.runlevel.DefaultRunLevelService$Worker.upActiveRecorder(DefaultRunLevelService.java:1060) at org.jvnet.hk2.component.internal.runlevel.DefaultRunLevelService$Worker.run(DefaultRunLevelService.java:1026) at org.jvnet.hk2.component.internal.runlevel.DefaultRunLevelService$SyncProceedToOp.proceedTo(DefaultRunLevelService.java:1256) at org.jvnet.hk2.component.internal.runlevel.DefaultRunLevelService.proceedTo(DefaultRunLevelService.java:797) at org.jvnet.hk2.component.internal.runlevel.DefaultRunLevelService.proceedTo(DefaultRunLevelService.java:759) at com.sun.enterprise.v3.server.AppServerStartup.proceedTo(AppServerStartup.java:360) at com.sun.enterprise.v3.server.AppServerStartup.run(AppServerStartup.java:254) at com.sun.enterprise.v3.server.AppServerStartup.doStart(AppServerStartup.java:172) at com.sun.enterprise.v3.server.AppServerStartup.start(AppServerStartup.java:163) at com.sun.enterprise.glassfish.bootstrap.GlassFishImpl.start(GlassFishImpl.java:79) at com.sun.enterprise.glassfish.bootstrap.GlassFishDecorator.start(GlassFishDecorator.java:63) at com.sun.enterprise.glassfish.bootstrap.osgi.OSGiGlassFishImpl.start(OSGiGlassFishImpl.java:71) at com.sun.enterprise.glassfish.bootstrap.GlassFishMain$Launcher.launch(GlassFishMain.java:117) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at com.sun.enterprise.glassfish.bootstrap.GlassFishMain.main(GlassFishMain.java:97) at com.sun.enterprise.glassfish.bootstrap.ASMain.main(ASMain.java:55) Caused by: javax.resource.NotSupportedException: MQRA:EC:Error:createRemoteMessageConsumer failed:aborting due to:[C4003]: Error occurred on connection creation [localhost:1117,mq://localhost:1117/,mq://localhost:3337/,mq://localhost:2227/]. at com.sun.messaging.jms.ra.EndpointConsumer.createRemoteMessageConsumer(EndpointConsumer.java:478) at com.sun.messaging.jms.ra.EndpointConsumer._init(EndpointConsumer.java:308) at com.sun.messaging.jms.ra.EndpointConsumer.(EndpointConsumer.java:185) at com.sun.messaging.jms.ra.ResourceAdapter.endpointActivation(ResourceAdapter.java:478) at com.sun.enterprise.connectors.inbound.ConnectorMessageBeanClient.setup(ConnectorMessageBeanClient.java:225) ... 48 more Caused by: com.sun.messaging.jmq.jmsclient.ConnectException: [C4003]: Error occurred on connection creation [localhost:1117,mq://localhost:1117/,mq://localhost:3337/,mq://localhost:2227/]. at com.sun.messaging.jmq.jmsclient.ConnectionInitiator.createConnectionNew(ConnectionInitiator.java:316) at com.sun.messaging.jmq.jmsclient.ConnectionInitiator.createConnection(ConnectionInitiator.java:214) at com.sun.messaging.jmq.jmsclient.ConnectionInitiator.createConnection(ConnectionInitiator.java:164) at com.sun.messaging.jmq.jmsclient.ProtocolHandler.init(ProtocolHandler.java:843) at com.sun.messaging.jmq.jmsclient.ProtocolHandler.(ProtocolHandler.java:1562) at com.sun.messaging.jmq.jmsclient.ConnectionImpl.openConnection(ConnectionImpl.java:2383) at com.sun.messaging.jmq.jmsclient.ConnectionImpl.init(ConnectionImpl.java:1064) at com.sun.messaging.jmq.jmsclient.ConnectionImpl.(ConnectionImpl.java:442) at com.sun.messaging.jmq.jmsclient.UnifiedConnectionImpl.(UnifiedConnectionImpl.java:66) at com.sun.messaging.jmq.jmsclient.XAConnectionImpl.(XAConnectionImpl.java:64) at com.sun.messaging.XAConnectionFactory.createXAConnection(XAConnectionFactory.java:97) at com.sun.messaging.XAConnectionFactory.createXAConnection(XAConnectionFactory.java:75) at com.sun.messaging.jms.ra.EndpointConsumer.createRemoteMessageConsumer(EndpointConsumer.java:456) ... 52 more

| #] |

[#|2012-03-14T12:50:07.680-0700|SEVERE|44.0|javax.enterprise.system.tools.deployment.org.glassfish.deployment.common|_ThreadID=1;_ThreadName=Thread-2;|Exception while invoking class org.glassfish.ejb.startup.EjbDeployer load method java.lang.RuntimeException: EJB Container initialization error at org.glassfish.ejb.startup.EjbApplication.loadContainers(EjbApplication.java:240) at org.glassfish.ejb.startup.EjbDeployer.load(EjbDeployer.java:294) at org.glassfish.ejb.startup.EjbDeployer.load(EjbDeployer.java:102) at org.glassfish.internal.data.ModuleInfo.load(ModuleInfo.java:186) at org.glassfish.internal.data.ApplicationInfo.load(ApplicationInfo.java:264) at com.sun.enterprise.v3.server.ApplicationLifecycle.deploy(ApplicationLifecycle.java:508) at com.sun.enterprise.v3.server.ApplicationLoaderService.processApplication(ApplicationLoaderService.java:388) at com.sun.enterprise.v3.server.ApplicationLoaderService.postConstruct(ApplicationLoaderService.java:224) at com.sun.hk2.component.AbstractCreatorImpl.inject(AbstractCreatorImpl.java:132) at com.sun.hk2.component.ConstructorCreator.initialize(ConstructorCreator.java:117) at com.sun.hk2.component.AbstractCreatorImpl.get(AbstractCreatorImpl.java:84) at com.sun.hk2.component.SingletonInhabitant.get(SingletonInhabitant.java:67) at com.sun.hk2.component.EventPublishingInhabitant.get(EventPublishingInhabitant.java:141) at com.sun.hk2.component.AbstractInhabitantImpl.get(AbstractInhabitantImpl.java:135) at com.sun.enterprise.v3.server.StartupRunLevelBridge.activate(StartupRunLevelBridge.java:93) at com.sun.enterprise.v3.server.RunLevelBridge.postConstruct(RunLevelBridge.java:110) at com.sun.enterprise.v3.server.StartupRunLevelBridge.postConstruct(StartupRunLevelBridge.java:65) at com.sun.hk2.component.AbstractCreatorImpl.inject(AbstractCreatorImpl.java:132) at com.sun.hk2.component.ConstructorCreator.initialize(ConstructorCreator.java:117) at com.sun.hk2.component.AbstractCreatorImpl.get(AbstractCreatorImpl.java:84) at com.sun.hk2.component.SingletonInhabitant.get(SingletonInhabitant.java:67) at com.sun.hk2.component.EventPublishingInhabitant.get(EventPublishingInhabitant.java:141) at com.sun.hk2.component.EventPublishingInhabitant.get(EventPublishingInhabitant.java:141) at com.sun.hk2.component.RunLevelInhabitant.get(RunLevelInhabitant.java:110) at com.sun.hk2.component.AbstractInhabitantImpl.get(AbstractInhabitantImpl.java:135) at com.sun.enterprise.v3.server.AppServerStartup$StartupInhabitantActivator.activate(AppServerStartup.java:526) at org.jvnet.hk2.component.internal.runlevel.DefaultRunLevelService$Worker.activateRunLevel(DefaultRunLevelService.java:1106) at org.jvnet.hk2.component.internal.runlevel.DefaultRunLevelService$Worker.upActiveRecorder(DefaultRunLevelService.java:1060) at org.jvnet.hk2.component.internal.runlevel.DefaultRunLevelService$Worker.run(DefaultRunLevelService.java:1026) at org.jvnet.hk2.component.internal.runlevel.DefaultRunLevelService$SyncProceedToOp.proceedTo(DefaultRunLevelService.java:1256) at org.jvnet.hk2.component.internal.runlevel.DefaultRunLevelService.proceedTo(DefaultRunLevelService.java:797) at org.jvnet.hk2.component.internal.runlevel.DefaultRunLevelService.proceedTo(DefaultRunLevelService.java:759) at com.sun.enterprise.v3.server.AppServerStartup.proceedTo(AppServerStartup.java:360) at com.sun.enterprise.v3.server.AppServerStartup.run(AppServerStartup.java:254) at com.sun.enterprise.v3.server.AppServerStartup.doStart(AppServerStartup.java:172) at com.sun.enterprise.v3.server.AppServerStartup.start(AppServerStartup.java:163) at com.sun.enterprise.glassfish.bootstrap.GlassFishImpl.start(GlassFishImpl.java:79) at com.sun.enterprise.glassfish.bootstrap.GlassFishDecorator.start(GlassFishDecorator.java:63) at com.sun.enterprise.glassfish.bootstrap.osgi.OSGiGlassFishImpl.start(OSGiGlassFishImpl.java:71) at com.sun.enterprise.glassfish.bootstrap.GlassFishMain$Launcher.launch(GlassFishMain.java:117) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at com.sun.enterprise.glassfish.bootstrap.GlassFishMain.main(GlassFishMain.java:97) at com.sun.enterprise.glassfish.bootstrap.ASMain.main(ASMain.java:55) Caused by: java.lang.Exception at com.sun.enterprise.connectors.inbound.ConnectorMessageBeanClient.setup(ConnectorMessageBeanClient.java:233) at com.sun.ejb.containers.MessageBeanContainer.(MessageBeanContainer.java:208) at com.sun.ejb.containers.ContainerFactoryImpl.createContainer(ContainerFactoryImpl.java:113) at org.glassfish.ejb.startup.EjbApplication.loadContainers(EjbApplication.java:228) ... 45 more Caused by: javax.resource.NotSupportedException: MQRA:EC:Error:createRemoteMessageConsumer failed:aborting due to:[C4003]: Error occurred on connection creation [localhost:1117,mq://localhost:1117/,mq://localhost:3337/,mq://localhost:2227/]. at com.sun.messaging.jms.ra.EndpointConsumer.createRemoteMessageConsumer(EndpointConsumer.java:478) at com.sun.messaging.jms.ra.EndpointConsumer._init(EndpointConsumer.java:308) at com.sun.messaging.jms.ra.EndpointConsumer.(EndpointConsumer.java:185) at com.sun.messaging.jms.ra.ResourceAdapter.endpointActivation(ResourceAdapter.java:478) at com.sun.enterprise.connectors.inbound.ConnectorMessageBeanClient.setup(ConnectorMessageBeanClient.java:225) ... 48 more Caused by: com.sun.messaging.jmq.jmsclient.ConnectException: [C4003]: Error occurred on connection creation [localhost:1117,mq://localhost:1117/,mq://localhost:3337/,mq://localhost:2227/]. at com.sun.messaging.jmq.jmsclient.ConnectionInitiator.createConnectionNew(ConnectionInitiator.java:316) at com.sun.messaging.jmq.jmsclient.ConnectionInitiator.createConnection(ConnectionInitiator.java:214) at com.sun.messaging.jmq.jmsclient.ConnectionInitiator.createConnection(ConnectionInitiator.java:164) at com.sun.messaging.jmq.jmsclient.ProtocolHandler.init(ProtocolHandler.java:843) at com.sun.messaging.jmq.jmsclient.ProtocolHandler.(ProtocolHandler.java:1562) at com.sun.messaging.jmq.jmsclient.ConnectionImpl.openConnection(ConnectionImpl.java:2383) at com.sun.messaging.jmq.jmsclient.ConnectionImpl.init(ConnectionImpl.java:1064) at com.sun.messaging.jmq.jmsclient.ConnectionImpl.(ConnectionImpl.java:442) at com.sun.messaging.jmq.jmsclient.UnifiedConnectionImpl.(UnifiedConnectionImpl.java:66) at com.sun.messaging.jmq.jmsclient.XAConnectionImpl.(XAConnectionImpl.java:64) at com.sun.messaging.XAConnectionFactory.createXAConnection(XAConnectionFactory.java:97) at com.sun.messaging.XAConnectionFactory.createXAConnection(XAConnectionFactory.java:75) at com.sun.messaging.jms.ra.EndpointConsumer.createRemoteMessageConsumer(EndpointConsumer.java:456) ... 52 more

| #] |

In the broker log it displayed the following message: [14/Mar/2012:12:44:14 PDT] WARNING [B2105]: Attempting to initiate a cluster connection to mq://10.133.185.9:3337/?instName=???&brokerSessionUID=null&ha=false failed: Connection refused [14/Mar/2012:12:44:14 PDT] WARNING [B2105]: Attempting to initiate a cluster connection to mq://10.133.185.9:2227/?instName=???&brokerSessionUID=null&ha=false failed: Connection refused [14/Mar/2012:12:44:14 PDT] [B1039]: Broker "gfjmscluster1gfjmsinstance1@jed-asqe-2:1117" ready. [14/Mar/2012:12:45:14 PDT] WARNING [B2180]: Still trying to connect to the master broker mq://10.133.185.9:3337/?instName=???&brokerSessionUID=null&ha=false. Client connections will have restricted JMS service until the master broker is ready. [14/Mar/2012:12:47:34 PDT] WARNING [B2105]: Attempting to initiate a cluster connection to mq://10.133.185.9:3337/?instName=???&brokerSessionUID=null&ha=false failed: Connection refused [14/Mar/2012:12:47:34 PDT] WARNING [B2105]: Attempting to initiate a cluster connection to mq://10.133.185.9:2227/?instName=???&brokerSessionUID=null&ha=false failed: Connection refused [14/Mar/2012:12:48:14 PDT] WARNING [B2180]: Still trying to connect to the master broker mq://10.133.185.9:3337/?instName=???&brokerSessionUID=null&ha=false. Client connections will have restricted JMS service until the master broker is ready. [14/Mar/2012:12:50:09 PDT] [B1251]: Closing cluster connection with mq://10.133.185.9:2227/?instName=gfjmscluster1gfjmsinstance2&brokerSessionUID=593971654969893120&ha=false. Waiting for master broker mq://10.133.185.9:3337/?instName=???&brokerSessionUID=null&ha=false ... [14/Mar/2012:12:50:09 PDT] WARNING :

5. I did some debugging, if I removed the following targets in the "run" target(That means if we don't restart the cluster after changing master broker), the test passed against GF4.0, no exceptions in the server.log. antcall target="jms-stop-cluster" /> antcall target="jms-start-cluster"/>

---

Attached domain1 server.log, instance1 server.log and instance1 broker log.

Environment

solaris10/jdk1.7.0

Affected Versions

[4.0_dev]

[glassfishrobot]

• Issue Imported From: https://github.com/javaee/glassfish/issues/18510
• Original Issue Raised By:@glassfishrobot
• Original Issue Assigned To: @yaminikb
• Original Issue Closed By:@glassfishrobot

[glassfishrobot]

@glassfishrobot Commented liang.x.zhao said: Fixed it by initializing jms lazy grizzly proxy when mdb deployment at glassfish startup. Checkin 53199.

[glassfishrobot]

@glassfishrobot Commented File: log.txt.instance1 Attached By: sonialiu

[glassfishrobot]

@glassfishrobot Commented File: server.log.domain1 Attached By: sonialiu

[glassfishrobot]

@glassfishrobot Commented File: server.log.instance1 Attached By: sonialiu

[glassfishrobot]

@glassfishrobot Commented Was assigned to liang.x.zhao

[glassfishrobot]

@glassfishrobot Commented This issue was imported from java.net JIRA GLASSFISH-18510

[glassfishrobot]

@glassfishrobot Commented Reported by sonialiu

[glassfishrobot]

@glassfishrobot Commented Marked as fixed on Tuesday, March 27th 2012, 6:06:24 pm