search

eclipse-ee4j / glassfish

Eclipse GlassFish
https://eclipse-ee4j.github.io/glassfish/
386 stars 144 forks source link

MS1042: failed to send heartbeatmessage with state=aliveandready to group #16721

Closed glassfishrobot closed 13 years ago

glassfishrobot commented 13 years ago

I have an Apache and glassfish with mod_jk. Many times, I have an error http 500 when i request my apache.

On my log, i have always this line [#|2011-05-23T19:54:19.749+0200|WARNING|glassfish3.1|ShoalLogger|_ThreadID=11;_ThreadName=Thread-1;|GMS1042: failed to send heartbeatmessage with state=aliveandready to group autre-voyage. Reason: IOException:Operation not permitted|#]

For info, start log:

19-May-2011 23:10:42 com.sun.enterprise.admin.launcher.GFLauncherLogger info INFO: JVM invocation command line: /usr/lib/jvm/java-6-openjdk/bin/java -cp /opt/glassfish3/glassfish/modules/glassfish.jar -XX:+UnlockDiagnosticVMOptions -XX:MaxPermSize=192m -XX:+AggressiveOpts -XX:NewRatio=2 -XX:+UseParallelGC -Xmx512m -Xmn192m -javaagent:/opt/glassfish3/glassfish/lib/monitor/btrace-agent.jar=unsafe=true,noServer=true -server -Dosgi.shell.telnet.maxconn=1 -Djdbc.drivers=org.apache.derby.jdbc.ClientDriver -Dfelix.fileinstall.disableConfigSave=false -Dfelix.fileinstall.dir=/opt/glassfish3/glassfish/modules/autostart/ -Djavax.net.ssl.keyStore=/opt/glassfish3/glassfish/domains/domain1/config/keystore.jks -Dosgi.shell.telnet.port=6666 -Djava.security.policy=/opt/glassfish3/glassfish/domains/domain1/config/server.policy -Dfelix.fileinstall.log.level=2 -Dfelix.fileinstall.poll=5000 -Dcom.sun.aas.instanceRoot=/opt/glassfish3/glassfish/domains/domain1 -Dosgi.shell.telnet.ip=127.0.0.1 -Dcom.sun.enterprise.config.config_environment_factory_class=com.sun.enterprise.config.serverbeans.AppserverConfigEnvironmentFactory -Djava.endorsed.dirs=/opt/glassfish3/glassfish/modules/endorsed:/opt/glassfish3/glassfish/lib/endorsed -Dcom.sun.aas.installRoot=/opt/glassfish3/glassfish -Djava.ext.dirs=/usr/lib/jvm/java-6-openjdk/lib/ext:/usr/lib/jvm/java-6-openjdk/jre/lib/ext:/opt/glassfish3/glassfish/domains/domain1/lib/ext -Dfelix.fileinstall.bundles.startTransient=true -Dfelix.fileinstall.bundles.new.start=true -Djavax.net.ssl.trustStore=/opt/glassfish3/glassfish/domains/domain1/config/cacerts.jks -Dorg.glassfish.additionalOSGiBundlesToStart=org.apache.felix.shell,org.apache.felix.gogo.runtime,org.apache.felix.gogo.shell,org.apache.felix.gogo.command -Dcom.sun.enterprise.security.httpsOutboundKeyAlias=s1as -DANTLR_USE_DIRECT_CLASS_LOADING=true -Djava.security.auth.login.config=/opt/glassfish3/glassfish/domains/domain1/config/login.conf Dgosh.args=-nointeractive -Djava.library.path=/opt/glassfish3/glassfish/lib:/usr/lib/jvm/java-6-openjdk/jre/lib/amd64/server:/usr/lib/jvm/java-6-openjdk/jre/lib/amd64:/usr/lib/jvm/java-6-openjdk/lib/amd64:/usr/java/packages/lib/amd64:/usr/lib/jni:/lib:/usr/lib com.sun.enterprise.glassfish.bootstrap.ASMain -domainname domain1 -asadmin-args -host,,,localhost,,,port,,,4848,,,secure=false,,,terse=false,,,echo=false,,,interactive=true,,,start-domain,,,verbose=false,,,debug=false,,,-domaindir,,,/opt/glassfish3/glassfish/domains,,,domain1 -instancename server -verbose false -debug false -asadmin-classpath /opt/glassfish3/glassfish/modules/admin-cli.jar -asadmin-classname com.sun.enterprise.admin.cli.AsadminMain -upgrade false -type DAS -domaindir /opt/glassfish3/glassfish/domains/domain1 -read-stdin true 19-May-2011 23:10:42 com.sun.enterprise.admin.launcher.GFLauncherLogger info INFO: Successfully launched in 8 msec. [#|2011-05-19T23:11:11.590+0200|INFO|glassfish3.1|null|_ThreadID=1;_ThreadName=Thread-1;|Running GlassFish Version: GlassFish Server Open Source Edition 3.1 (build 43)|#]

[#|2011-05-19T23:11:11.821+0200|INFO|glassfish3.1|javax.org.glassfish.gms.org.glassfish.gms|_ThreadID=1;_ThreadName=Thread-1;|GMSAD1005: Member server joined group autre-voyage|#]

[#|2011-05-19T23:11:11.828+0200|INFO|glassfish3.1|javax.org.glassfish.gms.org.glassfish.gms|_ThreadID=1;_ThreadName=Thread-1;|GMSAD1004: Started GMS for instance server in group autre-voyage|#]

[#|2011-05-19T23:11:11.882+0200|INFO|glassfish3.1|javax.org.glassfish.gms.org.glassfish.gms|_ThreadID=1;_ThreadName=Thread-1;|GMSAD1005: Member server joined group evasion|#]

[#|2011-05-19T23:11:11.883+0200|INFO|glassfish3.1|javax.org.glassfish.gms.org.glassfish.gms|_ThreadID=1;_ThreadName=Thread-1;|GMSAD1004: Started GMS for instance server in group evasion|#]

[#|2011-05-19T23:11:12.117+0200|INFO|glassfish3.1|javax.enterprise.system.core.com.sun.enterprise.v3.services.impl|_ThreadID=59;_ThreadName=Thread-1;|Grizzly Framework 1.9.31 started in: 76ms - bound to [0.0.0.0:4848]|#]

[#|2011-05-19T23:11:12.117+0200|INFO|glassfish3.1|javax.enterprise.system.core.com.sun.enterprise.v3.services.impl|_ThreadID=56;_ThreadName=Thread-1;|Grizzly Framework 1.9.31 started in: 87ms - bound to [0.0.0.0:8181]|#]

[#|2011-05-19T23:11:12.112+0200|INFO|glassfish3.1|javax.enterprise.system.core.com.sun.enterprise.v3.services.impl|_ThreadID=57;_ThreadName=Thread-1;|Grizzly Framework 1.9.31 started in: 32ms - bound to [0.0.0.0:7676]|#]

[#|2011-05-19T23:11:12.112+0200|INFO|glassfish3.1|javax.enterprise.system.core.com.sun.enterprise.v3.services.impl|_ThreadID=58;_ThreadName=Thread-1;|Grizzly Framework 1.9.31 started in: 49ms - bound to [0.0.0.0:3700]|#]

[#|2011-05-19T23:11:12.140+0200|INFO|glassfish3.1|javax.enterprise.system.core.com.sun.enterprise.v3.admin.adapter|_ThreadID=1;_ThreadName=Thread-1;|The Admin Console is already installed, but not yet loaded.|#]

[#|2011-05-19T23:11:12.556+0200|INFO|glassfish3.1|javax.enterprise.system.container.web.com.sun.enterprise.web|_ThreadID=1;_ThreadName=Thread-1;|WEB0170: Apache mod_jk/jk2 attached to virtual-server [server] listening on port [8,080]|#]

[#|2011-05-19T23:11:12.560+0200|INFO|glassfish3.1|javax.enterprise.system.container.web.com.sun.enterprise.web|_ThreadID=1;_ThreadName=Thread-1;|WEB0169: Created HTTP listener [http-listener-2] on host/port [0.0.0.0:8181]|#]

[#|2011-05-19T23:11:12.567+0200|INFO|glassfish3.1|javax.enterprise.system.container.web.com.sun.enterprise.web|_ThreadID=1;_ThreadName=Thread-1;|WEB0169: Created HTTP listener [admin-listener] on host/port [0.0.0.0:4848]|#]

[#|2011-05-19T23:11:12.596+0200|INFO|glassfish3.1|javax.enterprise.system.container.web.com.sun.enterprise.web|_ThreadID=1;_ThreadName=Thread-1;|WEB0171: Created virtual server [server]|#]

[#|2011-05-19T23:11:12.599+0200|INFO|glassfish3.1|javax.enterprise.system.container.web.com.sun.enterprise.web|_ThreadID=1;_ThreadName=Thread-1;|WEB0171: Created virtual server [__asadmin]|#]

[#|2011-05-19T23:11:13.015+0200|INFO|glassfish3.1|javax.enterprise.system.container.web.com.sun.enterprise.web|_ThreadID=1;_ThreadName=Thread-1;|WEB0172: Virtual server [server] loaded default web module []|#]

[#|2011-05-19T23:11:13.117+0200|SEVERE|glassfish3.1|org.apache.jk.server.JkMain|_ThreadID=10;_ThreadName=Thread-1;|No class name for worker.gfAutreVoyage worker|#]

[#|2011-05-19T23:11:13.120+0200|SEVERE|glassfish3.1|org.apache.jk.server.JkMain|_ThreadID=10;_ThreadName=Thread-1;|No class name for worker.gfAutreVoyage worker|#]

[#|2011-05-19T23:11:13.120+0200|SEVERE|glassfish3.1|org.apache.jk.server.JkMain|_ThreadID=10;_ThreadName=Thread-1;|No class name for worker.gfAdmin worker|#]

[#|2011-05-19T23:11:13.121+0200|SEVERE|glassfish3.1|org.apache.jk.server.JkMain|_ThreadID=10;_ThreadName=Thread-1;|No class name for worker.gfAdmin worker|#]

[#|2011-05-19T23:11:13.121+0200|SEVERE|glassfish3.1|org.apache.jk.server.JkMain|_ThreadID=10;_ThreadName=Thread-1;|No class name for worker.gfEvasion worker|#]

[#|2011-05-19T23:11:13.121+0200|SEVERE|glassfish3.1|org.apache.jk.server.JkMain|_ThreadID=10;_ThreadName=Thread-1;|No class name for worker.gfAutreVoyage worker|#]

[#|2011-05-19T23:11:13.121+0200|SEVERE|glassfish3.1|org.apache.jk.server.JkMain|_ThreadID=10;_ThreadName=Thread-1;|No class name for worker.gfAdmin worker|#]

[#|2011-05-19T23:11:13.122+0200|SEVERE|glassfish3.1|org.apache.jk.server.JkMain|_ThreadID=10;_ThreadName=Thread-1;|No class name for worker.gfEvasion worker|#]

[#|2011-05-19T23:11:13.122+0200|SEVERE|glassfish3.1|org.apache.jk.server.JkMain|_ThreadID=10;_ThreadName=Thread-1;|No class name for worker.gfAutreVoyage worker|#]

[#|2011-05-19T23:11:13.122+0200|SEVERE|glassfish3.1|org.apache.jk.server.JkMain|_ThreadID=10;_ThreadName=Thread-1;|No class name for worker.gfAutreVoyage worker|#]

[#|2011-05-19T23:11:13.122+0200|SEVERE|glassfish3.1|org.apache.jk.server.JkMain|_ThreadID=10;_ThreadName=Thread-1;|No class name for worker.gfAutreVoyage worker|#]

[#|2011-05-19T23:11:13.123+0200|SEVERE|glassfish3.1|org.apache.jk.server.JkMain|_ThreadID=10;_ThreadName=Thread-1;|No class name for worker.gfAutreVoyage worker|#]

[#|2011-05-19T23:11:13.123+0200|SEVERE|glassfish3.1|org.apache.jk.server.JkMain|_ThreadID=10;_ThreadName=Thread-1;|No class name for worker.gfEvasion worker|#]

[#|2011-05-19T23:11:13.123+0200|SEVERE|glassfish3.1|org.apache.jk.server.JkMain|_ThreadID=10;_ThreadName=Thread-1;|No class name for worker.gfAutreVoyage worker|#]

[#|2011-05-19T23:11:13.124+0200|SEVERE|glassfish3.1|org.apache.jk.server.JkMain|_ThreadID=10;_ThreadName=Thread-1;|No class name for worker.gfAutreVoyage worker|#]

[#|2011-05-19T23:11:13.124+0200|SEVERE|glassfish3.1|org.apache.jk.server.JkMain|_ThreadID=10;_ThreadName=Thread-1;|No class name for worker worker|#]

[#|2011-05-19T23:11:13.124+0200|SEVERE|glassfish3.1|org.apache.jk.server.JkMain|_ThreadID=10;_ThreadName=Thread-1;|No class name for worker.gfAdmin worker|#]

[#|2011-05-19T23:11:13.124+0200|SEVERE|glassfish3.1|org.apache.jk.server.JkMain|_ThreadID=10;_ThreadName=Thread-1;|No class name for worker.gfEvasion worker|#]

[#|2011-05-19T23:11:13.124+0200|SEVERE|glassfish3.1|org.apache.jk.server.JkMain|_ThreadID=10;_ThreadName=Thread-1;|No class name for worker.gfAdmin worker|#]

[#|2011-05-19T23:11:13.125+0200|SEVERE|glassfish3.1|org.apache.jk.server.JkMain|_ThreadID=10;_ThreadName=Thread-1;|No class name for worker.gfEvasion worker|#]

[#|2011-05-19T23:11:13.138+0200|WARNING|glassfish3.1|org.apache.tomcat.util.threads.ThreadPool|_ThreadID=10;_ThreadName=Thread-1;|threadpool.max_threads_too_low|#]

[#|2011-05-19T23:11:14.206+0200|INFO|glassfish3.1|javax.enterprise.system.core.com.sun.enterprise.v3.services.impl|_ThreadID=1;_ThreadName=Thread-1;|core.start_container_done|#]

and my mod_jk config: glassfish-jk.properties file -->

  1. Define 1 real worker using ajp13 worker.list=gfEvasion, gfAdmin, gfAutreVoyage

  2. Set properties for worker1 (ajp13) worker.gfAdmin.type=ajp13 worker.gfAdmin.host=localhost worker.gfAdmin.port=8080 worker.gfAdmin.lbfactor=25 worker.gfAdmin.cachesize=10

    worker.gfAdmin.cache_timeout=600

    worker.gfAdmin.socket_keepalive=1

    worker.gfAdmin.socket_timeout=300

  3. Set properties for worker1 (ajp13) worker.gfEvasion.type=ajp13 worker.gfEvasion.host=localhost worker.gfEvasion.port=28081 worker.gfEvasion.lbfactor=25 worker.gfEvasion.cachesize=10

    worker.gfEvasion.cache_timeout=600

    worker.gfEvasion.socket_keepalive=1

    worker.gfEvasion.socket_timeout=300

worker.gfAutreVoyage.type=ajp13 worker.gfAutreVoyage.host=localhost worker.gfAutreVoyage.port=28080 worker.gfAutreVoyage.lbfactor=25 worker.gfAutreVoyage.cachesize=10 worker.gfAutreVoyage.cache_timeout=600 worker.gfAutreVoyage.socket_keepalive=1 worker.gfAutreVoyage.socket_timeout=300 worker.gfAutreVoyage.connection_pool_timeout=600

My virtual host config: JkMount /* gfAutreVoyage

and my httpd.conf

JkWorkersFile /opt/glassfish3/glassfish/domains/domain1/config/glassfish-jk.properties

  1. Where to put jk logs JkLogFile /var/log/apache2/mod_jk.log
  2. Set the jk log level [debug/error/info] JkLogLevel debug
  3. Select the log format JkLogStampFormat "[%a %b %d %H:%M:%S %Y] "
  4. JkOptions indicate to send SSL KEY SIZE, JkOptions +ForwardKeySize +ForwardURICompat -ForwardDirectories
  5. JkRequestLogFormat set the request format JkRequestLogFormat "%w %V %T"

    Environment

    ubuntu, glassfish 3.1, apache 2.2.1, mod_jk 1.2.26

    Affected Versions

    [3.1]

glassfishrobot commented 6 years ago
glassfishrobot commented 13 years ago

@glassfishrobot Commented ai109478 said: User is using openJDK. Perhaps it would be a good idea to try HotSpot (Sun) JDK from here: http://www.oracle.com/technetwork/java/javase/downloads/index.html

glassfishrobot commented 13 years ago

@glassfishrobot Commented @jfialli said: These comments only pertain to following reported log message and the subject of this issue.

[#|2011-05-23T19:54:19.749+0200|WARNING|glassfish3.1|ShoalLogger|_ThreadID=11;_ThreadName=Thread-1;|GMS1042: failed to send heartbeatmessage with state=aliveandready to group autre-voyage. Reason: IOException:Operation not permitted|#]

Shoal GMS requires UDP multicast to be configured properly to be used between glassfish clustered instances. The above failure indicates that multicast may not be configured to be allowed at the OS networking layer.

The following link provides guidance on verifying UDP multicast is configured properly. http://blogs.oracle.com/bobby/entry/validating_multicast_transport_where_d

The reported failure is a network or security configuration issue that is not allowing the UDP multicast operation to succeed so this issue will be marked invalid.

The following error message is not a GMS issue.

[#|2011-05-19T23:11:13.117+0200|SEVERE|glassfish3.1|org.apache.jk.server.JkMain|_ThreadID=10;_ThreadName=Thread-1;|No class name for worker.gfAutreVoyage worker|#]

It should be filed as different issue with a different subcomponent.

glassfishrobot commented 13 years ago

@glassfishrobot Commented @jfialli said: Marking this issue as incomplete since information was not provided that illustrated that UDP multicast was verified to be working properly. The reported failure "Operation invalid" on sending a UDP multicast message indicates that UDP multicast is not enabled.

Following the directions on this link: http://blogs.oracle.com/bobby/entry/validating_multicast_transport_where_d should resolve the network configuration issue.

If it does not, please submit "ifconfig -a" output (or equivalent if OS does not support that command), the DAS server.log with ShoalLogger log level of CONFIG. (provides what network addresses and ports are being used)

% asadmin set-log-levels ShoalLogger=CONFIG

Run the above before creating cluster.

The ifconfig output should have "MULTICAST" on a line similar to following: en0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500

glassfishrobot commented 7 years ago

@glassfishrobot Commented This issue was imported from java.net JIRA GLASSFISH-16721

glassfishrobot commented 13 years ago

@glassfishrobot Commented Reported by sebglon

glassfishrobot commented 13 years ago

@glassfishrobot Commented Marked as incomplete on Wednesday, May 25th 2011, 2:51:41 am