Created cluster sends heartbeat to 228.8.7.9 by Default

[glassfishrobot]

Create cluster using both either wiki instruction: https://glassfish.dev.java.net/javaee5/build/GlassFish_LB_Cluster.html/Setting Up a One Machine Cluster

Or the following CLI steps:

[sunshine]:mei:/home/mei/work/glassfish/bin > asadmin create-cluster cluster1 [sunshine]:mei:/home/mei/work/glassfish/bin > asadmin create-node-agent cluster1-nodeagent [sunshine]:mei:/home/mei/work/glassfish/bin > asadmin start-node-agent cluster1-nodeagent [sunshine]:mei:/home/mei/work/glassfish/bin > asadmin create-instance --cluster cluster1 --nodeagent cluster1-nodeagent --systemproperties "JMX_SYSTEM_CONNECTOR_PORT=18687:IIOP_LISTENER_PORT=13330:IIOP_SSL_LISTENER_PORT=14440:IIOPSSL\ MUTUALAUTH_PORT=15550:HTTP_LISTENER_PORT=11110:HTTP_SSL_LISTENER_PORT=12220" instance-ONE

[sunshine]:mei:/home/mei/work/glassfish/bin > asadmin create-instance --cluster cluster1 --nodeagent cluster1-nodeagent --systemproperties "JMX_SYSTEM_CONNECTOR_PORT=18688:IIOP_LISTENER_PORT=13331:IIOP_SSL_LISTENER_PORT=14441:IIOPSSL\ MUTUALAUTH_PORT=15551:HTTP_LISTENER_PORT=11111:HTTP_SSL_LISTENER_PORT=12221" instance-TWO Using 37,677 for JMS_PROVIDER_PORT.

The created nodeagent and instances all send heartbeat to address: 228.8.7.9 :

[sunshine]:mei:/home/mei/work/glassfish/nodeagents > findInFiles 228.8.7.9 ./cluster1-nodeagent/agent/config/domain.xml: <cluster config-ref="cluster1-config" heartbeat-address="228.8.7.9" heartbeat-enabled="true" heartbeat-port="12934" name="cluster1"> ./cluster1-nodeagent/instance-ONE/config/domain.xml: <cluster config-ref="cluster1-config" heartbeat-address="228.8.7.9" heartbeat-enabled="true" heartbeat-port="12934" name="cluster1"> ./cluster1-nodeagent/instance-TWO/config/domain.xml: <cluster config-ref="cluster1-config" heartbeat-address="228.8.7.9" heartbeat-enabled="true" heartbeat-port="12934" name="cluster1">

This brings the network to a crawl, I'm not sure if the cluster set up by default is actually working or not

Environment

Operating System: All Platform: All

Affected Versions

[9.1peur1]

[glassfishrobot]

• Issue Imported From: https://github.com/javaee/glassfish/issues/4000
• Original Issue Raised By:@glassfishrobot
• Original Issue Assigned To: @jfialli
• Original Issue Closed By:@glassfishrobot

[glassfishrobot]

@glassfishrobot Commented mei_wu said: Please not the version of glassfish is :v2ur1-b09c

[glassfishrobot]

@glassfishrobot Commented shreedhar_ganapathy said: Hi Could you use v2 ur1 FCS which is b09d We had integrated a new Jxta jar after fixing a memory leak therein.

And for clarification, although the domain.xml is shred in all instances and nodeagents, only the das and cluster instances are involved in sending heartbeats in the network. Could you please share your machine details in terms of memory capacity, cpu power, and also the jdk version you are using? Thanks Shreedhar

[glassfishrobot]

@glassfishrobot Commented shreedhar_ganapathy said: Also can you provide us with information on how you reached to the conclusion that there is network saturation? Is it possible you do not have enough RAM for two instances and a DAS ? Are you running other applications on this machine such as an IDE which can also consume RAM space ?

Lowering to a p3 pending response from the bug submitter.

[glassfishrobot]

@glassfishrobot Commented mei_wu said: I'm using Sun Java System Application Server 9.1_01 (build b09d-fcs), this is the latest valid GF that JBI runtime core supports, please see http://download.java.net/jbi/binaries/open-esb/main/common/install-jbi-runtime-jar.html

System capacity info:

1. /usr/platform/uname -i/sbin/prtdiag System Configuration: Sun Microsystems sun4u Sun Ultra 45 Workstation System clock frequency: 200 MHZ Memory size: 4GB

==================================== CPUs ==================================== E$ CPU CPU CPU Freq Size Implementation Mask Status Location — -------- ---------- --------------------- ----- ------ -------- 0 1600 MHz 1MB SUNW,UltraSPARC-IIIi 3.4 on-line MB/0 1 1600 MHz 1MB SUNW,UltraSPARC-IIIi 3.4 on-line MB/1

================================= IO Devices ================================= Bus Freq Slot + Name + Type MHz Status Path Model

---

pci 200 MB pci10b9,5455 (sound) okay /pci@1e,600000/pci@0/pci@1/pci@0/sound

pci 200 MB pci10b9,5229 (ide) okay /pci@1e,600000/pci@0/pci@1/pci@0/ide

pci 200 MB pci14e4,1678 (network) okay /pci@1e,600000/pci@0/pci@2/pci@0/network@4

pci 200 MB pci14e4,1678 (network) okay /pci@1e,600000/pci@0/pci@2/pci@0/network@4,1

pci 200 MB scsi-pci1000,50 (scsi-2) LSI,1064 okay /pci@1e,600000/pci@0/pci@9/pci@0/scsi@1

pci 200 MB/PCIX0 SUNW,XVR-100 (display) SUNW,375-3290 okay /pci@1e,600000/pci@0/pci@9/pci@0,2/SUNW,XVR-100@2

============================ Memory Configuration ============================ Segment Table:

Base Address Size Interleave Factor Contains

0x0 2GB 2 BankIDs 0,1 0x200000000 2GB 1 BankIDs 2

Bank Table:

Physical Location ID ControllerID GroupID Size Interleave Way

0 0 0 1GB 0,1 1 0 0 1GB 2 0 1 2GB 0

Memory Module Groups:

ControllerID GroupID Labels Status

0 0 MB/DIMM3 0 0 MB/DIMM1 0 1 MB/DIMM2 0 1 MB/DIMM0

=============================== usb Devices ===============================

Name Port#

---

mouse 1 keyboard 2

=============================== usb Devices ===============================

Name Port#

---

hub 2 hub 7

=============================== hub#2 Devices ===============================

Name Port#

---

device 4

=============================== device#4 Devices ===============================

Name Port#

---

keyboard mouse #

I believe it is due to the wrong heart beat port because before manually changing the port, the DAS can not be reached via NFS and the pint returns in a extremely slow speed. Internet on DAS is slow to a crawl. After I manually changed the port to point to DAS, everything goes back to normal and nodeagent and cluster instances are not affected.

[glassfishrobot]

@glassfishrobot Commented mei_wu said: Java version:

1. java -version java version "1.5.0_11" Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_11-b03) Java HotSpot(TM) Server VM (build 1.5.0_11-b03, mixed mode)

[glassfishrobot]

@glassfishrobot Commented shreedhar_ganapathy said:

I believe it is due to the wrong heart beat port because before manually changing the port, the DAS can not be reached via NFS and the pint returns in a extremely slow speed. Internet on DAS is slow to a crawl.

The heartbeat port is a multicast address based port. It has nothing to do with DAS's port. The fact that DAS cannot be reached via NFS could be a different matter. Could you tell us why you need to reach DAS via NFS?

After I manually changed the port to point to DAS, everything goes back to normal and nodeagent and cluster instances are not affected.

Which port are you referring to?

Also could you share with us steps to reproduce the behavior you are seeing? If possible please share your application.

Are you running under load ? If so, what tool do you use to generate load?

How many requests are you sending to the cluster?

Have you tuned the appserver both in terms of heap space and the number of request processing threads to support your load assuming you are seeing this problem under load ?

Responses to the above would help us a lot in narrowing down the problem.

[glassfishrobot]

@glassfishrobot Commented mei_wu said: About NFS, in fact, I'm only testing clustering and install GF in cluster mode on the solaris working machine that I use for daily dev work, it is not a dedicated server. I use NFS to link file systems of solaris box and a linux box, but it breaks down after starting nodeagent.

I then pinged DAS from my linux box and it shows extremely slow return on 1 request every a few requests.

I then locally try to access internet through DAS, ping works okay but slow, but browing websites, such as sun wiki seem to halt.

It is fairly easy to reproduce, as I said in the ticket, you can either follow : Create cluster using both either wiki instruction: https://glassfish.dev.java.net/javaee5/build/GlassFish_LB_Cluster.html/Setting Up a One Machine Cluster

r the following CLI steps:

[sunshine]:mei:/home/mei/work/glassfish/bin > asadmin create-cluster cluster1 [sunshine]:mei:/home/mei/work/glassfish/bin > asadmin create-node-agent cluster1-nodeagent [sunshine]:mei:/home/mei/work/glassfish/bin > asadmin start-node-agent cluster1-nodeagent [sunshine]:mei:/home/mei/work/glassfish/bin > asadmin create-instance --cluster cluster1 --nodeagent cluster1-nodeagent --systemproperties "JMX_SYSTEM_CONNECTOR_PORT=18687:IIOP_LISTENER_PORT=13330:IIOP_SSL_LISTENER_PORT=14440:IIOPSSL\ MUTUALAUTH_PORT=15550:HTTP_LISTENER_PORT=11110:HTTP_SSL_LISTENER_PORT=12220" instance-ONE

You may need to make sure 228.8.7.9 is unreachable from you machine, as my machine isn't on SWAN, it is unreachable for me.

You do not have to run any apps or send any requests, it is reproducible for me every time I go through the process

[glassfishrobot]

@glassfishrobot Commented mei_wu said: You may do a text search on 228.8.7.9 on domain.xml after creating the node-agent and instances.

[glassfishrobot]

@glassfishrobot Commented mei_wu said: On my machine, after creating the node agent, I do a text search, I got: [sunshine]:mei:/home/mei/work/glassfish > findInFiles 228.8.7.9 ./domains/domain1/config/domain.xml: <cluster config-ref="cluster1-config" heartbeat-address="228.8.7.9" heartbeat-enabled="true" heartbeat-port="43147" name="cluster1">

[glassfishrobot]

@glassfishrobot Commented harpreet said: Approving for v2.1 based on feedback from Sheetal

[glassfishrobot]

@glassfishrobot Commented harpreet said: removing from approved list as issue not critical to release.

[glassfishrobot]

@glassfishrobot Commented sanandal said: "Reclassifying as P4 because this issue is not deemed "must fix" for this v2.1 release whose primary release driver is SailFin. This issue will be scrubbed after this release and will be given the right priority for the next release."

[glassfishrobot]

@glassfishrobot Commented @jfialli said: not enough information in this report to investigate it.

Basic Shoal GMS QA testing creates a 9 instance cluster and runs various tests on that configuration. There is nothing described in this report that differs from that basic test. Perhaps there is a machine configuration issue that is causing this issue, but not enough info to confirm that.

If this behavior is recreated using v3.1, this issue should be reopened.

For glassfish v3.1, the auto generated multicast address (when one is not specified via --multicastaddress parameter to create-cluster) is "228.7.X.Y" where X and Y are randomly generated between 0-255. Additionally, the multicastport is generated between 2000 and 32000 when it is not specified when creating the cluster.

[glassfishrobot]

@glassfishrobot Commented @jfialli said: we were unable to recreate this in the past so this bug is being closed.

[glassfishrobot]

@glassfishrobot Commented This issue was imported from java.net JIRA GLASSFISH-4000

[glassfishrobot]

@glassfishrobot Commented Reported by mei_wu

[glassfishrobot]

@glassfishrobot Commented Marked as cannot reproduce on Thursday, October 14th 2010, 8:10:53 am