MasterNode ensure delivery of GMS notifications over UDP

[glassfishrobot]

GMS membership notifications such as JOIN, JOIN_AND_READY, FAILURE, PLANNED_SHUTDOWN, FAILURE_SUSPECTED, GroupLeadership are broadcasted from MasterNode over UDP. A protocol will be developed between masters and other members of group to ensure that this event is delivered and, if not, master will resend the event to ALIVE instances that have not acked receiving the notification.

Currently, ensuring that the MasterNode is not a heavily loaded application, (such as Domain Application Server in Glassfish that does not run apps) and configuring via OS tuning of UDP buffers has ensured UDP messages are not dropped. Addressing this issue will provide robustness of event delivery w/o requiring OS tuning or partition of application load from Shoal GMS MasterNode.

Environment

Operating System: All Platform: All

Affected Versions

[current]

[glassfishrobot]

• Issue Imported From: https://github.com/javaee/shoal/issues/107
• Original Issue Raised By:@glassfishrobot
• Original Issue Assigned To: @jfialli
• Original Issue Closed By:@glassfishrobot

[glassfishrobot]

@glassfishrobot Commented Reported by @jfialli

[glassfishrobot]

@glassfishrobot Commented @jfialli said: Fix checked in.

The master sends along latest MasterViewID with every heartbeat message it broadcasts. The gms group members record each MasterChangeEvent MasterViewID it has received. When gms group member detects that it has not received a specific masterViewID, it requests the master resend to just itself (via more reliable TCP).

Tested this with simulated failure injection. Wrote ReliableMulticast junit test.

[glassfishrobot]

@glassfishrobot Commented This issue was imported from java.net JIRA SHOAL-107

[glassfishrobot]

@glassfishrobot Commented Marked as fixed on Thursday, October 7th 2010, 3:40:51 am