openlvc / portico

Portico is an open source, cross-platform, fully supported HLA RTI implementation. Designed with modularity and flexibility in mind, Portico is a production-grade RTI for the Simulation and Training Community, so come say hi!
http://www.porticoproject.org
152 stars 81 forks source link

Running Portico example across Docker Weave does not work #140

Closed bergtwvd closed 8 years ago

bergtwvd commented 9 years ago

I set up a test with the sample program installed in a Docker container (Dockerfile to be provided later).

Test environment:

Test:

A exits with a stacktrace. Logfile are attached.

Following test succeeds: On container A: java -cp $RTI_HOME/lib/portico.jar org.jgroups.tests.McastReceiverTest -mcast_addr 228.10.10.10 -port 20913

On container B: java -cp $RTI_HOME/lib/portico.jar org.jgroups.tests.McastSenderTest -mcast_addr 228.10.10.10 -port 20913

Route table on A: Destination Gateway Genmask Flags Metric Ref Use Iface default 10.1.0.1 0.0.0.0 UG 0 0 0 eth0 10.1.0.0 * 255.255.0.0 U 0 0 0 eth0 10.10.0.0 * 255.255.255.0 U 0 0 0 ethwe 224.0.0.0 * 240.0.0.0 U 0 0 0 ethwe

Route tabel on B: Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface default 10.2.0.1 0.0.0.0 UG 0 0 0 eth0 10.2.0.0 * 255.255.0.0 U 0 0 0 eth0 10.10.0.0 * 255.255.255.0 U 0 0 0 ethwe 224.0.0.0 * 240.0.0.0 U 0 0 0 ethwe

bergtwvd commented 9 years ago

LOGFILE FOR CONTAINER B:

WARN [main] portico.lrc: MOM support is currently unsupported in IEEE-1516e federations. DEBUG [main] org.jgroups.conf.ClassConfigurator: Using jg-magic-map.xml as magic number file and jg-protocol-ids.xml for protocol IDs DEBUG [main] org.jgroups.stack.Configurator: set property UDP.bind_addr to default value /10.10.0.2 DEBUG [main] org.jgroups.stack.Configurator: set property UDP.diagnostics_addr to default value /224.0.75.75 DEBUG [main] org.jgroups.protocols.FRAG2: received CONFIG event: {bind_addr=/10.10.0.2} TRACE [main] org.jgroups.blocks.MessageDispatcher$ProtocolAdapter: setting local_addr (null) to e96f542b7216-17605 DEBUG [main] org.jgroups.protocols.FRAG2: received CONFIG event: {flush_supported=true} TRACE [main] org.jgroups.protocols.pbcast.STABLE: stable task started TRACE [main] org.jgroups.protocols.UNICAST2: e96f542b7216-17605: stable task started DEBUG [main] org.jgroups.protocols.UDP: sockets will use interface 10.10.0.2 WARN [main] org.jgroups.protocols.UDP: [JGRP00014] the send buffer of socket DatagramSocket was set to 640KB, but the OS only allocated 212.99KB. This might lead to performance problems. Please set your max send buffer in the OS correctly (e.g. net.core.wmem_max on Linux) WARN [main] org.jgroups.protocols.UDP: [JGRP00014] the receive buffer of socket DatagramSocket was set to 8MB, but the OS only allocated 212.99KB. This might lead to performance problems. Please set your max receive buffer in the OS correctly (e.g. net.core.rmem_max on Linux) WARN [main] org.jgroups.protocols.UDP: [JGRP00014] the send buffer of socket MulticastSocket was set to 640KB, but the OS only allocated 212.99KB. This might lead to performance problems. Please set your max send buffer in the OS correctly (e.g. net.core.wmem_max on Linux) WARN [main] org.jgroups.protocols.UDP: [JGRP00014] the receive buffer of socket MulticastSocket was set to 8MB, but the OS only allocated 212.99KB. This might lead to performance problems. Please set your max receive buffer in the OS correctly (e.g. net.core.rmem_max on Linux) DEBUG [main] org.jgroups.protocols.UDP: socket information: , mcast_addr=239.255.20.16:20913, bind_addr=/10.10.0.2, ttl=8 sock: bound to 10.10.0.2:56777, receive buffer size=212992, send buffer size=212992 mcast_sock: bound to 10.10.0.2:20913, send buffer size=212992, receive buffer size=212992 TRACE [main] org.jgroups.protocols.UDP: joined /224.0.75.75:7500 on ethwe TRACE [main] org.jgroups.protocols.UDP: joined /224.0.75.75:7500 on eth0 TRACE [main] org.jgroups.protocols.UDP: joined /224.0.75.75:7500 on lo TRACE [main] org.jgroups.protocols.UDP: sending msg to null, src=e96f542b7216-17605, headers are PING: [PING: type=GET_MBRS_REQ, cluster=ExampleFederation, arg=e96f542b7216-17605, view_id=, is_server=false, is_coord=false, logical_name=e96f542b7216-17605, physical_addrs=10.10.0.2:56777], UDP: [channel_name=ExampleFederation] TRACE [main] org.jgroups.protocols.UDP: looping back message [dst: , src: e96f542b7216-17605 (2 headers), size=0 bytes, flags=OOB] TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.UDP: received [dst: , src: e96f542b7216-17605 (2 headers), size=0 bytes, flags=OOB], headers are PING: [PING: type=GET_MBRS_REQ, cluster=ExampleFederation, arg=e96f542b7216-17605, view_id=, is_server=false, is_coord=false, logical_name=e96f542b7216-17605, physical_addrs=10.10.0.2:56777], UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.UDP: received [dst: , src: e96f542b7216-17605 (2 headers), size=0 bytes, flags=OOB], headers are PING: [PING: type=GET_MBRS_REQ, cluster=ExampleFederation, arg=e96f542b7216-17605, view_id=, is_server=false, is_coord=false, logical_name=e96f542b7216-17605, physical_addrs=10.10.0.2:56777], UDP: [channel_name=ExampleFederation] TRACE [Incoming] org.jgroups.protocols.PING: e96f542b7216-17605: received GET_MBRS_REQ from e96f542b7216-17605, sending staggered response [PING: type=GET_MBRS_RSP, arg=e96f542b7216-17605, view_id=, is_server=false, is_coord=false, logical_name=e96f542b7216-17605, physical_addrs=10.10.0.2:56777] TRACE [Incoming] org.jgroups.protocols.UDP: sending msg to e96f542b7216-17605, src=e96f542b7216-17605, headers are PING: [PING: type=GET_MBRS_RSP, arg=e96f542b7216-17605, view_id=, is_server=false, is_coord=false, logical_name=e96f542b7216-17605, physical_addrs=10.10.0.2:56777], UDP: [channel_name=ExampleFederation] TRACE [Incoming] org.jgroups.protocols.UDP: looping back message [dst: e96f542b7216-17605, src: e96f542b7216-17605 (2 headers), size=0 bytes, flags=OOB] TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.UDP: received [dst: e96f542b7216-17605, src: e96f542b7216-17605 (2 headers), size=0 bytes, flags=OOB], headers are PING: [PING: type=GET_MBRS_RSP, arg=e96f542b7216-17605, view_id=, is_server=false, is_coord=false, logical_name=e96f542b7216-17605, physical_addrs=10.10.0.2:56777], UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.PING: e96f542b7216-17605: received GET_MBRS_RSP from e96f542b7216-17605: e96f542b7216-17605, view_id=, is_server=false, is_coord=false, logical_name=e96f542b7216-17605, physical_addrs=10.10.0.2:56777 TRACE [main] org.jgroups.protocols.PING: discovery took 5007 ms: responses: 1 total (0 servers (0 coord), 1 clients) TRACE [main] org.jgroups.protocols.pbcast.GMS: e96f542b7216-17605: no initial members discovered: creating group as first member DEBUG [main] org.jgroups.protocols.pbcast.NAKACK2: [e96f542b7216-17605 setDigest()] existing digest: [] new digest: e96f542b7216-17605: [0 (0)] resulting digest: e96f542b7216-17605: [0 (0)] DEBUG [main] org.jgroups.protocols.pbcast.GMS: e96f542b7216-17605: installing view [e96f542b7216-17605|0] [e96f542b7216-17605] TRACE [main] org.jgroups.protocols.pbcast.STABLE: e96f542b7216-17605: resetting digest from NAKACK: e96f542b7216-17605: [0] DEBUG [main] org.jgroups.protocols.pbcast.STABLE: resuming message garbage collection TRACE [main] org.jgroups.protocols.pbcast.STABLE: e96f542b7216-17605: resetting digest from NAKACK: e96f542b7216-17605: [0] DEBUG [main] org.jgroups.protocols.pbcast.STABLE: [ergonomics] setting max_bytes to 4MB (1 members) TRACE [main] org.jgroups.protocols.UFC: new membership: [e96f542b7216-17605] TRACE [main] org.jgroups.protocols.MFC: new membership: [e96f542b7216-17605] DEBUG [main] org.jgroups.protocols.pbcast.FLUSH: e96f542b7216-17605: installing view [e96f542b7216-17605|0] [e96f542b7216-17605] DEBUG [main] org.jgroups.protocols.pbcast.FLUSH: e96f542b7216-17605: received STOP_FLUSH, unblocking FLUSH.down() and sending UNBLOCK up TRACE [main] org.jgroups.protocols.pbcast.STABLE: e96f542b7216-17605: resetting digest from NAKACK: e96f542b7216-17605: [0] DEBUG [main] org.jgroups.protocols.pbcast.STABLE: resuming message garbage collection DEBUG [main] org.jgroups.protocols.pbcast.GMS: created group (first member). My view is [e96f542b7216-17605|0], impl is org.jgroups.protocols.pbcast.CoordGmsImpl TRACE [main] org.jgroups.protocols.RSVP: e96f542b7216-17605: REQ --> null TRACE [main] org.jgroups.protocols.pbcast.NAKACK2: e96f542b7216-17605 sending e96f542b7216-17605#1 TRACE [main] org.jgroups.protocols.UDP: sending msg to null, src=e96f542b7216-17605, headers are 7777: CreateFederation, RSVP: REQ(0), NAKACK2: [MSG, seqno=1], UDP: [channel_name=ExampleFederation] TRACE [main] org.jgroups.protocols.UDP: looping back message [dst: , src: e96f542b7216-17605 (4 headers), size=24647 bytes, flags=OOB|DONT_BUNDLE|NO_FC|RSVP] TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.UDP: received [dst: , src: e96f542b7216-17605 (4 headers), size=24647 bytes, flags=OOB|DONT_BUNDLE|NO_FC|RSVP], headers are 7777: CreateFederation, RSVP: REQ(0), NAKACK2: [MSG, seqno=1], UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.pbcast.NAKACK2: e96f542b7216-17605: received e96f542b7216-17605#1 TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.pbcast.NAKACK2: e96f542b7216-17605: delivering e96f542b7216-17605#1 TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.RSVP: e96f542b7216-17605: REQ <-- e96f542b7216-17605 TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.UDP: received [dst: , src: e96f542b7216-17605 (4 headers), size=24647 bytes, flags=OOB|DONT_BUNDLE|NO_FC|RSVP], headers are 7777: CreateFederation, RSVP: REQ(0), NAKACK2: [MSG, seqno=1], UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.RSVP: e96f542b7216-17605: RSP --> e96f542b7216-17605 TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.UNICAST2: e96f542b7216-17605: created connection to e96f542b7216-17605 (conn_id=0) TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.UNICAST2: e96f542b7216-17605 --> DATA(e96f542b7216-17605: #1, conn_id=0, first) TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.UDP: sending msg to e96f542b7216-17605, src=e96f542b7216-17605, headers are RSVP: RSP(0), UNICAST2: DATA, seqno=1, first, UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.UDP: looping back message [dst: e96f542b7216-17605, src: e96f542b7216-17605 (3 headers), size=0 bytes, flags=OOB|RSVP] TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.UDP: received [dst: e96f542b7216-17605, src: e96f542b7216-17605 (3 headers), size=0 bytes, flags=OOB|RSVP], headers are RSVP: RSP(0), UNICAST2: DATA, seqno=1, first, UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.UNICAST2: e96f542b7216-17605 <-- DATA(e96f542b7216-17605: #1, first) TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.UNICAST2: e96f542b7216-17605: created receiver window for e96f542b7216-17605 at seqno=#1 for conn-id=0 TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.RSVP: e96f542b7216-17605: RSP <-- e96f542b7216-17605 TRACE [main] org.jgroups.protocols.RSVP: e96f542b7216-17605: REQ --> null TRACE [main] org.jgroups.protocols.pbcast.NAKACK2: e96f542b7216-17605 sending e96f542b7216-17605#2 TRACE [main] org.jgroups.protocols.UDP: sending msg to null, src=e96f542b7216-17605, headers are 7777: JoinFederation, RSVP: REQ(1), NAKACK2: [MSG, seqno=2], UDP: [channel_name=ExampleFederation] TRACE [main] org.jgroups.protocols.UDP: looping back message [dst: , src: e96f542b7216-17605 (4 headers), size=1 bytes, flags=OOB|DONT_BUNDLE|NO_FC|RSVP] TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.UDP: received [dst: , src: e96f542b7216-17605 (4 headers), size=1 bytes, flags=OOB|DONT_BUNDLE|NO_FC|RSVP], headers are 7777: JoinFederation, RSVP: REQ(1), NAKACK2: [MSG, seqno=2], UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.pbcast.NAKACK2: e96f542b7216-17605: received e96f542b7216-17605#2 TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.pbcast.NAKACK2: e96f542b7216-17605: delivering e96f542b7216-17605#2 TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.RSVP: e96f542b7216-17605: REQ <-- e96f542b7216-17605 TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.RSVP: e96f542b7216-17605: RSP --> e96f542b7216-17605 TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.UNICAST2: e96f542b7216-17605 --> DATA(e96f542b7216-17605: #2, conn_id=0) TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.UDP: sending msg to e96f542b7216-17605, src=e96f542b7216-17605, headers are RSVP: RSP(1), UNICAST2: DATA, seqno=2, UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.UDP: looping back message [dst: e96f542b7216-17605, src: e96f542b7216-17605 (3 headers), size=0 bytes, flags=OOB|RSVP] TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.UDP: received [dst: e96f542b7216-17605, src: e96f542b7216-17605 (3 headers), size=0 bytes, flags=OOB|RSVP], headers are RSVP: RSP(1), UNICAST2: DATA, seqno=2, UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.UNICAST2: e96f542b7216-17605 <-- DATA(e96f542b7216-17605: #2) TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.RSVP: e96f542b7216-17605: RSP <-- e96f542b7216-17605 TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.UDP: received [dst: , src: e96f542b7216-17605 (4 headers), size=1 bytes, flags=OOB|DONT_BUNDLE|NO_FC|RSVP], headers are 7777: JoinFederation, RSVP: REQ(1), NAKACK2: [MSG, seqno=2], UDP: [channel_name=ExampleFederation] TRACE [main] org.jgroups.protocols.pbcast.NAKACK2: e96f542b7216-17605 sending e96f542b7216-17605#3 TRACE [main] org.jgroups.protocols.UDP: sending msg to null, src=e96f542b7216-17605, headers are NAKACK2: [MSG, seqno=3], UDP: [channel_name=ExampleFederation] TRACE [main] org.jgroups.protocols.UDP: looping back message [dst: , src: e96f542b7216-17605 (2 headers), size=2525 bytes] TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.UDP: received [dst: , src: e96f542b7216-17605 (2 headers), size=2525 bytes], headers are NAKACK2: [MSG, seqno=3], UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.pbcast.NAKACK2: e96f542b7216-17605: received e96f542b7216-17605#3 TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.pbcast.NAKACK2: e96f542b7216-17605: delivering e96f542b7216-17605#3 TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.MFC: e96f542b7216-17605 used 2525 credits, 3997475 remaining TRACE [main] org.jgroups.protocols.pbcast.NAKACK2: e96f542b7216-17605 sending e96f542b7216-17605#4 TRACE [main] org.jgroups.protocols.UDP: sending msg to null, src=e96f542b7216-17605, headers are NAKACK2: [MSG, seqno=4], UDP: [channel_name=ExampleFederation] TRACE [main] org.jgroups.protocols.UDP: looping back message [dst: , src: e96f542b7216-17605 (2 headers), size=293 bytes] TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.UDP: received [dst: , src: e96f542b7216-17605 (2 headers), size=293 bytes], headers are NAKACK2: [MSG, seqno=4], UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.pbcast.NAKACK2: e96f542b7216-17605: received e96f542b7216-17605#4 TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.pbcast.NAKACK2: e96f542b7216-17605: delivering e96f542b7216-17605#4 TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.MFC: e96f542b7216-17605 used 293 credits, 3997182 remaining TRACE [TransferQueueBundler,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.TP$TransferQueueBundler: sending 2 msgs (2930 bytes (4.65% of max_bundle_size) to 1 destination(s) TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.UDP: received [dst: , src: e96f542b7216-17605 (2 headers), size=2525 bytes], headers are NAKACK2: [MSG, seqno=3], UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.UDP: received [dst: , src: e96f542b7216-17605 (2 headers), size=293 bytes], headers are NAKACK2: [MSG, seqno=4], UDP: [channel_name=ExampleFederation] TRACE [main] org.jgroups.protocols.pbcast.NAKACK2: e96f542b7216-17605 sending e96f542b7216-17605#5 TRACE [main] org.jgroups.protocols.UDP: sending msg to null, src=e96f542b7216-17605, headers are NAKACK2: [MSG, seqno=5], UDP: [channel_name=ExampleFederation] TRACE [main] org.jgroups.protocols.UDP: looping back message [dst: , src: e96f542b7216-17605 (2 headers), size=303 bytes] TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.UDP: received [dst: , src: e96f542b7216-17605 (2 headers), size=303 bytes], headers are NAKACK2: [MSG, seqno=5], UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.pbcast.NAKACK2: e96f542b7216-17605: received e96f542b7216-17605#5 TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.pbcast.NAKACK2: e96f542b7216-17605: delivering e96f542b7216-17605#5 TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.MFC: e96f542b7216-17605 used 303 credits, 3996879 remaining TRACE [TransferQueueBundler,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.TP$TransferQueueBundler: sending 1 msgs (359 bytes (0.57% of max_bundle_size) to 1 destination(s) TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.UDP: received [dst: , src: e96f542b7216-17605 (2 headers), size=303 bytes], headers are NAKACK2: [MSG, seqno=5], UDP: [channel_name=ExampleFederation] TRACE [Incoming] org.jgroups.protocols.pbcast.STABLE: e96f542b7216-17605: setting latest_local_digest from NAKACK: e96f542b7216-17605: [5] TRACE [Incoming] org.jgroups.protocols.pbcast.STABLE: e96f542b7216-17605: sending stable msg e96f542b7216-17605: [5] TRACE [Incoming] org.jgroups.protocols.UDP: sending msg to null, src=e96f542b7216-17605, headers are STABLE: [STABLE_GOSSIP]: digest is e96f542b7216-17605: [5 (5)], UDP: [channel_name=ExampleFederation] TRACE [Incoming] org.jgroups.protocols.UDP: looping back message [dst: , src: e96f542b7216-17605 (2 headers), size=0 bytes, flags=OOB|NO_RELIABILITY] TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.UDP: received [dst: , src: e96f542b7216-17605 (2 headers), size=0 bytes, flags=OOB|NO_RELIABILITY], headers are STABLE: [STABLE_GOSSIP]: digest is e96f542b7216-17605: [5 (5)], UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.UDP: received [dst: , src: e96f542b7216-17605 (2 headers), size=0 bytes, flags=OOB|NO_RELIABILITY], headers are STABLE: [STABLE_GOSSIP]: digest is e96f542b7216-17605: [5 (5)], UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.pbcast.STABLE: e96f542b7216-17605: handling digest from e96f542b7216-17605 (0 votes): mine: e96f542b7216-17605: [0] other: e96f542b7216-17605: [5] result: e96f542b7216-17605: [0]

TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.pbcast.STABLE: e96f542b7216-17605: sending stability msg (in 294 ms) e96f542b7216-17605: [0] TRACE [Incoming] org.jgroups.protocols.pbcast.STABLE: e96f542b7216-17605: sending stability msg e96f542b7216-17605: [0] TRACE [Incoming] org.jgroups.protocols.UDP: sending msg to null, src=e96f542b7216-17605, headers are STABLE: [STABILITY]: digest is e96f542b7216-17605: [0 (5)], UDP: [channel_name=ExampleFederation] TRACE [Incoming] org.jgroups.protocols.UDP: looping back message [dst: , src: e96f542b7216-17605 (2 headers), size=0 bytes, flags=OOB|NO_RELIABILITY] TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.UDP: received [dst: , src: e96f542b7216-17605 (2 headers), size=0 bytes, flags=OOB|NO_RELIABILITY], headers are STABLE: [STABILITY]: digest is e96f542b7216-17605: [0 (5)], UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.pbcast.STABLE: e96f542b7216-17605: received stability msg from e96f542b7216-17605: e96f542b7216-17605: [0] TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.pbcast.STABLE: e96f542b7216-17605: resetting digest from NAKACK: e96f542b7216-17605: [5] TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.pbcast.NAKACK2: e96f542b7216-17605: received stable digest e96f542b7216-17605: [0 (5)] TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.pbcast.NAKACK2: e96f542b7216-17605: deleting msgs <= 0 from e96f542b7216-17605 TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.UDP: received [dst: , src: e96f542b7216-17605 (2 headers), size=0 bytes, flags=OOB|NO_RELIABILITY], headers are STABLE: [STABILITY]: digest is e96f542b7216-17605: [0 (5)], UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.UDP: received [dst: , src: 9b033d68-c4bb-c2cb-4dc9-c9065e70c606 (2 headers), size=0 bytes, flags=OOB], headers are PING: [PING: type=GET_MBRS_REQ, cluster=ExampleFederation, arg=9b033d68-c4bb-c2cb-4dc9-c9065e70c606, view_id=, is_server=false, is_coord=false, logical_name=a627c2c18b9a-60763, physical_addrs=10.10.0.1:48841], UDP: [channel_name=ExampleFederation] TRACE [Incoming] org.jgroups.protocols.PING: e96f542b7216-17605: received GET_MBRS_REQ from a627c2c18b9a-60763, sending staggered response [PING: type=GET_MBRS_RSP, arg=e96f542b7216-17605, view_id=[e96f542b7216-17605|0], is_server=true, is_coord=true, logical_name=e96f542b7216-17605, physical_addrs=10.10.0.2:56777] TRACE [Incoming] org.jgroups.protocols.UDP: sending msg to a627c2c18b9a-60763, src=e96f542b7216-17605, headers are PING: [PING: type=GET_MBRS_RSP, arg=e96f542b7216-17605, view_id=[e96f542b7216-17605|0], is_server=true, is_coord=true, logical_name=e96f542b7216-17605, physical_addrs=10.10.0.2:56777], UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.UDP: received [dst: e96f542b7216-17605, src: a627c2c18b9a-60763 (3 headers), size=0 bytes, flags=OOB], headers are GMS: GmsHeader[JOIN_REQ_WITH_STATE_TRANSFER], UNICAST2: DATA, seqno=1, first, UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.UNICAST2: e96f542b7216-17605 <-- DATA(a627c2c18b9a-60763: #1, first) TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.UNICAST2: e96f542b7216-17605: created receiver window for a627c2c18b9a-60763 at seqno=#1 for conn-id=0 TRACE [ViewHandler,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.pbcast.GMS: e96f542b7216-17605: new members=[a627c2c18b9a-60763], suspected=[], leaving=[], new view: [e96f542b7216-17605|1] [e96f542b7216-17605, a627c2c18b9a-60763] TRACE [ViewHandler,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.pbcast.NAKACK2: e96f542b7216-17605 sending e96f542b7216-17605#6 TRACE [ViewHandler,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.UDP: sending msg to null, src=e96f542b7216-17605, headers are FLUSH: FLUSH[type=START_FLUSH,viewId=0,members=[e96f542b7216-17605]], NAKACK2: [MSG, seqno=6], UDP: [channel_name=ExampleFederation] TRACE [ViewHandler,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.UDP: looping back message [dst: , src: e96f542b7216-17605 (3 headers), size=0 bytes] TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.UDP: received [dst: , src: e96f542b7216-17605 (3 headers), size=0 bytes], headers are FLUSH: FLUSH[type=START_FLUSH,viewId=0,members=[e96f542b7216-17605]], NAKACK2: [MSG, seqno=6], UDP: [channel_name=ExampleFederation] DEBUG [ViewHandler,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.pbcast.FLUSH: e96f542b7216-17605: flush coordinator is starting FLUSH with participants [e96f542b7216-17605] TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.pbcast.NAKACK2: e96f542b7216-17605: received e96f542b7216-17605#6 TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.pbcast.NAKACK2: e96f542b7216-17605: delivering e96f542b7216-17605#6 TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.UNICAST2: e96f542b7216-17605 --> DATA(e96f542b7216-17605: #3, conn_id=0) TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.UDP: sending msg to e96f542b7216-17605, src=e96f542b7216-17605, headers are FLUSH: FLUSH[type=FLUSH_COMPLETED,viewId=0], UNICAST2: DATA, seqno=3, UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.UDP: looping back message [dst: e96f542b7216-17605, src: e96f542b7216-17605 (3 headers), size=0 bytes] DEBUG [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.pbcast.FLUSH: e96f542b7216-17605: received START_FLUSH, responded with FLUSH_COMPLETED to e96f542b7216-17605 TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.UDP: received [dst: e96f542b7216-17605, src: e96f542b7216-17605 (3 headers), size=0 bytes], headers are FLUSH: FLUSH[type=FLUSH_COMPLETED,viewId=0], UNICAST2: DATA, seqno=3, UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.UNICAST2: e96f542b7216-17605 <-- DATA(e96f542b7216-17605: #3) DEBUG [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.pbcast.FLUSH: e96f542b7216-17605: FLUSH_COMPLETED from e96f542b7216-17605, completed true, flushMembers [e96f542b7216-17605], flushCompleted [e96f542b7216-17605] DEBUG [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.pbcast.FLUSH: e96f542b7216-17605: all FLUSH_COMPLETED received TRACE [ViewHandler,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.pbcast.GMS: e96f542b7216-17605: successful GMS flush by coordinator DEBUG [ViewHandler,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.pbcast.STABLE: suspending message garbage collection DEBUG [ViewHandler,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.pbcast.STABLE: resume task started, max_suspend_time=33000 TRACE [ViewHandler,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.pbcast.GMS: e96f542b7216-17605: mcasting view [e96f542b7216-17605|1] [e96f542b7216-17605, a627c2c18b9a-60763](2 mbrs)

TRACE [ViewHandler,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.pbcast.NAKACK2: e96f542b7216-17605 sending e96f542b7216-17605#7 TRACE [ViewHandler,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.UDP: sending msg to null, src=e96f542b7216-17605, headers are GMS: GmsHeader[VIEW]: view=[e96f542b7216-17605|1] [e96f542b7216-17605, a627c2c18b9a-60763], NAKACK2: [MSG, seqno=7], UDP: [channel_name=ExampleFederation] TRACE [ViewHandler,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.UDP: looping back message [dst: , src: e96f542b7216-17605 (3 headers), size=0 bytes] TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.UDP: received [dst: , src: e96f542b7216-17605 (3 headers), size=0 bytes], headers are GMS: GmsHeader[VIEW]: view=[e96f542b7216-17605|1] [e96f542b7216-17605, a627c2c18b9a-60763], NAKACK2: [MSG, seqno=7], UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.pbcast.NAKACK2: e96f542b7216-17605: received e96f542b7216-17605#7 TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.pbcast.NAKACK2: e96f542b7216-17605: delivering e96f542b7216-17605#7 DEBUG [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.pbcast.NAKACK2: [e96f542b7216-17605 setDigest()] existing digest: e96f542b7216-17605: [7 (7)] new digest: e96f542b7216-17605: [6 (6)], a627c2c18b9a-60763: [0 (0)] resulting digest: a627c2c18b9a-60763: [0 (0)], e96f542b7216-17605: [7 (7)] DEBUG [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.pbcast.GMS: e96f542b7216-17605: installing view [e96f542b7216-17605|1] [e96f542b7216-17605, a627c2c18b9a-60763] TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.pbcast.STABLE: e96f542b7216-17605: resetting digest from NAKACK: a627c2c18b9a-60763: [0], e96f542b7216-17605: [7] DEBUG [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.pbcast.STABLE: [ergonomics] setting max_bytes to 8MB (2 members) TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.UFC: new membership: [e96f542b7216-17605, a627c2c18b9a-60763] TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.MFC: new membership: [e96f542b7216-17605, a627c2c18b9a-60763] DEBUG [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.pbcast.FLUSH: e96f542b7216-17605: installing view [e96f542b7216-17605|1] [e96f542b7216-17605, a627c2c18b9a-60763] TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.UNICAST2: e96f542b7216-17605 --> DATA(e96f542b7216-17605: #4, conn_id=0) TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.UDP: sending msg to e96f542b7216-17605, src=e96f542b7216-17605, headers are GMS: GmsHeader[VIEW_ACK]: view=null, UNICAST2: DATA, seqno=4, UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.UDP: looping back message [dst: e96f542b7216-17605, src: e96f542b7216-17605 (3 headers), size=0 bytes, flags=OOB] TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.UDP: received [dst: e96f542b7216-17605, src: e96f542b7216-17605 (3 headers), size=0 bytes, flags=OOB], headers are GMS: GmsHeader[VIEW_ACK]: view=null, UNICAST2: DATA, seqno=4, UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.UNICAST2: e96f542b7216-17605 <-- DATA(e96f542b7216-17605: #4) TRACE [ViewHandler,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.pbcast.GMS: e96f542b7216-17605: received all 1 ACKs from members for view [e96f542b7216-17605|1] TRACE [ViewHandler,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.UNICAST2: e96f542b7216-17605: created connection to a627c2c18b9a-60763 (conn_id=1) TRACE [ViewHandler,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.UNICAST2: e96f542b7216-17605 --> DATA(a627c2c18b9a-60763: #1, conn_id=1, first) TRACE [ViewHandler,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.UDP: sending msg to a627c2c18b9a-60763, src=e96f542b7216-17605, headers are GMS: GmsHeader[JOIN_RSP]: join_rsp=view: [e96f542b7216-17605|1] [e96f542b7216-17605, a627c2c18b9a-60763], digest: e96f542b7216-17605: [6 (6)], a627c2c18b9a-60763: [0 (0)], UNICAST2: DATA, seqno=1, conn_id=1, first, UDP: [channel_name=ExampleFederation] TRACE [TransferQueueBundler,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.TP$TransferQueueBundler: sending 2 msgs (247 bytes (0.39% of max_bundle_size) to 1 destination(s) TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.UDP: received [dst: , src: e96f542b7216-17605 (3 headers), size=0 bytes], headers are FLUSH: FLUSH[type=START_FLUSH,viewId=0,members=[e96f542b7216-17605]], NAKACK2: [MSG, seqno=6], UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.UDP: received [dst: , src: e96f542b7216-17605 (3 headers), size=0 bytes], headers are GMS: GmsHeader[VIEW]: view=[e96f542b7216-17605|1] [e96f542b7216-17605, a627c2c18b9a-60763], NAKACK2: [MSG, seqno=7], UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.UDP: received [dst: e96f542b7216-17605, src: a627c2c18b9a-60763 (3 headers), size=0 bytes, flags=OOB], headers are GMS: GmsHeader[VIEW_ACK]: view=null, UNICAST2: DATA, seqno=2, UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.UNICAST2: e96f542b7216-17605 <-- DATA(a627c2c18b9a-60763: #2) TRACE [ViewHandler,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.pbcast.GMS: e96f542b7216-17605: received all ACKs (1) from joiners for view [e96f542b7216-17605|1] TRACE [ViewHandler,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.pbcast.STABLE: e96f542b7216-17605: resetting digest from NAKACK: a627c2c18b9a-60763: [0], e96f542b7216-17605: [7] DEBUG [ViewHandler,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.pbcast.STABLE: resuming message garbage collection TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.UDP: received [dst: e96f542b7216-17605, src: a627c2c18b9a-60763 (3 headers), size=0 bytes, flags=OOB], headers are GMS: GmsHeader[VIEW_ACK]: view=null, UNICAST2: DATA, seqno=3, UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.UNICAST2: e96f542b7216-17605 <-- DATA(a627c2c18b9a-60763: #3) TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.UDP: received [dst: e96f542b7216-17605, src: a627c2c18b9a-60763 (3 headers), size=0 bytes], headers are STATE_TRANSFER: type=STATE_REQ, UNICAST2: DATA, seqno=4, UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.UNICAST2: e96f542b7216-17605 <-- DATA(a627c2c18b9a-60763: #4) DEBUG [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.pbcast.STATE_TRANSFER: e96f542b7216-17605: received state request from a627c2c18b9a-60763 TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.pbcast.STATE_TRANSFER: e96f542b7216-17605: sending state to a627c2c18b9a-60763 (size=25.63KB) TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.UFC: bypassing flow control because of synchronous response Thread[Incoming,ExampleFederation,e96f542b7216-17605,5,JGroups channel] TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.UNICAST2: e96f542b7216-17605 --> DATA(a627c2c18b9a-60763: #2, conn_id=1) TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.UDP: sending msg to a627c2c18b9a-60763, src=e96f542b7216-17605, headers are STATE_TRANSFER: type=STATE_RSP, UNICAST2: DATA, seqno=2, conn_id=1, UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.UDP: received [dst: , src: a627c2c18b9a-60763 (3 headers), size=0 bytes], headers are FLUSH: FLUSH[type=STOP_FLUSH,viewId=1], NAKACK2: [MSG, seqno=1], UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.pbcast.NAKACK2: e96f542b7216-17605: received a627c2c18b9a-60763#1 TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.pbcast.NAKACK2: e96f542b7216-17605: delivering a627c2c18b9a-60763#1 DEBUG [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.pbcast.FLUSH: e96f542b7216-17605: received STOP_FLUSH, unblocking FLUSH.down() and sending UNBLOCK up TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.UDP: received [dst: , src: a627c2c18b9a-60763 (4 headers), size=1 bytes, flags=OOB|DONT_BUNDLE|NO_FC|RSVP], headers are 7777: JoinFederation, RSVP: REQ(0), NAKACK2: [MSG, seqno=2], UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.pbcast.NAKACK2: e96f542b7216-17605: received a627c2c18b9a-60763#2 TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.pbcast.NAKACK2: e96f542b7216-17605: delivering a627c2c18b9a-60763#2 TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.RSVP: e96f542b7216-17605: REQ <-- a627c2c18b9a-60763 TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.RSVP: e96f542b7216-17605: RSP --> a627c2c18b9a-60763 TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.UNICAST2: e96f542b7216-17605 --> DATA(a627c2c18b9a-60763: #3, conn_id=1) TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.UDP: sending msg to a627c2c18b9a-60763, src=e96f542b7216-17605, headers are RSVP: RSP(0), UNICAST2: DATA, seqno=3, conn_id=1, UDP: [channel_name=ExampleFederation] TRACE [Incoming] org.jgroups.protocols.UDP: sending msg to null, src=e96f542b7216-17605, headers are FD_ALL: heartbeat, UDP: [channel_name=ExampleFederation] TRACE [Incoming] org.jgroups.protocols.UDP: looping back message [dst: , src: e96f542b7216-17605 (2 headers), size=0 bytes, flags=OOB] TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.UDP: received [dst: , src: e96f542b7216-17605 (2 headers), size=0 bytes, flags=OOB], headers are FD_ALL: heartbeat, UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.UDP: received [dst: , src: e96f542b7216-17605 (2 headers), size=0 bytes, flags=OOB], headers are FD_ALL: heartbeat, UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.UDP: received [dst: , src: a627c2c18b9a-60763 (2 headers), size=0 bytes, flags=OOB], headers are FD_ALL: heartbeat, UDP: [channel_name=ExampleFederation] TRACE [Incoming] org.jgroups.protocols.UDP: sending msg to null, src=e96f542b7216-17605, headers are FD_ALL: heartbeat, UDP: [channel_name=ExampleFederation] TRACE [Incoming] org.jgroups.protocols.UDP: looping back message [dst: , src: e96f542b7216-17605 (2 headers), size=0 bytes, flags=OOB] TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.UDP: received [dst: , src: e96f542b7216-17605 (2 headers), size=0 bytes, flags=OOB], headers are FD_ALL: heartbeat, UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.UDP: received [dst: , src: e96f542b7216-17605 (2 headers), size=0 bytes, flags=OOB], headers are FD_ALL: heartbeat, UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.UDP: received [dst: , src: a627c2c18b9a-60763 (2 headers), size=0 bytes, flags=OOB], headers are FD_ALL: heartbeat, UDP: [channel_name=ExampleFederation] TRACE [Incoming] org.jgroups.protocols.UDP: sending msg to null, src=e96f542b7216-17605, headers are FD_ALL: heartbeat, UDP: [channel_name=ExampleFederation] TRACE [Incoming] org.jgroups.protocols.UDP: looping back message [dst: , src: e96f542b7216-17605 (2 headers), size=0 bytes, flags=OOB] TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.UDP: received [dst: , src: e96f542b7216-17605 (2 headers), size=0 bytes, flags=OOB], headers are FD_ALL: heartbeat, UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.UDP: received [dst: , src: e96f542b7216-17605 (2 headers), size=0 bytes, flags=OOB], headers are FD_ALL: heartbeat, UDP: [channel_name=ExampleFederation] TRACE [Incoming] org.jgroups.protocols.UDP: sending msg to null, src=e96f542b7216-17605, headers are FD_ALL: heartbeat, UDP: [channel_name=ExampleFederation] TRACE [Incoming] org.jgroups.protocols.UDP: looping back message [dst: , src: e96f542b7216-17605 (2 headers), size=0 bytes, flags=OOB] TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.UDP: received [dst: , src: e96f542b7216-17605 (2 headers), size=0 bytes, flags=OOB], headers are FD_ALL: heartbeat, UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.UDP: received [dst: , src: e96f542b7216-17605 (2 headers), size=0 bytes, flags=OOB], headers are FD_ALL: heartbeat, UDP: [channel_name=ExampleFederation] TRACE [Incoming] org.jgroups.protocols.pbcast.STABLE: e96f542b7216-17605: setting latest_local_digest from NAKACK: a627c2c18b9a-60763: [2], e96f542b7216-17605: [7] TRACE [Incoming] org.jgroups.protocols.pbcast.STABLE: e96f542b7216-17605: sending stable msg a627c2c18b9a-60763: [2], e96f542b7216-17605: [7] TRACE [Incoming] org.jgroups.protocols.UDP: sending msg to null, src=e96f542b7216-17605, headers are STABLE: [STABLE_GOSSIP]: digest is a627c2c18b9a-60763: [2 (2)], e96f542b7216-17605: [7 (7)], UDP: [channel_name=ExampleFederation] TRACE [Incoming] org.jgroups.protocols.UDP: looping back message [dst: , src: e96f542b7216-17605 (2 headers), size=0 bytes, flags=OOB|NO_RELIABILITY] TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.UDP: received [dst: , src: e96f542b7216-17605 (2 headers), size=0 bytes, flags=OOB|NO_RELIABILITY], headers are STABLE: [STABLE_GOSSIP]: digest is a627c2c18b9a-60763: [2 (2)], e96f542b7216-17605: [7 (7)], UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.pbcast.STABLE: e96f542b7216-17605: handling digest from e96f542b7216-17605 (0 votes): mine: a627c2c18b9a-60763: [0], e96f542b7216-17605: [7] other: a627c2c18b9a-60763: [2], e96f542b7216-17605: [7] result: a627c2c18b9a-60763: [0], e96f542b7216-17605: [7]

TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.UDP: received [dst: , src: e96f542b7216-17605 (2 headers), size=0 bytes, flags=OOB|NO_RELIABILITY], headers are STABLE: [STABLE_GOSSIP]: digest is a627c2c18b9a-60763: [2 (2)], e96f542b7216-17605: [7 (7)], UDP: [channel_name=ExampleFederation] TRACE [Incoming] org.jgroups.protocols.UDP: sending msg to null, src=e96f542b7216-17605, headers are FD_ALL: heartbeat, UDP: [channel_name=ExampleFederation] TRACE [Incoming] org.jgroups.protocols.UDP: looping back message [dst: , src: e96f542b7216-17605 (2 headers), size=0 bytes, flags=OOB] TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.UDP: received [dst: , src: e96f542b7216-17605 (2 headers), size=0 bytes, flags=OOB], headers are FD_ALL: heartbeat, UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.UDP: received [dst: , src: e96f542b7216-17605 (2 headers), size=0 bytes, flags=OOB], headers are FD_ALL: heartbeat, UDP: [channel_name=ExampleFederation] DEBUG [Incoming] org.jgroups.protocols.FD_ALL: haven't received a heartbeat from a627c2c18b9a-60763 for 9977 ms, adding it to suspect list DEBUG [Incoming] org.jgroups.protocols.FD_ALL: suspecting [a627c2c18b9a-60763] TRACE [Incoming] org.jgroups.protocols.VERIFY_SUSPECT: verifying that a627c2c18b9a-60763 is dead TRACE [Incoming] org.jgroups.protocols.UDP: sending msg to a627c2c18b9a-60763, src=e96f542b7216-17605, headers are VERIFY_SUSPECT: [VERIFY_SUSPECT: ARE_YOU_DEAD], UDP: [channel_name=ExampleFederation] TRACE [VERIFY_SUSPECT.TimerThread,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.VERIFY_SUSPECT: a627c2c18b9a-60763 is dead (passing up SUSPECT event) DEBUG [VERIFY_SUSPECT.TimerThread,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.pbcast.FLUSH: e96f542b7216-17605: suspect is a627c2c18b9a-60763, completed false, flushOkSet {}, flushMembers [] DEBUG [VERIFY_SUSPECT.TimerThread,ExampleFederation,e96f542b7216-17605] org.jgroups.blocks.RequestCorrelator: suspect=a627c2c18b9a-60763 WARN [VERIFY_SUSPECT.TimerThread,ExampleFederation,e96f542b7216-17605] portico.lrc.jgroups: Detected that federate [1] may have crashed, investigating... TRACE [Incoming] org.jgroups.protocols.UDP: sending msg to null, src=e96f542b7216-17605, headers are FD_ALL: heartbeat, UDP: [channel_name=ExampleFederation] TRACE [Incoming] org.jgroups.protocols.UDP: looping back message [dst: , src: e96f542b7216-17605 (2 headers), size=0 bytes, flags=OOB] TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.UDP: received [dst: , src: e96f542b7216-17605 (2 headers), size=0 bytes, flags=OOB], headers are FD_ALL: heartbeat, UDP: [channel_name=ExampleFederation] DEBUG [Incoming] org.jgroups.protocols.FD_ALL: haven't received a heartbeat from a627c2c18b9a-60763 for 12015 ms, adding it to suspect list DEBUG [Incoming] org.jgroups.protocols.FD_ALL: suspecting [a627c2c18b9a-60763] TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.UDP: received [dst: , src: e96f542b7216-17605 (2 headers), size=0 bytes, flags=OOB], headers are FD_ALL: heartbeat, UDP: [channel_name=ExampleFederation] TRACE [Incoming] org.jgroups.protocols.VERIFY_SUSPECT: verifying that a627c2c18b9a-60763 is dead TRACE [Incoming] org.jgroups.protocols.UDP: sending msg to a627c2c18b9a-60763, src=e96f542b7216-17605, headers are VERIFY_SUSPECT: [VERIFY_SUSPECT: ARE_YOU_DEAD], UDP: [channel_name=ExampleFederation] TRACE [ViewHandler,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.pbcast.GMS: e96f542b7216-17605: new members=[], suspected=[a627c2c18b9a-60763], leaving=[], new view: [e96f542b7216-17605|2] [e96f542b7216-17605] TRACE [ViewHandler,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.pbcast.NAKACK2: e96f542b7216-17605 sending e96f542b7216-17605#8 TRACE [ViewHandler,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.UDP: sending msg to null, src=e96f542b7216-17605, headers are FLUSH: FLUSH[type=START_FLUSH,viewId=1,members=[e96f542b7216-17605]], NAKACK2: [MSG, seqno=8], UDP: [channel_name=ExampleFederation] TRACE [ViewHandler,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.UDP: looping back message [dst: , src: e96f542b7216-17605 (3 headers), size=0 bytes] DEBUG [ViewHandler,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.pbcast.FLUSH: e96f542b7216-17605: flush coordinator is starting FLUSH with participants [e96f542b7216-17605] TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.UDP: received [dst: , src: e96f542b7216-17605 (3 headers), size=0 bytes], headers are FLUSH: FLUSH[type=START_FLUSH,viewId=1,members=[e96f542b7216-17605]], NAKACK2: [MSG, seqno=8], UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.pbcast.NAKACK2: e96f542b7216-17605: received e96f542b7216-17605#8 TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.pbcast.NAKACK2: e96f542b7216-17605: delivering e96f542b7216-17605#8 TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.UNICAST2: e96f542b7216-17605 --> DATA(e96f542b7216-17605: #5, conn_id=0) TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.UDP: sending msg to e96f542b7216-17605, src=e96f542b7216-17605, headers are FLUSH: FLUSH[type=FLUSH_COMPLETED,viewId=1], UNICAST2: DATA, seqno=5, UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.UDP: looping back message [dst: e96f542b7216-17605, src: e96f542b7216-17605 (3 headers), size=0 bytes] DEBUG [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.pbcast.FLUSH: e96f542b7216-17605: received START_FLUSH, responded with FLUSH_COMPLETED to e96f542b7216-17605 TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.UDP: received [dst: e96f542b7216-17605, src: e96f542b7216-17605 (3 headers), size=0 bytes], headers are FLUSH: FLUSH[type=FLUSH_COMPLETED,viewId=1], UNICAST2: DATA, seqno=5, UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.UNICAST2: e96f542b7216-17605 <-- DATA(e96f542b7216-17605: #5) DEBUG [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.pbcast.FLUSH: e96f542b7216-17605: FLUSH_COMPLETED from e96f542b7216-17605, completed true, flushMembers [e96f542b7216-17605], flushCompleted [e96f542b7216-17605] DEBUG [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.pbcast.FLUSH: e96f542b7216-17605: all FLUSH_COMPLETED received TRACE [ViewHandler,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.pbcast.GMS: e96f542b7216-17605: successful GMS flush by coordinator TRACE [ViewHandler,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.pbcast.GMS: e96f542b7216-17605: mcasting view [e96f542b7216-17605|2] [e96f542b7216-17605](1 mbrs)

TRACE [ViewHandler,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.pbcast.NAKACK2: e96f542b7216-17605 sending e96f542b7216-17605#9 TRACE [ViewHandler,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.UDP: sending msg to null, src=e96f542b7216-17605, headers are GMS: GmsHeader[VIEW]: view=[e96f542b7216-17605|2] [e96f542b7216-17605], NAKACK2: [MSG, seqno=9], UDP: [channel_name=ExampleFederation] TRACE [ViewHandler,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.UDP: looping back message [dst: , src: e96f542b7216-17605 (3 headers), size=0 bytes] TRACE [TransferQueueBundler,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.TP$TransferQueueBundler: sending 1 msgs (85 bytes (0.13% of max_bundle_size) to 1 destination(s) TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.UDP: received [dst: , src: e96f542b7216-17605 (3 headers), size=0 bytes], headers are GMS: GmsHeader[VIEW]: view=[e96f542b7216-17605|2] [e96f542b7216-17605], NAKACK2: [MSG, seqno=9], UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.pbcast.NAKACK2: e96f542b7216-17605: received e96f542b7216-17605#9 TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.pbcast.NAKACK2: e96f542b7216-17605: delivering e96f542b7216-17605#9 DEBUG [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.pbcast.GMS: e96f542b7216-17605: installing view [e96f542b7216-17605|2] [e96f542b7216-17605] TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.UNICAST2: e96f542b7216-17605: removing non members [a627c2c18b9a-60763] TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.UNICAST2: e96f542b7216-17605 --> STABLE(a627c2c18b9a-60763: 4-4, conn_id=0) TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.UDP: sending msg to a627c2c18b9a-60763, src=e96f542b7216-17605, headers are UNICAST2: STABLE, seqno=4, UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.UDP: received [dst: , src: e96f542b7216-17605 (3 headers), size=0 bytes], headers are FLUSH: FLUSH[type=START_FLUSH,viewId=1,members=[e96f542b7216-17605]], NAKACK2: [MSG, seqno=8], UDP: [channel_name=ExampleFederation] DEBUG [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.pbcast.NAKACK2: removed a627c2c18b9a-60763 from xmit_table (not member anymore) TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.pbcast.STABLE: e96f542b7216-17605: resetting digest from NAKACK: e96f542b7216-17605: [9] DEBUG [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.pbcast.STABLE: [ergonomics] setting max_bytes to 4MB (1 members) TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.UFC: new membership: [e96f542b7216-17605] TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.MFC: new membership: [e96f542b7216-17605] TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.FRAG2: [VIEW_CHANGE] removed a627c2c18b9a-60763 from fragmentation table TRACE [TransferQueueBundler,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.TP$TransferQueueBundler: sending 1 msgs (105 bytes (0.17% of max_bundle_size) to 1 destination(s) TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.UDP: received [dst: , src: e96f542b7216-17605 (3 headers), size=0 bytes], headers are GMS: GmsHeader[VIEW]: view=[e96f542b7216-17605|2] [e96f542b7216-17605], NAKACK2: [MSG, seqno=9], UDP: [channel_name=ExampleFederation] DEBUG [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.pbcast.FLUSH: e96f542b7216-17605: installing view [e96f542b7216-17605|2] [e96f542b7216-17605] TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.UNICAST2: e96f542b7216-17605 --> DATA(e96f542b7216-17605: #6, conn_id=0) TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.UDP: sending msg to e96f542b7216-17605, src=e96f542b7216-17605, headers are GMS: GmsHeader[VIEW_ACK]: view=null, UNICAST2: DATA, seqno=6, UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.UDP: looping back message [dst: e96f542b7216-17605, src: e96f542b7216-17605 (3 headers), size=0 bytes, flags=OOB] TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.UDP: received [dst: e96f542b7216-17605, src: e96f542b7216-17605 (3 headers), size=0 bytes, flags=OOB], headers are GMS: GmsHeader[VIEW_ACK]: view=null, UNICAST2: DATA, seqno=6, UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.UNICAST2: e96f542b7216-17605 <-- DATA(e96f542b7216-17605: #6) TRACE [ViewHandler,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.pbcast.GMS: e96f542b7216-17605: received all 1 ACKs from members for view [e96f542b7216-17605|2] TRACE [ViewHandler,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.pbcast.GMS: e96f542b7216-17605: sending RESUME event DEBUG [ViewHandler,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.pbcast.FLUSH: e96f542b7216-17605: received RESUME, sending STOP_FLUSH to all TRACE [ViewHandler,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.pbcast.NAKACK2: e96f542b7216-17605 sending e96f542b7216-17605#10 TRACE [ViewHandler,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.UDP: sending msg to null, src=e96f542b7216-17605, headers are FLUSH: FLUSH[type=STOP_FLUSH,viewId=2], NAKACK2: [MSG, seqno=10], UDP: [channel_name=ExampleFederation] TRACE [ViewHandler,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.UDP: looping back message [dst: , src: e96f542b7216-17605 (3 headers), size=0 bytes] TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.UDP: received [dst: , src: e96f542b7216-17605 (3 headers), size=0 bytes], headers are FLUSH: FLUSH[type=STOP_FLUSH,viewId=2], NAKACK2: [MSG, seqno=10], UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.pbcast.NAKACK2: e96f542b7216-17605: received e96f542b7216-17605#10 TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.pbcast.NAKACK2: e96f542b7216-17605: delivering e96f542b7216-17605#10 DEBUG [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.pbcast.FLUSH: e96f542b7216-17605: received STOP_FLUSH, unblocking FLUSH.down() and sending UNBLOCK up TRACE [TransferQueueBundler,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.TP$TransferQueueBundler: sending 1 msgs (68 bytes (0.11% of max_bundle_size) to 1 destination(s) TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.UDP: received [dst: , src: e96f542b7216-17605 (3 headers), size=0 bytes], headers are FLUSH: FLUSH[type=STOP_FLUSH,viewId=2], NAKACK2: [MSG, seqno=10], UDP: [channel_name=ExampleFederation] TRACE [Incoming] org.jgroups.protocols.UNICAST2: e96f542b7216-17605 --> STABLE(e96f542b7216-17605: 6-6, conn_id=0) TRACE [Incoming] org.jgroups.protocols.UDP: sending msg to e96f542b7216-17605, src=e96f542b7216-17605, headers are UNICAST2: STABLE, seqno=6, UDP: [channel_name=ExampleFederation] TRACE [Incoming] org.jgroups.protocols.UDP: looping back message [dst: e96f542b7216-17605, src: e96f542b7216-17605 (2 headers), size=0 bytes, flags=OOB] TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.UDP: received [dst: e96f542b7216-17605, src: e96f542b7216-17605 (2 headers), size=0 bytes, flags=OOB], headers are UNICAST2: STABLE, seqno=6, UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,e96f542b7216-17605] org.jgroups.protocols.UNICAST2: e96f542b7216-17605 <-- STABLE(e96f542b7216-17605: 6-6, conn_id=0)

bergtwvd commented 9 years ago

LOGFILE FOR CONTAINER A:

WARN [main] portico.lrc: MOM support is currently unsupported in IEEE-1516e federations. DEBUG [main] org.jgroups.conf.ClassConfigurator: Using jg-magic-map.xml as magic number file and jg-protocol-ids.xml for protocol IDs DEBUG [main] org.jgroups.stack.Configurator: set property UDP.bind_addr to default value /10.10.0.1 DEBUG [main] org.jgroups.stack.Configurator: set property UDP.diagnostics_addr to default value /224.0.75.75 DEBUG [main] org.jgroups.protocols.FRAG2: received CONFIG event: {bind_addr=/10.10.0.1} TRACE [main] org.jgroups.blocks.MessageDispatcher$ProtocolAdapter: setting local_addr (null) to a627c2c18b9a-60763 DEBUG [main] org.jgroups.protocols.FRAG2: received CONFIG event: {flush_supported=true} TRACE [main] org.jgroups.protocols.pbcast.STABLE: stable task started TRACE [main] org.jgroups.protocols.UNICAST2: a627c2c18b9a-60763: stable task started DEBUG [main] org.jgroups.protocols.UDP: sockets will use interface 10.10.0.1 WARN [main] org.jgroups.protocols.UDP: [JGRP00014] the send buffer of socket DatagramSocket was set to 640KB, but the OS only allocated 212.99KB. This might lead to performance problems. Please set your max send buffer in the OS correctly (e.g. net.core.wmem_max on Linux) WARN [main] org.jgroups.protocols.UDP: [JGRP00014] the receive buffer of socket DatagramSocket was set to 8MB, but the OS only allocated 212.99KB. This might lead to performance problems. Please set your max receive buffer in the OS correctly (e.g. net.core.rmem_max on Linux) WARN [main] org.jgroups.protocols.UDP: [JGRP00014] the send buffer of socket MulticastSocket was set to 640KB, but the OS only allocated 212.99KB. This might lead to performance problems. Please set your max send buffer in the OS correctly (e.g. net.core.wmem_max on Linux) WARN [main] org.jgroups.protocols.UDP: [JGRP00014] the receive buffer of socket MulticastSocket was set to 8MB, but the OS only allocated 212.99KB. This might lead to performance problems. Please set your max receive buffer in the OS correctly (e.g. net.core.rmem_max on Linux) DEBUG [main] org.jgroups.protocols.UDP: socket information: , mcast_addr=239.255.20.16:20913, bind_addr=/10.10.0.1, ttl=8 sock: bound to 10.10.0.1:48841, receive buffer size=212992, send buffer size=212992 mcast_sock: bound to 10.10.0.1:20913, send buffer size=212992, receive buffer size=212992 TRACE [main] org.jgroups.protocols.UDP: joined /224.0.75.75:7500 on ethwe TRACE [main] org.jgroups.protocols.UDP: joined /224.0.75.75:7500 on eth0 TRACE [main] org.jgroups.protocols.UDP: joined /224.0.75.75:7500 on lo TRACE [main] org.jgroups.protocols.UDP: sending msg to null, src=a627c2c18b9a-60763, headers are PING: [PING: type=GET_MBRS_REQ, cluster=ExampleFederation, arg=a627c2c18b9a-60763, view_id=, is_server=false, is_coord=false, logical_name=a627c2c18b9a-60763, physical_addrs=10.10.0.1:48841], UDP: [channel_name=ExampleFederation] TRACE [main] org.jgroups.protocols.UDP: looping back message [dst: , src: a627c2c18b9a-60763 (2 headers), size=0 bytes, flags=OOB] TRACE [Incoming,ExampleFederation,a627c2c18b9a-60763] org.jgroups.protocols.UDP: received [dst: , src: a627c2c18b9a-60763 (2 headers), size=0 bytes, flags=OOB], headers are PING: [PING: type=GET_MBRS_REQ, cluster=ExampleFederation, arg=a627c2c18b9a-60763, view_id=, is_server=false, is_coord=false, logical_name=a627c2c18b9a-60763, physical_addrs=10.10.0.1:48841], UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,a627c2c18b9a-60763] org.jgroups.protocols.UDP: received [dst: , src: a627c2c18b9a-60763 (2 headers), size=0 bytes, flags=OOB], headers are PING: [PING: type=GET_MBRS_REQ, cluster=ExampleFederation, arg=a627c2c18b9a-60763, view_id=, is_server=false, is_coord=false, logical_name=a627c2c18b9a-60763, physical_addrs=10.10.0.1:48841], UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,a627c2c18b9a-60763] org.jgroups.protocols.UDP: received [dst: a627c2c18b9a-60763, src: faa656d7-c535-0659-1a1c-827c484ab81e (2 headers), size=0 bytes, flags=OOB], headers are PING: [PING: type=GET_MBRS_RSP, arg=faa656d7-c535-0659-1a1c-827c484ab81e, view_id=[faa656d7-c535-0659-1a1c-827c484ab81e|0], is_server=true, is_coord=true, logical_name=e96f542b7216-17605, physical_addrs=10.10.0.2:56777], UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,a627c2c18b9a-60763] org.jgroups.protocols.PING: a627c2c18b9a-60763: received GET_MBRS_RSP from e96f542b7216-17605: e96f542b7216-17605, view_id=[e96f542b7216-17605|0], is_server=true, is_coord=true, logical_name=e96f542b7216-17605, physical_addrs=10.10.0.2:56777 TRACE [main] org.jgroups.protocols.PING: discovery took 16 ms: responses: 2 total (1 servers (1 coord), 1 clients) TRACE [main] org.jgroups.protocols.pbcast.GMS: a627c2c18b9a-60763: initial_mbrs are e96f542b7216-17605 DEBUG [main] org.jgroups.protocols.pbcast.GMS: election results: {e96f542b7216-17605=1} DEBUG [main] org.jgroups.protocols.pbcast.GMS: sending JOIN(a627c2c18b9a-60763) to e96f542b7216-17605 TRACE [main] org.jgroups.protocols.UNICAST2: a627c2c18b9a-60763: created connection to e96f542b7216-17605 (conn_id=0) TRACE [Incoming] org.jgroups.protocols.PING: a627c2c18b9a-60763: received GET_MBRS_REQ from a627c2c18b9a-60763, sending staggered response [PING: type=GET_MBRS_RSP, arg=a627c2c18b9a-60763, view_id=, is_server=false, is_coord=false, logical_name=a627c2c18b9a-60763, physical_addrs=10.10.0.1:48841] TRACE [main] org.jgroups.protocols.UNICAST2: a627c2c18b9a-60763 --> DATA(e96f542b7216-17605: #1, conn_id=0, first) TRACE [main] org.jgroups.protocols.UDP: sending msg to e96f542b7216-17605, src=a627c2c18b9a-60763, headers are GMS: GmsHeader[JOIN_REQ_WITH_STATE_TRANSFER], UNICAST2: DATA, seqno=1, first, UDP: [channel_name=ExampleFederation] TRACE [Incoming] org.jgroups.protocols.UDP: sending msg to a627c2c18b9a-60763, src=a627c2c18b9a-60763, headers are PING: [PING: type=GET_MBRS_RSP, arg=a627c2c18b9a-60763, view_id=, is_server=false, is_coord=false, logical_name=a627c2c18b9a-60763, physical_addrs=10.10.0.1:48841], UDP: [channel_name=ExampleFederation] TRACE [Incoming] org.jgroups.protocols.UDP: looping back message [dst: a627c2c18b9a-60763, src: a627c2c18b9a-60763 (2 headers), size=0 bytes, flags=OOB] TRACE [Incoming,ExampleFederation,a627c2c18b9a-60763] org.jgroups.protocols.UDP: received [dst: a627c2c18b9a-60763, src: a627c2c18b9a-60763 (2 headers), size=0 bytes, flags=OOB], headers are PING: [PING: type=GET_MBRS_RSP, arg=a627c2c18b9a-60763, view_id=, is_server=false, is_coord=false, logical_name=a627c2c18b9a-60763, physical_addrs=10.10.0.1:48841], UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,a627c2c18b9a-60763] org.jgroups.protocols.PING: a627c2c18b9a-60763: received GET_MBRS_RSP from a627c2c18b9a-60763: a627c2c18b9a-60763, view_id=, is_server=false, is_coord=false, logical_name=a627c2c18b9a-60763, physical_addrs=10.10.0.1:48841 TRACE [Incoming,ExampleFederation,a627c2c18b9a-60763] org.jgroups.protocols.UDP: received [dst: a627c2c18b9a-60763, src: e96f542b7216-17605 (3 headers), size=0 bytes], headers are GMS: GmsHeader[JOIN_RSP]: join_rsp=view: [e96f542b7216-17605|1] [e96f542b7216-17605, a627c2c18b9a-60763], digest: e96f542b7216-17605: [6 (6)], a627c2c18b9a-60763: [0 (0)], UNICAST2: DATA, seqno=1, conn_id=1, first, UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,a627c2c18b9a-60763] org.jgroups.protocols.UNICAST2: a627c2c18b9a-60763 <-- DATA(e96f542b7216-17605: #1, conn_id=1, first) TRACE [Incoming,ExampleFederation,a627c2c18b9a-60763] org.jgroups.protocols.UNICAST2: a627c2c18b9a-60763: created receiver window for e96f542b7216-17605 at seqno=#1 for conn-id=1 TRACE [main] org.jgroups.protocols.pbcast.GMS: a627c2c18b9a-60763: JOIN-RSP=[e96f542b7216-17605|1] [e96f542b7216-17605, a627c2c18b9a-60763] [size=2]

DEBUG [main] org.jgroups.protocols.pbcast.NAKACK2: [a627c2c18b9a-60763 setDigest()] existing digest: [] new digest: e96f542b7216-17605: [6 (6)], a627c2c18b9a-60763: [0 (0)] resulting digest: a627c2c18b9a-60763: [0 (0)], e96f542b7216-17605: [6 (6)] DEBUG [main] org.jgroups.protocols.pbcast.GMS: a627c2c18b9a-60763: installing view [e96f542b7216-17605|1] [e96f542b7216-17605, a627c2c18b9a-60763] TRACE [main] org.jgroups.protocols.pbcast.STABLE: a627c2c18b9a-60763: resetting digest from NAKACK: a627c2c18b9a-60763: [0], e96f542b7216-17605: [6] TRACE [Incoming,ExampleFederation,a627c2c18b9a-60763] org.jgroups.protocols.UDP: received [dst: , src: e96f542b7216-17605 (3 headers), size=0 bytes], headers are FLUSH: FLUSH[type=START_FLUSH,viewId=0,members=[e96f542b7216-17605]], NAKACK2: [MSG, seqno=6], UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,a627c2c18b9a-60763] org.jgroups.protocols.UDP: received [dst: , src: e96f542b7216-17605 (3 headers), size=0 bytes], headers are GMS: GmsHeader[VIEW]: view=[e96f542b7216-17605|1] [e96f542b7216-17605, a627c2c18b9a-60763], NAKACK2: [MSG, seqno=7], UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,a627c2c18b9a-60763] org.jgroups.protocols.pbcast.NAKACK2: a627c2c18b9a-60763: received e96f542b7216-17605#7 TRACE [Incoming,ExampleFederation,a627c2c18b9a-60763] org.jgroups.protocols.pbcast.NAKACK2: a627c2c18b9a-60763: delivering e96f542b7216-17605#7 TRACE [Incoming,ExampleFederation,a627c2c18b9a-60763] org.jgroups.protocols.UNICAST2: a627c2c18b9a-60763 --> DATA(e96f542b7216-17605: #2, conn_id=0) TRACE [Incoming,ExampleFederation,a627c2c18b9a-60763] org.jgroups.protocols.UDP: sending msg to e96f542b7216-17605, src=a627c2c18b9a-60763, headers are GMS: GmsHeader[VIEW_ACK]: view=null, UNICAST2: DATA, seqno=2, UDP: [channel_name=ExampleFederation] DEBUG [main] org.jgroups.protocols.pbcast.STABLE: [ergonomics] setting max_bytes to 8MB (2 members) TRACE [main] org.jgroups.protocols.UFC: new membership: [e96f542b7216-17605, a627c2c18b9a-60763] TRACE [main] org.jgroups.protocols.MFC: new membership: [e96f542b7216-17605, a627c2c18b9a-60763] DEBUG [main] org.jgroups.protocols.pbcast.FLUSH: a627c2c18b9a-60763: installing view [e96f542b7216-17605|1] [e96f542b7216-17605, a627c2c18b9a-60763] TRACE [main] org.jgroups.protocols.UNICAST2: a627c2c18b9a-60763 --> DATA(e96f542b7216-17605: #3, conn_id=0) TRACE [main] org.jgroups.protocols.UDP: sending msg to e96f542b7216-17605, src=a627c2c18b9a-60763, headers are GMS: GmsHeader[VIEW_ACK]: view=null, UNICAST2: DATA, seqno=3, UDP: [channel_name=ExampleFederation] DEBUG [main] org.jgroups.protocols.pbcast.STATE_TRANSFER: a627c2c18b9a-60763: asking e96f542b7216-17605 for state TRACE [main] org.jgroups.protocols.UNICAST2: a627c2c18b9a-60763 --> DATA(e96f542b7216-17605: #4, conn_id=0) TRACE [main] org.jgroups.protocols.UDP: sending msg to e96f542b7216-17605, src=a627c2c18b9a-60763, headers are STATE_TRANSFER: type=STATE_REQ, UNICAST2: DATA, seqno=4, UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,a627c2c18b9a-60763] org.jgroups.protocols.UDP: received [dst: a627c2c18b9a-60763, src: e96f542b7216-17605 (3 headers), size=25634 bytes], headers are STATE_TRANSFER: type=STATE_RSP, UNICAST2: DATA, seqno=2, conn_id=1, UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,a627c2c18b9a-60763] org.jgroups.protocols.UNICAST2: a627c2c18b9a-60763 <-- DATA(e96f542b7216-17605: #2, conn_id=1) TRACE [Incoming,ExampleFederation,a627c2c18b9a-60763] org.jgroups.protocols.UFC: e96f542b7216-17605 used 25634 credits, 3974366 remaining DEBUG [Incoming,ExampleFederation,a627c2c18b9a-60763] org.jgroups.protocols.pbcast.STATE_TRANSFER: a627c2c18b9a-60763: received state, size=25.63KB, time=30 milliseconds DEBUG [main] org.jgroups.protocols.pbcast.FLUSH: a627c2c18b9a-60763: received RESUME, sending STOP_FLUSH to all TRACE [main] org.jgroups.protocols.pbcast.NAKACK2: a627c2c18b9a-60763 sending a627c2c18b9a-60763#1 TRACE [main] org.jgroups.protocols.UDP: sending msg to null, src=a627c2c18b9a-60763, headers are FLUSH: FLUSH[type=STOP_FLUSH,viewId=1], NAKACK2: [MSG, seqno=1], UDP: [channel_name=ExampleFederation] TRACE [main] org.jgroups.protocols.UDP: looping back message [dst: , src: a627c2c18b9a-60763 (3 headers), size=0 bytes] TRACE [Incoming,ExampleFederation,a627c2c18b9a-60763] org.jgroups.protocols.UDP: received [dst: , src: a627c2c18b9a-60763 (3 headers), size=0 bytes], headers are FLUSH: FLUSH[type=STOP_FLUSH,viewId=1], NAKACK2: [MSG, seqno=1], UDP: [channel_name=ExampleFederation] ERROR [main] portico.lrc.jgroups: FAILURE createFederation: already exists, name=ExampleFederation TRACE [Incoming,ExampleFederation,a627c2c18b9a-60763] org.jgroups.protocols.pbcast.NAKACK2: a627c2c18b9a-60763: received a627c2c18b9a-60763#1 TRACE [Incoming,ExampleFederation,a627c2c18b9a-60763] org.jgroups.protocols.pbcast.NAKACK2: a627c2c18b9a-60763: delivering a627c2c18b9a-60763#1 ERROR [main] portico.lrc: org.portico.lrc.compat.JFederationExecutionAlreadyExists: federation exists: ExampleFederation DEBUG [Incoming,ExampleFederation,a627c2c18b9a-60763] org.jgroups.protocols.pbcast.FLUSH: a627c2c18b9a-60763: received STOP_FLUSH, unblocking FLUSH.down() and sending UNBLOCK up TRACE [TransferQueueBundler,ExampleFederation,a627c2c18b9a-60763] org.jgroups.protocols.TP$TransferQueueBundler: sending 1 msgs (68 bytes (0.11% of max_bundle_size) to 1 destination(s) TRACE [Incoming,ExampleFederation,a627c2c18b9a-60763] org.jgroups.protocols.UDP: received [dst: , src: a627c2c18b9a-60763 (3 headers), size=0 bytes], headers are FLUSH: FLUSH[type=STOP_FLUSH,viewId=1], NAKACK2: [MSG, seqno=1], UDP: [channel_name=ExampleFederation] TRACE [main] org.jgroups.protocols.RSVP: a627c2c18b9a-60763: REQ --> null TRACE [main] org.jgroups.protocols.pbcast.NAKACK2: a627c2c18b9a-60763 sending a627c2c18b9a-60763#2 TRACE [main] org.jgroups.protocols.UDP: sending msg to null, src=a627c2c18b9a-60763, headers are 7777: JoinFederation, RSVP: REQ(0), NAKACK2: [MSG, seqno=2], UDP: [channel_name=ExampleFederation] TRACE [main] org.jgroups.protocols.UDP: looping back message [dst: , src: a627c2c18b9a-60763 (4 headers), size=1 bytes, flags=OOB|DONT_BUNDLE|NO_FC|RSVP] TRACE [Incoming,ExampleFederation,a627c2c18b9a-60763] org.jgroups.protocols.UDP: received [dst: , src: a627c2c18b9a-60763 (4 headers), size=1 bytes, flags=OOB|DONT_BUNDLE|NO_FC|RSVP], headers are 7777: JoinFederation, RSVP: REQ(0), NAKACK2: [MSG, seqno=2], UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,a627c2c18b9a-60763] org.jgroups.protocols.UDP: received [dst: , src: a627c2c18b9a-60763 (4 headers), size=1 bytes, flags=OOB|DONT_BUNDLE|NO_FC|RSVP], headers are 7777: JoinFederation, RSVP: REQ(0), NAKACK2: [MSG, seqno=2], UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,a627c2c18b9a-60763] org.jgroups.protocols.pbcast.NAKACK2: a627c2c18b9a-60763: received a627c2c18b9a-60763#2 TRACE [Incoming,ExampleFederation,a627c2c18b9a-60763] org.jgroups.protocols.pbcast.NAKACK2: a627c2c18b9a-60763: delivering a627c2c18b9a-60763#2 TRACE [Incoming,ExampleFederation,a627c2c18b9a-60763] org.jgroups.protocols.RSVP: a627c2c18b9a-60763: REQ <-- a627c2c18b9a-60763 TRACE [Incoming,ExampleFederation,a627c2c18b9a-60763] org.jgroups.protocols.RSVP: a627c2c18b9a-60763: RSP --> a627c2c18b9a-60763 TRACE [Incoming,ExampleFederation,a627c2c18b9a-60763] org.jgroups.protocols.UNICAST2: a627c2c18b9a-60763: created connection to a627c2c18b9a-60763 (conn_id=1) TRACE [Incoming,ExampleFederation,a627c2c18b9a-60763] org.jgroups.protocols.UNICAST2: a627c2c18b9a-60763 --> DATA(a627c2c18b9a-60763: #1, conn_id=1, first) TRACE [Incoming,ExampleFederation,a627c2c18b9a-60763] org.jgroups.protocols.UDP: sending msg to a627c2c18b9a-60763, src=a627c2c18b9a-60763, headers are RSVP: RSP(0), UNICAST2: DATA, seqno=1, conn_id=1, first, UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,a627c2c18b9a-60763] org.jgroups.protocols.UDP: looping back message [dst: a627c2c18b9a-60763, src: a627c2c18b9a-60763 (3 headers), size=0 bytes, flags=OOB|RSVP] TRACE [Incoming,ExampleFederation,a627c2c18b9a-60763] org.jgroups.protocols.UDP: received [dst: a627c2c18b9a-60763, src: a627c2c18b9a-60763 (3 headers), size=0 bytes, flags=OOB|RSVP], headers are RSVP: RSP(0), UNICAST2: DATA, seqno=1, conn_id=1, first, UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,a627c2c18b9a-60763] org.jgroups.protocols.UNICAST2: a627c2c18b9a-60763 <-- DATA(a627c2c18b9a-60763: #1, conn_id=1, first) TRACE [Incoming,ExampleFederation,a627c2c18b9a-60763] org.jgroups.protocols.UNICAST2: a627c2c18b9a-60763: created receiver window for a627c2c18b9a-60763 at seqno=#1 for conn-id=1 TRACE [Incoming,ExampleFederation,a627c2c18b9a-60763] org.jgroups.protocols.RSVP: a627c2c18b9a-60763: RSP <-- a627c2c18b9a-60763 TRACE [Incoming,ExampleFederation,a627c2c18b9a-60763] org.jgroups.protocols.UDP: received [dst: a627c2c18b9a-60763, src: e96f542b7216-17605 (3 headers), size=0 bytes, flags=OOB|RSVP], headers are RSVP: RSP(0), UNICAST2: DATA, seqno=3, conn_id=1, UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,a627c2c18b9a-60763] org.jgroups.protocols.UNICAST2: a627c2c18b9a-60763 <-- DATA(e96f542b7216-17605: #3, conn_id=1) TRACE [Incoming,ExampleFederation,a627c2c18b9a-60763] org.jgroups.protocols.RSVP: a627c2c18b9a-60763: RSP <-- e96f542b7216-17605 TRACE [main] org.jgroups.protocols.pbcast.NAKACK2: a627c2c18b9a-60763 sending a627c2c18b9a-60763#3 TRACE [main] org.jgroups.protocols.UDP: sending msg to null, src=a627c2c18b9a-60763, headers are NAKACK2: [MSG, seqno=3], UDP: [channel_name=ExampleFederation] TRACE [main] org.jgroups.protocols.UDP: looping back message [dst: , src: a627c2c18b9a-60763 (2 headers), size=2226 bytes] TRACE [Incoming,ExampleFederation,a627c2c18b9a-60763] org.jgroups.protocols.UDP: received [dst: , src: a627c2c18b9a-60763 (2 headers), size=2226 bytes], headers are NAKACK2: [MSG, seqno=3], UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,a627c2c18b9a-60763] org.jgroups.protocols.pbcast.NAKACK2: a627c2c18b9a-60763: received a627c2c18b9a-60763#3 TRACE [Incoming,ExampleFederation,a627c2c18b9a-60763] org.jgroups.protocols.pbcast.NAKACK2: a627c2c18b9a-60763: delivering a627c2c18b9a-60763#3 TRACE [Incoming,ExampleFederation,a627c2c18b9a-60763] org.jgroups.protocols.MFC: a627c2c18b9a-60763 used 2226 credits, 3997774 remaining TRACE [TransferQueueBundler,ExampleFederation,a627c2c18b9a-60763] org.jgroups.protocols.TP$TransferQueueBundler: sending 1 msgs (2282 bytes (3.62% of max_bundle_size) to 1 destination(s) TRACE [Incoming,ExampleFederation,a627c2c18b9a-60763] org.jgroups.protocols.UDP: received [dst: , src: a627c2c18b9a-60763 (2 headers), size=2226 bytes], headers are NAKACK2: [MSG, seqno=3], UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,a627c2c18b9a-60763] org.jgroups.protocols.UDP: received [dst: , src: e96f542b7216-17605 (2 headers), size=0 bytes, flags=OOB], headers are FD_ALL: heartbeat, UDP: [channel_name=ExampleFederation] TRACE [Incoming] org.jgroups.protocols.UDP: sending msg to null, src=a627c2c18b9a-60763, headers are FD_ALL: heartbeat, UDP: [channel_name=ExampleFederation] TRACE [Incoming] org.jgroups.protocols.UDP: looping back message [dst: , src: a627c2c18b9a-60763 (2 headers), size=0 bytes, flags=OOB] TRACE [Incoming,ExampleFederation,a627c2c18b9a-60763] org.jgroups.protocols.UDP: received [dst: , src: a627c2c18b9a-60763 (2 headers), size=0 bytes, flags=OOB], headers are FD_ALL: heartbeat, UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,a627c2c18b9a-60763] org.jgroups.protocols.UDP: received [dst: , src: a627c2c18b9a-60763 (2 headers), size=0 bytes, flags=OOB], headers are FD_ALL: heartbeat, UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,a627c2c18b9a-60763] org.jgroups.protocols.UDP: received [dst: , src: e96f542b7216-17605 (2 headers), size=0 bytes, flags=OOB], headers are FD_ALL: heartbeat, UDP: [channel_name=ExampleFederation] TRACE [Incoming] org.jgroups.protocols.UDP: sending msg to null, src=a627c2c18b9a-60763, headers are FD_ALL: heartbeat, UDP: [channel_name=ExampleFederation] TRACE [Incoming] org.jgroups.protocols.UDP: looping back message [dst: , src: a627c2c18b9a-60763 (2 headers), size=0 bytes, flags=OOB] TRACE [Incoming,ExampleFederation,a627c2c18b9a-60763] org.jgroups.protocols.UDP: received [dst: , src: a627c2c18b9a-60763 (2 headers), size=0 bytes, flags=OOB], headers are FD_ALL: heartbeat, UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,a627c2c18b9a-60763] org.jgroups.protocols.UDP: received [dst: , src: a627c2c18b9a-60763 (2 headers), size=0 bytes, flags=OOB], headers are FD_ALL: heartbeat, UDP: [channel_name=ExampleFederation] ERROR [main] portico.lrc: org.portico.lrc.compat.JRTIinternalError: Waited 5 seconds for RoleCall from federate [1], none received, connection error

bergtwvd commented 9 years ago

STACK TRACE OF CONTAINER A:

ERROR [main] portico.lrc: org.portico.lrc.compat.JRTIinternalError: Waited 5 seconds for RoleCall from federate [1], none received, connection error hla.rti1516e.exceptions.RTIinternalError: org.portico.lrc.compat.JRTIinternalError: Waited 5 seconds for RoleCall from federate [1], none received, connection error at org.portico.impl.hla1516e.Rti1516eAmbassador.joinFederationExecution(Rti1516eAmbassador.java:668) at ieee1516e.ExampleFederate.runFederate(Unknown Source) at ieee1516e.ExampleFederate.main(Unknown Source) Caused by: org.portico.lrc.compat.JRTIinternalError: Waited 5 seconds for RoleCall from federate [1], none received, connection error at org.portico.lrc.services.federation.handlers.outgoing.JoinFederationHandler.process(JoinFederationHandler.java:146) at org.portico.utils.messaging.MessageSink.process(MessageSink.java:187) at org.portico.impl.hla1516e.Impl1516eHelper.processMessage(Impl1516eHelper.java:99) at org.portico.impl.hla1516e.Rti1516eAmbassador.processMessage(Rti1516eAmbassador.java:5554) at org.portico.impl.hla1516e.Rti1516eAmbassador.joinFederationExecution(Rti1516eAmbassador.java:647) ... 2 more

bergtwvd commented 9 years ago

Dockerfile #1

FROM ubuntu:14.04 MAINTAINER Tom van den Berg tom.vandenberg@tno.nl

install supervisor

RUN apt-get update && apt-get install -y supervisor

install portico RTI

COPY ./port /usr/local/portico/ WORKDIR /usr/local/portico RUN tar -xvf portico-2.0.1-linux64.tar RUN rm portico-2.0.1-linux64.tar

ENV RTI_HOME=/usr/local/portico/portico-2.0.1 ENV JAVA_HOME=$RTI_HOME/jre ENV CLASSPATH=$CLASSPATH:$RTI_HOME/lib/portico.jar ENV PATH=$PATH:$JAVA_HOME/bin

CMD ["/bin/bash"]

bergtwvd commented 9 years ago

Dockerfile #2

FROM bergtwvd/po-rti-base:2.0.1

install program

COPY ./sample /tmp/sample

set entrypoint

WORKDIR /tmp/sample ENTRYPOINT ["/bin/sh", "./start.sh"]

bergtwvd commented 9 years ago

start.sh starts the sample program:

!/bin/sh

java -cp ./sample.jar:$RTI_HOME/lib/portico.jar ieee1516e.ExampleFederate $*

bergtwvd commented 9 years ago

Additonal tests show:

who is at fault here? weave, jgroups or Portico?

michaelrfraser commented 9 years ago

Hi there bergtwvd, thanks for your report. Just a quick note to advise that it has been received.

I'll have a chat with Tim about what might be at fault here. UDP can be a bit of a fickle beast, especially when VMs are involved.

Some suggestions below:

  1. In the past I've had to change the Virtual Box network adaptor type between NAT and Bridged to get UDP connectivity with the physical network (can't remember which one of the two worked, but I think it might be Bridged). I'm not familiar with the details with weave, or how it fits into the Virtual Box eco-system, but it sounds like it replaces the Virtual Box network adaptor completely?
  2. If your machines have a statically assigned IP address, try setting a default gateway in your network config
  3. The other thing I'd recommend would be to sit wireshark on both machines and sniff traffic on 224.0.75.75:7500. Check if wireshark can see traffic from both the local machine (which should mean that jgroups is working), and the remote machine (what weave will have the most influence over).

Hope that helps,

Michael

bergtwvd commented 9 years ago

Michael, thanks for your input.

I have different setups for the VB networking, depending on which network my laptop to connected (work or home).

At home I set up both VB instances as follows:

At work I use Host Only for adapter 2, with some additional network configuration inside docker for IP routing.

Weave creates an overlay network, using the IP address of adapter 2 to communicate between hosts. Weave does not create additional adapters. When I start containers and attach them to weave I can ping them on their weave overlay IP address. When using the Pitch RTI (configured to use TCP/IP), the weave net works fine. I can for example run part of the federation in AWS and part on my laptop, using weave. So I am flexible where I move my containers to.

The following Jgroups multicast test appears to succeed:

Where: The 11.x.x.x address is a weave overlay address. The 10.1.x.x address is for VB#1 The 10.2.x.x address is for VB#2

On a VB#2 container:

root@475d52fd2266:/tmp/sample# java -cp $RTI_HOME/lib/portico.jar org.jgroups.tests.McastReceiverTest -mcast_addr 228.10.10.10 -port 20913 Socket=0.0.0.0/0.0.0.0:20913, bind interface=/fe80:0:0:0:b460:cdff:fec3:1c8d%ethwe Socket=0.0.0.0/0.0.0.0:20913, bind interface=/11.11.11.3 Socket=0.0.0.0/0.0.0.0:20913, bind interface=/fe80:0:0:0:42:aff:fe02:104%eth0 Socket=0.0.0.0/0.0.0.0:20913, bind interface=/10.2.1.4 Socket=0.0.0.0/0.0.0.0:20913, bind interface=/0:0:0:0:0:0:0:1%lo Socket=0.0.0.0/0.0.0.0:20913, bind interface=/127.0.0.1 HELLO [sender=11.11.11.1:20913] HELLO [sender=11.11.11.1:20913] HELLO [sender=11.11.11.1:20913] HELLO [sender=11.11.11.1:20913] HELLO [sender=11.11.11.1:20913] HELLO [sender=11.11.11.1:20913] HELLO [sender=11.11.11.1:20913] HELLO [sender=11.11.11.1:20913]

On a VB#1 container: root@00f7305172ed:/tmp/sample# java -cp $RTI_HOME/lib/portico.jar org.jgroups.tests.McastSenderTest -mcast_addr 228.10.10.10 -port 20913 Socket #1=0.0.0.0/0.0.0.0:20913, ttl=32, bind interface=/fe80:0:0:0:f07e:4eff:fe8e:a744%ethwe Socket #2=0.0.0.0/0.0.0.0:20913, ttl=32, bind interface=/11.11.11.1 Socket #3=0.0.0.0/0.0.0.0:20913, ttl=32, bind interface=/fe80:0:0:0:42:aff:fe01:104%eth0 Socket #4=0.0.0.0/0.0.0.0:20913, ttl=32, bind interface=/10.1.1.4 Socket #5=0.0.0.0/0.0.0.0:20913, ttl=32, bind interface=/0:0:0:0:0:0:0:1%lo Socket #6=0.0.0.0/0.0.0.0:20913, ttl=32, bind interface=/127.0.0.1

HELLO

As a fall back I would like to configure JGroups to use TCP and see if that works. Did you ever run the Portico RTI with such a JGroups configuration?

-- Tom

From: Michael Fraser [mailto:notifications@github.com] Sent: dinsdag 2 juni 2015 13:49 To: openlvc/portico Cc: Berg, T.W. (Tom) van den Subject: Re: [portico] Running Portico example across Docker Weave does not work (#140)

Hi there bergtwvd, thanks for your report. Just a quick note to advise that it has been received.

I'll have a chat with Tim about what might be at fault here. UDP can be a bit of a fickle beast, especially when VMs are involved.

Some suggestions below:

  1. In the past I've had to change the Virtual Box network adaptor type between NAT and Bridged to get UDP connectivity with the physical network (can't remember which one of the two worked, but I think it might be Bridged). I'm not familiar with the details with weave, or how it fits into the Virtual Box eco-system, but it sounds like it replaces the Virtual Box network adaptor completely?
  2. If your machines have a statically assigned IP address, try setting a default gateway in your network config
  3. The other thing I'd recommend would be to sit wireshark on both machines and sniff traffic on 224.0.75.75:7500. Check if wireshark can see traffic from both the local machine (which should mean that jgroups is working), and the remote machine (what weave will have the most influence over).

Hope that helps,

Michael

— Reply to this email directly or view it on GitHubhttps://github.com/openlvc/portico/issues/140#issuecomment-107926379. This message may contain information that is not intended for you. If you are not the addressee or if this message was sent to you by mistake, you are requested to inform the sender and delete the message. TNO accepts no liability for the content of this e-mail, for the manner in which you use it and for damage of any kind resulting from the risks inherent to the electronic transmission of messages.

bergtwvd commented 9 years ago

I have uploaded a Docker image on the Docker Hub that can be used for testing.

For more information see: https://registry.hub.docker.com/u/bergtwvd/po/

-- Tom

rade commented 9 years ago

The trace output suggests that the two containers did find each other.

In B we have

TRACE [main] org.jgroups.protocols.PING: discovery took 5007 ms: responses: 1 total (0 servers (0 coord), 1 clients)
TRACE [main] org.jgroups.protocols.pbcast.GMS: e96f542b7216-17605: no initial members discovered: creating group as first member

whereas A shows

TRACE [main] org.jgroups.protocols.PING: discovery took 16 ms: responses: 2 total (1 servers (1 coord), 1 clients)
TRACE [main] org.jgroups.protocols.pbcast.GMS: a627c2c18b9a-60763: initial_mbrs are e96f542b7216-17605
DEBUG [main] org.jgroups.protocols.pbcast.GMS: election results: {e96f542b7216-17605=1}
DEBUG [main] org.jgroups.protocols.pbcast.GMS: sending JOIN(a627c2c18b9a-60763) to e96f542b7216-17605
TRACE [main] org.jgroups.protocols.UNICAST2: a627c2c18b9a-60763: created connection to e96f542b7216-17605 (conn_id=0)
bergtwvd commented 9 years ago

Yes, that is correct, that is what I see happening as well. It goes wrong after that.

At application level, federate A wants to join the federation after its attempt to create it first. Both federates A and B run the same code shown below (a fraction).

I probably do another run to get additional debug from Portico, since it is hard to put the JGroups log statements in context.

<.../>
///////////////////////////////////////////////////////////////////////////
////////////////////////// Main Simulation Method /////////////////////////
///////////////////////////////////////////////////////////////////////////
/**
 * This is the main simulation loop. It can be thought of as the main method of
 * the federate. For a description of the basic flow of this federate, see the
 * class level comments
 */
public void runFederate( String federateName ) throws Exception
{
    /////////////////////////////////////////////////
    // 1 & 2. create the RTIambassador and Connect //
    /////////////////////////////////////////////////
    log( "Creating RTIambassador" );
    rtiamb = RtiFactoryFactory.getRtiFactory().getRtiAmbassador();
    encoderFactory = RtiFactoryFactory.getRtiFactory().getEncoderFactory();

    // connect
    log( "Connecting..." );
    fedamb = new ExampleFederateAmbassador( this );
    rtiamb.connect( fedamb, CallbackModel.HLA_EVOKED );

    //////////////////////////////
    // 3. create the federation //
    //////////////////////////////
    log( "Creating Federation..." );
    // We attempt to create a new federation with the first three of the
    // restaurant FOM modules covering processes, food and drink
    try
    {
        URL[] modules = new URL[]{
            (new File("foms/RestaurantProcesses.xml")).toURI().toURL(),
            (new File("foms/RestaurantFood.xml")).toURI().toURL(),
            (new File("foms/RestaurantDrinks.xml")).toURI().toURL()
        };

        rtiamb.createFederationExecution( "ExampleFederation", modules );
        log( "Created Federation" );
    }
    catch( FederationExecutionAlreadyExists exists )
    {
        log( "Didn't create federation, it already existed" );
    }
    catch( MalformedURLException urle )
    {
        log( "Exception loading one of the FOM modules from disk: " + urle.getMessage() );
        urle.printStackTrace();
        return;
    }

    ////////////////////////////
    // 4. join the federation //
    ////////////////////////////
    URL[] joinModules = new URL[]{
        (new File("foms/RestaurantSoup.xml")).toURI().toURL()
    };

    rtiamb.joinFederationExecution( federateName,            // name for the federate
                                    "ExampleFederateType",   // federate type
                                    "ExampleFederation",     // name of federation
                                    joinModules );           // modules we want to add

    log( "Joined Federation as " + federateName );
<.../>
bergtwvd commented 9 years ago

I have run the test again, with TRACE logging enabled for both Portico and JGroups. Federate XXX is started first (IP 11.11.11.1/24). When XXX waits for user input, federate YYY is started (IP 11.11.11.2/24).

Both IP addresses can be pinged in the Weave network.

Again a stacktrace at YYY. See next posts.

Some obervations: Log XXX line 253: GET_MBRS_REQ message from YYY (corresponds roughly to Log YYY line 104) Log XXX line 308: JOIN_RSP message to YYY (corresonds to Log YYY line 121) Log YYY line 163: portico.lrc.jgroups: SUCCESS Connected to channel Log YYY line 181-184: YYY creates a connection to itself?

bergtwvd commented 9 years ago

FEDERATE XXX

DEBUG [main] portico.lrc: Creating new LRC DEBUG [main] portico.lrc: Portico version: 2.0.1 (build 0) DEBUG [main] portico.lrc: Interface: IEEE1516e WARN [main] portico.lrc: MOM support is currently unsupported in IEEE-1516e federations. TRACE [main] portico.lrc: Provided connection implementation is "org.portico.bindings.jgroups.JGroupsConnection" TRACE [main] portico.lrc: Trying to load connection class: org.portico.bindings.jgroups.JGroupsConnection TRACE [main] portico.lrc: ATTEMPT create IConnection, class= class org.portico.bindings.jgroups.JGroupsConnection TRACE [main] portico.lrc: SUCCESS created IConnection, class= class org.portico.bindings.jgroups.JGroupsConnection TRACE [main] portico.lrc: Applying modules using component keyword: lrc1516e TRACE [main] portico.lrc: STARTING Apply module [lrc-base] to LRC TRACE [main] portico.lrc: Applied [82/92] handlers TRACE [main] portico.lrc: STARTING Apply module [lrc1516-callback] to LRC TRACE [main] portico.lrc: Applied [0/11] handlers TRACE [main] portico.lrc: STARTING Apply module [lrc1516e-callback] to LRC TRACE [main] portico.lrc: Applied [24/24] handlers TRACE [main] portico.lrc: STARTING Apply module [lrc13-callback] to LRC TRACE [main] portico.lrc: Applied [0/24] handlers DEBUG [main] portico.lrc: Messaging framework configuration complete INFO [main] portico.lrc: LRC initialized (HLA version: IEEE1516e) INFO [main] portico.lrc: Opening LRC Connection INFO [main] portico.lrc.jgroups: jgroups connection is up and running DEBUG [main] portico.lrc.fom: Parsing FED file (format=ieee1516e): file:/usr/local/portico/portico-2.0.1/examples/java/ieee1516e/foms/RestaurantProcesses.xml DEBUG [main] portico.lrc.fom: Parsing FED file (format=ieee1516e): file:/usr/local/portico/portico-2.0.1/examples/java/ieee1516e/foms/RestaurantFood.xml DEBUG [main] portico.lrc.fom: Parsing FED file (format=ieee1516e): file:/usr/local/portico/portico-2.0.1/examples/java/ieee1516e/foms/RestaurantDrinks.xml DEBUG [main] portico.lrc: Standard MIM not present - adding it DEBUG [main] portico.lrc.fom: Parsing FED file (format=ieee1516e): jar:file:/usr/local/portico/portico-2.0.1/lib/portico.jar!/etc/ieee1516e/HLAstandardMIM.xml TRACE [main] portico.lrc.merger: Beginning merge of 4 FOM models TRACE [main] portico.lrc.merger: Merging [file:/usr/local/portico/portico-2.0.1/examples/java/ieee1516e/foms/RestaurantProcesses.xml] into combined FOM TRACE [main] portico.lrc.merger: -> Inserting class [HLAobjectRoot.Employee] TRACE [main] portico.lrc.merger: -> Inserting class [HLAobjectRoot.Employee.Waiter] TRACE [main] portico.lrc.merger: -> Inserting class [HLAobjectRoot.Employee.Cashier] TRACE [main] portico.lrc.merger: -> Inserting class [HLAobjectRoot.Employee.Greeter] TRACE [main] portico.lrc.merger: -> Inserting class [HLAobjectRoot.Employee.Dishwasher] TRACE [main] portico.lrc.merger: -> Inserting class [HLAobjectRoot.Employee.Cook] TRACE [main] portico.lrc.merger: -> Inserting class [HLAobjectRoot.Customer] TRACE [main] portico.lrc.merger: -> Inserting class [HLAobjectRoot.Order] TRACE [main] portico.lrc.merger: -> Inserting class [HLAinteractionRoot.CustomerTransactions] TRACE [main] portico.lrc.merger: -> Inserting class [HLAinteractionRoot.CustomerTransactions.CustomerSeated] TRACE [main] portico.lrc.merger: -> Inserting class [HLAinteractionRoot.CustomerTransactions.CustomerPays] TRACE [main] portico.lrc.merger: -> Inserting class [HLAinteractionRoot.CustomerTransactions.CustomerPays.ByCreditCard] TRACE [main] portico.lrc.merger: -> Inserting class [HLAinteractionRoot.CustomerTransactions.CustomerPays.ByCash] TRACE [main] portico.lrc.merger: -> Inserting class [HLAinteractionRoot.CustomerTransactions.OrderTaken] TRACE [main] portico.lrc.merger: -> Inserting class [HLAinteractionRoot.CustomerTransactions.OrderTaken.FromAdultMeny] TRACE [main] portico.lrc.merger: -> Inserting class [HLAinteractionRoot.CustomerTransactions.OrderTaken.FromKidsMenu] TRACE [main] portico.lrc.merger: -> Inserting class [HLAinteractionRoot.CustomerTransactions.FoodServed] TRACE [main] portico.lrc.merger: -> Inserting class [HLAinteractionRoot.CustomerTransactions.FoodServed.DessertServed] TRACE [main] portico.lrc.merger: -> Inserting class [HLAinteractionRoot.CustomerTransactions.FoodServed.MainCourseServed] TRACE [main] portico.lrc.merger: -> Inserting class [HLAinteractionRoot.CustomerTransactions.FoodServed.DrinkServed] TRACE [main] portico.lrc.merger: -> Inserting class [HLAinteractionRoot.CustomerTransactions.FoodServed.AppetizerServed] TRACE [main] portico.lrc.merger: -> Inserting class [HLAinteractionRoot.CustomerTransactions.CustomerLeaves] TRACE [main] portico.lrc.merger: Merging [file:/usr/local/portico/portico-2.0.1/examples/java/ieee1516e/foms/RestaurantFood.xml] into combined FOM TRACE [main] portico.lrc.merger: -> Inserting class [HLAobjectRoot.Food] TRACE [main] portico.lrc.merger: -> Inserting class [HLAobjectRoot.Food.Appetizers] TRACE [main] portico.lrc.merger: -> Inserting class [HLAobjectRoot.Food.Appetizers.Soup] TRACE [main] portico.lrc.merger: -> Inserting class [HLAobjectRoot.Food.Appetizers.Nachos] TRACE [main] portico.lrc.merger: -> Inserting class [HLAobjectRoot.Food.Drink] TRACE [main] portico.lrc.merger: -> Inserting class [HLAobjectRoot.Food.MainCourse] TRACE [main] portico.lrc.merger: -> Inserting class [HLAobjectRoot.Food.SideDish] TRACE [main] portico.lrc.merger: -> Inserting class [HLAobjectRoot.Food.SideDish.Broccoli] TRACE [main] portico.lrc.merger: -> Inserting class [HLAobjectRoot.Food.SideDish.BakedPotato] TRACE [main] portico.lrc.merger: -> Inserting class [HLAobjectRoot.Food.SideDish.Corn] TRACE [main] portico.lrc.merger: -> Inserting class [HLAobjectRoot.Food.Dessert] TRACE [main] portico.lrc.merger: -> Inserting class [HLAobjectRoot.Food.Dessert.Cake] TRACE [main] portico.lrc.merger: -> Inserting class [HLAobjectRoot.Food.Dessert.IceCream] TRACE [main] portico.lrc.merger: -> Inserting class [HLAobjectRoot.Food.Dessert.IceCream.Vanilla] TRACE [main] portico.lrc.merger: -> Inserting class [HLAobjectRoot.Food.Dessert.IceCream.Chocolate] TRACE [main] portico.lrc.merger: -> Inserting class [HLAobjectRoot.Food.Entree] TRACE [main] portico.lrc.merger: -> Inserting class [HLAobjectRoot.Food.Entree.Pasta] TRACE [main] portico.lrc.merger: -> Inserting class [HLAobjectRoot.Food.Entree.Beef] TRACE [main] portico.lrc.merger: -> Inserting class [HLAobjectRoot.Food.Entree.Chicken] TRACE [main] portico.lrc.merger: -> Inserting class [HLAobjectRoot.Food.Entree.Seafood] TRACE [main] portico.lrc.merger: -> Inserting class [HLAobjectRoot.Food.Entree.Seafood.Fish] TRACE [main] portico.lrc.merger: -> Inserting class [HLAobjectRoot.Food.Entree.Seafood.Lobster] TRACE [main] portico.lrc.merger: -> Inserting class [HLAobjectRoot.Food.Entree.Seafood.Shrimp] TRACE [main] portico.lrc.merger: Merging [file:/usr/local/portico/portico-2.0.1/examples/java/ieee1516e/foms/RestaurantDrinks.xml] into combined FOM TRACE [main] portico.lrc.merger: -> Inserting class [HLAobjectRoot.Food.Drink.Coffee] TRACE [main] portico.lrc.merger: -> Inserting class [HLAobjectRoot.Food.Drink.Water] TRACE [main] portico.lrc.merger: -> Inserting class [HLAobjectRoot.Food.Drink.Soda] DEBUG [main] portico.lrc: ATTEMPT Create federation execution [ExampleFederation] TRACE [main] portico.lrc.jgroups: ATTEMPT Connecting to channel [ExampleFederation] DEBUG [main] org.jgroups.conf.ClassConfigurator: Using jg-magic-map.xml as magic number file and jg-protocol-ids.xml for protocol IDs DEBUG [main] org.jgroups.stack.Configurator: set property UDP.diagnostics_addr to default value /224.0.75.75 DEBUG [main] org.jgroups.protocols.FRAG2: received CONFIG event: {bind_addr=/11.11.11.1} TRACE [main] org.jgroups.blocks.MessageDispatcher$ProtocolAdapter: setting local_addr (null) to 79e078919f2e-55593 DEBUG [main] org.jgroups.protocols.FRAG2: received CONFIG event: {flush_supported=true} TRACE [main] org.jgroups.protocols.pbcast.STABLE: stable task started TRACE [main] org.jgroups.protocols.UNICAST2: 79e078919f2e-55593: stable task started DEBUG [main] org.jgroups.protocols.UDP: sockets will use interface 11.11.11.1 WARN [main] org.jgroups.protocols.UDP: [JGRP00014] the send buffer of socket DatagramSocket was set to 640KB, but the OS only allocated 212.99KB. This might lead to performance problems. Please set your max send buffer in the OS correctly (e.g. net.core.wmem_max on Linux) WARN [main] org.jgroups.protocols.UDP: [JGRP00014] the receive buffer of socket DatagramSocket was set to 8MB, but the OS only allocated 212.99KB. This might lead to performance problems. Please set your max receive buffer in the OS correctly (e.g. net.core.rmem_max on Linux) WARN [main] org.jgroups.protocols.UDP: [JGRP00014] the send buffer of socket MulticastSocket was set to 640KB, but the OS only allocated 212.99KB. This might lead to performance problems. Please set your max send buffer in the OS correctly (e.g. net.core.wmem_max on Linux) WARN [main] org.jgroups.protocols.UDP: [JGRP00014] the receive buffer of socket MulticastSocket was set to 8MB, but the OS only allocated 212.99KB. This might lead to performance problems. Please set your max receive buffer in the OS correctly (e.g. net.core.rmem_max on Linux) DEBUG [main] org.jgroups.protocols.UDP: socket information: , mcast_addr=239.255.20.16:20913, bind_addr=/11.11.11.1, ttl=8 sock: bound to 11.11.11.1:33013, receive buffer size=212992, send buffer size=212992 mcast_sock: bound to 11.11.11.1:20913, send buffer size=212992, receive buffer size=212992 TRACE [main] org.jgroups.protocols.UDP: joined /224.0.75.75:7500 on ethwe TRACE [main] org.jgroups.protocols.UDP: joined /224.0.75.75:7500 on eth0 TRACE [main] org.jgroups.protocols.UDP: joined /224.0.75.75:7500 on lo TRACE [main] org.jgroups.protocols.UDP: sending msg to null, src=79e078919f2e-55593, headers are PING: [PING: type=GET_MBRS_REQ, cluster=ExampleFederation, arg=79e078919f2e-55593, view_id=, is_server=false, is_coord=false, logical_name=79e078919f2e-55593, physical_addrs=11.11.11.1:33013], UDP: [channel_name=ExampleFederation] TRACE [main] org.jgroups.protocols.UDP: looping back message [dst: , src: 79e078919f2e-55593 (2 headers), size=0 bytes, flags=OOB] TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UDP: received [dst: , src: 79e078919f2e-55593 (2 headers), size=0 bytes, flags=OOB], headers are PING: [PING: type=GET_MBRS_REQ, cluster=ExampleFederation, arg=79e078919f2e-55593, view_id=, is_server=false, is_coord=false, logical_name=79e078919f2e-55593, physical_addrs=11.11.11.1:33013], UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UDP: received [dst: , src: 79e078919f2e-55593 (2 headers), size=0 bytes, flags=OOB], headers are PING: [PING: type=GET_MBRS_REQ, cluster=ExampleFederation, arg=79e078919f2e-55593, view_id=, is_server=false, is_coord=false, logical_name=79e078919f2e-55593, physical_addrs=11.11.11.1:33013], UDP: [channel_name=ExampleFederation] TRACE [Incoming] org.jgroups.protocols.PING: 79e078919f2e-55593: received GET_MBRS_REQ from 79e078919f2e-55593, sending staggered response [PING: type=GET_MBRS_RSP, arg=79e078919f2e-55593, view_id=, is_server=false, is_coord=false, logical_name=79e078919f2e-55593, physical_addrs=11.11.11.1:33013] TRACE [Incoming] org.jgroups.protocols.UDP: sending msg to 79e078919f2e-55593, src=79e078919f2e-55593, headers are PING: [PING: type=GET_MBRS_RSP, arg=79e078919f2e-55593, view_id=, is_server=false, is_coord=false, logical_name=79e078919f2e-55593, physical_addrs=11.11.11.1:33013], UDP: [channel_name=ExampleFederation] TRACE [Incoming] org.jgroups.protocols.UDP: looping back message [dst: 79e078919f2e-55593, src: 79e078919f2e-55593 (2 headers), size=0 bytes, flags=OOB] TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UDP: received [dst: 79e078919f2e-55593, src: 79e078919f2e-55593 (2 headers), size=0 bytes, flags=OOB], headers are PING: [PING: type=GET_MBRS_RSP, arg=79e078919f2e-55593, view_id=, is_server=false, is_coord=false, logical_name=79e078919f2e-55593, physical_addrs=11.11.11.1:33013], UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.PING: 79e078919f2e-55593: received GET_MBRS_RSP from 79e078919f2e-55593: 79e078919f2e-55593, view_id=, is_server=false, is_coord=false, logical_name=79e078919f2e-55593, physical_addrs=11.11.11.1:33013 TRACE [main] org.jgroups.protocols.PING: discovery took 5007 ms: responses: 1 total (0 servers (0 coord), 1 clients) TRACE [main] org.jgroups.protocols.pbcast.GMS: 79e078919f2e-55593: no initial members discovered: creating group as first member DEBUG [main] org.jgroups.protocols.pbcast.NAKACK2: [79e078919f2e-55593 setDigest()] existing digest: [] new digest: 79e078919f2e-55593: [0 (0)] resulting digest: 79e078919f2e-55593: [0 (0)] DEBUG [main] org.jgroups.protocols.pbcast.GMS: 79e078919f2e-55593: installing view [79e078919f2e-55593|0] [79e078919f2e-55593] TRACE [main] org.jgroups.protocols.pbcast.STABLE: 79e078919f2e-55593: resetting digest from NAKACK: 79e078919f2e-55593: [0] DEBUG [main] org.jgroups.protocols.pbcast.STABLE: resuming message garbage collection TRACE [main] org.jgroups.protocols.pbcast.STABLE: 79e078919f2e-55593: resetting digest from NAKACK: 79e078919f2e-55593: [0] DEBUG [main] org.jgroups.protocols.pbcast.STABLE: [ergonomics] setting max_bytes to 4MB (1 members) TRACE [main] org.jgroups.protocols.UFC: new membership: [79e078919f2e-55593] TRACE [main] org.jgroups.protocols.MFC: new membership: [79e078919f2e-55593] DEBUG [main] org.jgroups.protocols.pbcast.FLUSH: 79e078919f2e-55593: installing view [79e078919f2e-55593|0] [79e078919f2e-55593] DEBUG [main] org.jgroups.protocols.pbcast.FLUSH: 79e078919f2e-55593: received STOP_FLUSH, unblocking FLUSH.down() and sending UNBLOCK up TRACE [main] org.jgroups.protocols.pbcast.STABLE: 79e078919f2e-55593: resetting digest from NAKACK: 79e078919f2e-55593: [0] DEBUG [main] org.jgroups.protocols.pbcast.STABLE: resuming message garbage collection DEBUG [main] org.jgroups.protocols.pbcast.GMS: created group (first member). My view is [79e078919f2e-55593|0], impl is org.jgroups.protocols.pbcast.CoordGmsImpl DEBUG [main] portico.lrc.jgroups: SUCCESS Connected to channel [ExampleFederation] DEBUG [main] portico.lrc.jgroups: REQUEST createFederation: name=ExampleFederation TRACE [main] org.jgroups.protocols.RSVP: 79e078919f2e-55593: REQ --> null TRACE [main] org.jgroups.protocols.pbcast.NAKACK2: 79e078919f2e-55593 sending 79e078919f2e-55593#1 TRACE [main] org.jgroups.protocols.UDP: sending msg to null, src=79e078919f2e-55593, headers are 7777: CreateFederation, RSVP: REQ(0), NAKACK2: [MSG, seqno=1], UDP: [channel_name=ExampleFederation] TRACE [main] org.jgroups.protocols.UDP: looping back message [dst: , src: 79e078919f2e-55593 (4 headers), size=24647 bytes, flags=OOB|DONT_BUNDLE|NO_FC|RSVP] TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UDP: received [dst: , src: 79e078919f2e-55593 (4 headers), size=24647 bytes, flags=OOB|DONT_BUNDLE|NO_FC|RSVP], headers are 7777: CreateFederation, RSVP: REQ(0), NAKACK2: [MSG, seqno=1], UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.pbcast.NAKACK2: 79e078919f2e-55593: received 79e078919f2e-55593#1 TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.pbcast.NAKACK2: 79e078919f2e-55593: delivering 79e078919f2e-55593#1 TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.RSVP: 79e078919f2e-55593: REQ <-- 79e078919f2e-55593 TRACE [Incoming,ExampleFederation,79e078919f2e-55593] portico.lrc.jgroups: (incoming) asynchronous, channel=ExampleFederation, size=24647, source=79e078919f2e-55593 DEBUG [Incoming,ExampleFederation,79e078919f2e-55593] portico.lrc.jgroups: Received federation creation notification: federation=ExampleFederation, fomSize=24647b, source=79e078919f2e-55593 TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UDP: received [dst: , src: 79e078919f2e-55593 (4 headers), size=24647 bytes, flags=OOB|DONT_BUNDLE|NO_FC|RSVP], headers are 7777: CreateFederation, RSVP: REQ(0), NAKACK2: [MSG, seqno=1], UDP: [channel_name=ExampleFederation] INFO [Incoming,ExampleFederation,79e078919f2e-55593] portico.lrc.jgroups: Federation [ExampleFederation] has been created TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.RSVP: 79e078919f2e-55593: RSP --> 79e078919f2e-55593 TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UNICAST2: 79e078919f2e-55593: created connection to 79e078919f2e-55593 (conn_id=0) TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UNICAST2: 79e078919f2e-55593 --> DATA(79e078919f2e-55593: #1, conn_id=0, first) TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UDP: sending msg to 79e078919f2e-55593, src=79e078919f2e-55593, headers are RSVP: RSP(0), UNICAST2: DATA, seqno=1, first, UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UDP: looping back message [dst: 79e078919f2e-55593, src: 79e078919f2e-55593 (3 headers), size=0 bytes, flags=OOB|RSVP] TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UDP: received [dst: 79e078919f2e-55593, src: 79e078919f2e-55593 (3 headers), size=0 bytes, flags=OOB|RSVP], headers are RSVP: RSP(0), UNICAST2: DATA, seqno=1, first, UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UNICAST2: 79e078919f2e-55593 <-- DATA(79e078919f2e-55593: #1, first) TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UNICAST2: 79e078919f2e-55593: created receiver window for 79e078919f2e-55593 at seqno=#1 for conn-id=0 TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.RSVP: 79e078919f2e-55593: RSP <-- 79e078919f2e-55593 INFO [main] portico.lrc.jgroups: SUCCESS createFederation: name=ExampleFederation INFO [main] portico.lrc: SUCCESS Created federation execution [ExampleFederation] DEBUG [main] portico.lrc: ATTEMPT Join federate [XXX] to federation [ExampleFederation] DEBUG [main] portico.lrc.fom: Parsing FED file (format=ieee1516e): file:/usr/local/portico/portico-2.0.1/examples/java/ieee1516e/foms/RestaurantSoup.xml DEBUG [main] portico.lrc: Parsed [1] additional FOM modules DEBUG [main] portico.lrc.jgroups: Validate that [1] modules can merge successfully with the existing FOM TRACE [main] portico.lrc.merger: Beginning merge of 2 FOM models TRACE [main] portico.lrc.merger: Merging [file:/usr/local/portico/portico-2.0.1/examples/java/ieee1516e/foms/RestaurantSoup.xml] into combined FOM TRACE [main] portico.lrc.merger: -> Inserting class [HLAobjectRoot.Food.Appetizers.Soup.ClamChowder] TRACE [main] portico.lrc.merger: -> Inserting class [HLAobjectRoot.Food.Appetizers.Soup.ClamChowder.Manhattan] TRACE [main] portico.lrc.merger: -> Inserting class [HLAobjectRoot.Food.Appetizers.Soup.ClamChowder.NewEngland] TRACE [main] portico.lrc.merger: -> Inserting class [HLAobjectRoot.Food.Appetizers.Soup.BeefBarley] DEBUG [main] portico.lrc.jgroups: Modules can be merged successfully, continue with join DEBUG [main] portico.lrc.jgroups: REQUEST joinFederation: federate=XXX, federation=ExampleFederation TRACE [main] org.jgroups.protocols.RSVP: 79e078919f2e-55593: REQ --> null TRACE [main] org.jgroups.protocols.pbcast.NAKACK2: 79e078919f2e-55593 sending 79e078919f2e-55593#2 TRACE [main] org.jgroups.protocols.UDP: sending msg to null, src=79e078919f2e-55593, headers are 7777: JoinFederation, RSVP: REQ(1), NAKACK2: [MSG, seqno=2], UDP: [channel_name=ExampleFederation] TRACE [main] org.jgroups.protocols.UDP: looping back message [dst: , src: 79e078919f2e-55593 (4 headers), size=3 bytes, flags=OOB|DONT_BUNDLE|NO_FC|RSVP] TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UDP: received [dst: , src: 79e078919f2e-55593 (4 headers), size=3 bytes, flags=OOB|DONT_BUNDLE|NO_FC|RSVP], headers are 7777: JoinFederation, RSVP: REQ(1), NAKACK2: [MSG, seqno=2], UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UDP: received [dst: , src: 79e078919f2e-55593 (4 headers), size=3 bytes, flags=OOB|DONT_BUNDLE|NO_FC|RSVP], headers are 7777: JoinFederation, RSVP: REQ(1), NAKACK2: [MSG, seqno=2], UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.pbcast.NAKACK2: 79e078919f2e-55593: received 79e078919f2e-55593#2 TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.pbcast.NAKACK2: 79e078919f2e-55593: delivering 79e078919f2e-55593#2 TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.RSVP: 79e078919f2e-55593: REQ <-- 79e078919f2e-55593 TRACE [Incoming,ExampleFederation,79e078919f2e-55593] portico.lrc.jgroups: (incoming) asynchronous, channel=ExampleFederation, size=3, source=79e078919f2e-55593 DEBUG [Incoming,ExampleFederation,79e078919f2e-55593] portico.lrc.jgroups: Received federate join notification: federate=XXX, federation=ExampleFederation, source=79e078919f2e-55593 INFO [Incoming,ExampleFederation,79e078919f2e-55593] portico.lrc.jgroups: Federate [XXX] joined federation [ExampleFederation] TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.RSVP: 79e078919f2e-55593: RSP --> 79e078919f2e-55593 TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UNICAST2: 79e078919f2e-55593 --> DATA(79e078919f2e-55593: #2, conn_id=0) TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UDP: sending msg to 79e078919f2e-55593, src=79e078919f2e-55593, headers are RSVP: RSP(1), UNICAST2: DATA, seqno=2, UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UDP: looping back message [dst: 79e078919f2e-55593, src: 79e078919f2e-55593 (3 headers), size=0 bytes, flags=OOB|RSVP] TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UDP: received [dst: 79e078919f2e-55593, src: 79e078919f2e-55593 (3 headers), size=0 bytes, flags=OOB|RSVP], headers are RSVP: RSP(1), UNICAST2: DATA, seqno=2, UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UNICAST2: 79e078919f2e-55593 <-- DATA(79e078919f2e-55593: #2) TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.RSVP: 79e078919f2e-55593: RSP <-- 79e078919f2e-55593 INFO [main] portico.lrc.jgroups: SUCCESS Joined federation with name=XXX DEBUG [main] portico.lrc.jgroups: Merging 1 additional FOM modules that we receive with join request TRACE [main] portico.lrc.merger: Beginning merge of 2 FOM models TRACE [main] portico.lrc.merger: Merging [file:/usr/local/portico/portico-2.0.1/examples/java/ieee1516e/foms/RestaurantSoup.xml] into combined FOM TRACE [main] portico.lrc.merger: -> Inserting class [HLAobjectRoot.Food.Appetizers.Soup.BeefBarley] TRACE [main] portico.lrc.merger: -> Inserting class [HLAobjectRoot.Food.Appetizers.Soup.ClamChowder] TRACE [main] portico.lrc.merger: -> Inserting class [HLAobjectRoot.Food.Appetizers.Soup.ClamChowder.NewEngland] TRACE [main] portico.lrc.merger: -> Inserting class [HLAobjectRoot.Food.Appetizers.Soup.ClamChowder.Manhattan] TRACE [main] portico.lrc.jgroups: (outgoing) payload=RoleCall, size=2572, channel=ExampleFederation TRACE [main] org.jgroups.protocols.pbcast.NAKACK2: 79e078919f2e-55593 sending 79e078919f2e-55593#3 TRACE [main] org.jgroups.protocols.UDP: sending msg to null, src=79e078919f2e-55593, headers are NAKACK2: [MSG, seqno=3], UDP: [channel_name=ExampleFederation] TRACE [main] org.jgroups.protocols.UDP: looping back message [dst: , src: 79e078919f2e-55593 (2 headers), size=2572 bytes] TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UDP: received [dst: , src: 79e078919f2e-55593 (2 headers), size=2572 bytes], headers are NAKACK2: [MSG, seqno=3], UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.pbcast.NAKACK2: 79e078919f2e-55593: received 79e078919f2e-55593#3 TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.pbcast.NAKACK2: 79e078919f2e-55593: delivering 79e078919f2e-55593#3 TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.MFC: 79e078919f2e-55593 used 2572 credits, 3997428 remaining TRACE [Incoming,ExampleFederation,79e078919f2e-55593] portico.lrc.jgroups: (incoming) asynchronous, channel=ExampleFederation, size=2572, source=79e078919f2e-55593 TRACE [main] portico.lrc: joined federation, waiting for RoleCalls from [1] INFO [main] portico.lrc: SUCCESS Joined federate [XXX] to federation [ExampleFederation]: handle=1 DEBUG [main] portico.lrc: ATTEMPT Register sync point [ReadyToRun] by [XXX] DEBUG [main] portico.lrc: PENDING Register synchronization point [ReadyToRun] by [XXX] TRACE [main] portico.lrc.jgroups: (outgoing) payload=SyncRegistrationRequest, size=293, channel=ExampleFederation TRACE [main] org.jgroups.protocols.pbcast.NAKACK2: 79e078919f2e-55593 sending 79e078919f2e-55593#4 TRACE [main] org.jgroups.protocols.UDP: sending msg to null, src=79e078919f2e-55593, headers are NAKACK2: [MSG, seqno=4], UDP: [channel_name=ExampleFederation] TRACE [main] org.jgroups.protocols.UDP: looping back message [dst: , src: 79e078919f2e-55593 (2 headers), size=293 bytes] TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UDP: received [dst: , src: 79e078919f2e-55593 (2 headers), size=293 bytes], headers are NAKACK2: [MSG, seqno=4], UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.pbcast.NAKACK2: 79e078919f2e-55593: received 79e078919f2e-55593#4 TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.pbcast.NAKACK2: 79e078919f2e-55593: delivering 79e078919f2e-55593#4 TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.MFC: 79e078919f2e-55593 used 293 credits, 3997135 remaining TRACE [Incoming,ExampleFederation,79e078919f2e-55593] portico.lrc.jgroups: (incoming) asynchronous, channel=ExampleFederation, size=293, source=79e078919f2e-55593 TRACE [TransferQueueBundler,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.TP$TransferQueueBundler: sending 2 msgs (2977 bytes (4.73% of max_bundle_size) to 1 destination(s) TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UDP: received [dst: , src: 79e078919f2e-55593 (2 headers), size=2572 bytes], headers are NAKACK2: [MSG, seqno=3], UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UDP: received [dst: , src: 79e078919f2e-55593 (2 headers), size=293 bytes], headers are NAKACK2: [MSG, seqno=4], UDP: [channel_name=ExampleFederation] TRACE [main] portico.lrc.jgroups: (outgoing) payload=SyncPointAnnouncement, size=303, channel=ExampleFederation TRACE [main] org.jgroups.protocols.pbcast.NAKACK2: 79e078919f2e-55593 sending 79e078919f2e-55593#5 TRACE [main] org.jgroups.protocols.UDP: sending msg to null, src=79e078919f2e-55593, headers are NAKACK2: [MSG, seqno=5], UDP: [channel_name=ExampleFederation] TRACE [main] org.jgroups.protocols.UDP: looping back message [dst: , src: 79e078919f2e-55593 (2 headers), size=303 bytes] TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UDP: received [dst: , src: 79e078919f2e-55593 (2 headers), size=303 bytes], headers are NAKACK2: [MSG, seqno=5], UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.pbcast.NAKACK2: 79e078919f2e-55593: received 79e078919f2e-55593#5 TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.pbcast.NAKACK2: 79e078919f2e-55593: delivering 79e078919f2e-55593#5 TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.MFC: 79e078919f2e-55593 used 303 credits, 3996832 remaining TRACE [Incoming,ExampleFederation,79e078919f2e-55593] portico.lrc.jgroups: (incoming) asynchronous, channel=ExampleFederation, size=303, source=79e078919f2e-55593 INFO [main] portico.lrc: SUCCESS Registered sync point [ReadyToRun] by [XXX] TRACE [main] portico.lrc: CALLBACK synchronizationPointRegistrationSucceeded(label=ReadyToRun) TRACE [main] portico.lrc: CALLBACK announceSynchronizationPoint(label=ReadyToRun) TRACE [TransferQueueBundler,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.TP$TransferQueueBundler: sending 1 msgs (359 bytes (0.57% of max_bundle_size) to 1 destination(s) TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UDP: received [dst: , src: 79e078919f2e-55593 (2 headers), size=303 bytes], headers are NAKACK2: [MSG, seqno=5], UDP: [channel_name=ExampleFederation] TRACE [Incoming] org.jgroups.protocols.pbcast.STABLE: 79e078919f2e-55593: setting latest_local_digest from NAKACK: 79e078919f2e-55593: [5] TRACE [Incoming] org.jgroups.protocols.pbcast.STABLE: 79e078919f2e-55593: sending stable msg 79e078919f2e-55593: [5] TRACE [Incoming] org.jgroups.protocols.UDP: sending msg to null, src=79e078919f2e-55593, headers are STABLE: [STABLE_GOSSIP]: digest is 79e078919f2e-55593: [5 (5)], UDP: [channel_name=ExampleFederation] TRACE [Incoming] org.jgroups.protocols.UDP: looping back message [dst: , src: 79e078919f2e-55593 (2 headers), size=0 bytes, flags=OOB|NO_RELIABILITY] TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UDP: received [dst: , src: 79e078919f2e-55593 (2 headers), size=0 bytes, flags=OOB|NO_RELIABILITY], headers are STABLE: [STABLE_GOSSIP]: digest is 79e078919f2e-55593: [5 (5)], UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UDP: received [dst: , src: 79e078919f2e-55593 (2 headers), size=0 bytes, flags=OOB|NO_RELIABILITY], headers are STABLE: [STABLE_GOSSIP]: digest is 79e078919f2e-55593: [5 (5)], UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.pbcast.STABLE: 79e078919f2e-55593: handling digest from 79e078919f2e-55593 (0 votes): mine: 79e078919f2e-55593: [0] other: 79e078919f2e-55593: [5] result: 79e078919f2e-55593: [0]

TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.pbcast.STABLE: 79e078919f2e-55593: sending stability msg (in 324 ms) 79e078919f2e-55593: [0] TRACE [Incoming] org.jgroups.protocols.pbcast.STABLE: 79e078919f2e-55593: sending stability msg 79e078919f2e-55593: [0] TRACE [Incoming] org.jgroups.protocols.UDP: sending msg to null, src=79e078919f2e-55593, headers are STABLE: [STABILITY]: digest is 79e078919f2e-55593: [0 (5)], UDP: [channel_name=ExampleFederation] TRACE [Incoming] org.jgroups.protocols.UDP: looping back message [dst: , src: 79e078919f2e-55593 (2 headers), size=0 bytes, flags=OOB|NO_RELIABILITY] TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UDP: received [dst: , src: 79e078919f2e-55593 (2 headers), size=0 bytes, flags=OOB|NO_RELIABILITY], headers are STABLE: [STABILITY]: digest is 79e078919f2e-55593: [0 (5)], UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UDP: received [dst: , src: 79e078919f2e-55593 (2 headers), size=0 bytes, flags=OOB|NO_RELIABILITY], headers are STABLE: [STABILITY]: digest is 79e078919f2e-55593: [0 (5)], UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.pbcast.STABLE: 79e078919f2e-55593: received stability msg from 79e078919f2e-55593: 79e078919f2e-55593: [0] TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.pbcast.STABLE: 79e078919f2e-55593: resetting digest from NAKACK: 79e078919f2e-55593: [5] TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.pbcast.NAKACK2: 79e078919f2e-55593: received stable digest 79e078919f2e-55593: [0 (5)] TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.pbcast.NAKACK2: 79e078919f2e-55593: deleting msgs <= 0 from 79e078919f2e-55593 TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UDP: received [dst: , src: 9634aa65-b422-3949-ab50-db1b896238f4 (2 headers), size=0 bytes, flags=OOB], headers are PING: [PING: type=GET_MBRS_REQ, cluster=ExampleFederation, arg=9634aa65-b422-3949-ab50-db1b896238f4, view_id=, is_server=false, is_coord=false, logical_name=5f7daed5d167-29951, physical_addrs=11.11.11.2:33187], UDP: [channel_name=ExampleFederation] TRACE [Incoming] org.jgroups.protocols.PING: 79e078919f2e-55593: received GET_MBRS_REQ from 5f7daed5d167-29951, sending staggered response [PING: type=GET_MBRS_RSP, arg=79e078919f2e-55593, view_id=[79e078919f2e-55593|0], is_server=true, is_coord=true, logical_name=79e078919f2e-55593, physical_addrs=11.11.11.1:33013] TRACE [Incoming] org.jgroups.protocols.UDP: sending msg to 5f7daed5d167-29951, src=79e078919f2e-55593, headers are PING: [PING: type=GET_MBRS_RSP, arg=79e078919f2e-55593, view_id=[79e078919f2e-55593|0], is_server=true, is_coord=true, logical_name=79e078919f2e-55593, physical_addrs=11.11.11.1:33013], UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UDP: received [dst: 79e078919f2e-55593, src: 5f7daed5d167-29951 (3 headers), size=0 bytes, flags=OOB], headers are GMS: GmsHeader[JOIN_REQ_WITH_STATE_TRANSFER], UNICAST2: DATA, seqno=1, first, UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UNICAST2: 79e078919f2e-55593 <-- DATA(5f7daed5d167-29951: #1, first) TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UNICAST2: 79e078919f2e-55593: created receiver window for 5f7daed5d167-29951 at seqno=#1 for conn-id=0 TRACE [ViewHandler,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.pbcast.GMS: 79e078919f2e-55593: new members=[5f7daed5d167-29951], suspected=[], leaving=[], new view: [79e078919f2e-55593|1] [79e078919f2e-55593, 5f7daed5d167-29951] TRACE [ViewHandler,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.pbcast.NAKACK2: 79e078919f2e-55593 sending 79e078919f2e-55593#6 TRACE [ViewHandler,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UDP: sending msg to null, src=79e078919f2e-55593, headers are FLUSH: FLUSH[type=START_FLUSH,viewId=0,members=[79e078919f2e-55593]], NAKACK2: [MSG, seqno=6], UDP: [channel_name=ExampleFederation] TRACE [ViewHandler,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UDP: looping back message [dst: , src: 79e078919f2e-55593 (3 headers), size=0 bytes] DEBUG [ViewHandler,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.pbcast.FLUSH: 79e078919f2e-55593: flush coordinator is starting FLUSH with participants [79e078919f2e-55593] TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UDP: received [dst: , src: 79e078919f2e-55593 (3 headers), size=0 bytes], headers are FLUSH: FLUSH[type=START_FLUSH,viewId=0,members=[79e078919f2e-55593]], NAKACK2: [MSG, seqno=6], UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.pbcast.NAKACK2: 79e078919f2e-55593: received 79e078919f2e-55593#6 TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.pbcast.NAKACK2: 79e078919f2e-55593: delivering 79e078919f2e-55593#6 TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UNICAST2: 79e078919f2e-55593 --> DATA(79e078919f2e-55593: #3, conn_id=0) TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UDP: sending msg to 79e078919f2e-55593, src=79e078919f2e-55593, headers are FLUSH: FLUSH[type=FLUSH_COMPLETED,viewId=0], UNICAST2: DATA, seqno=3, UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UDP: looping back message [dst: 79e078919f2e-55593, src: 79e078919f2e-55593 (3 headers), size=0 bytes] DEBUG [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.pbcast.FLUSH: 79e078919f2e-55593: received START_FLUSH, responded with FLUSH_COMPLETED to 79e078919f2e-55593 TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UDP: received [dst: 79e078919f2e-55593, src: 79e078919f2e-55593 (3 headers), size=0 bytes], headers are FLUSH: FLUSH[type=FLUSH_COMPLETED,viewId=0], UNICAST2: DATA, seqno=3, UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UNICAST2: 79e078919f2e-55593 <-- DATA(79e078919f2e-55593: #3) DEBUG [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.pbcast.FLUSH: 79e078919f2e-55593: FLUSH_COMPLETED from 79e078919f2e-55593, completed true, flushMembers [79e078919f2e-55593], flushCompleted [79e078919f2e-55593] DEBUG [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.pbcast.FLUSH: 79e078919f2e-55593: all FLUSH_COMPLETED received TRACE [ViewHandler,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.pbcast.GMS: 79e078919f2e-55593: successful GMS flush by coordinator DEBUG [ViewHandler,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.pbcast.STABLE: suspending message garbage collection TRACE [Incoming] org.jgroups.protocols.UDP: sending msg to null, src=79e078919f2e-55593, headers are MERGE3: INFO: view_id=[79e078919f2e-55593|0], logical_name=79e078919f2e-55593, physical_addr=[11.11.11.1:33013], UDP: [channel_name=ExampleFederation] TRACE [Incoming] org.jgroups.protocols.UDP: looping back message [dst: , src: 79e078919f2e-55593 (2 headers), size=0 bytes] TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UDP: received [dst: , src: 79e078919f2e-55593 (2 headers), size=0 bytes], headers are MERGE3: INFO: view_id=[79e078919f2e-55593|0], logical_name=79e078919f2e-55593, physical_addr=[11.11.11.1:33013], UDP: [channel_name=ExampleFederation] DEBUG [ViewHandler,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.pbcast.STABLE: resume task started, max_suspend_time=33000 TRACE [ViewHandler,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.pbcast.GMS: 79e078919f2e-55593: mcasting view [79e078919f2e-55593|1] [79e078919f2e-55593, 5f7daed5d167-29951](2 mbrs)

TRACE [ViewHandler,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.pbcast.NAKACK2: 79e078919f2e-55593 sending 79e078919f2e-55593#7 TRACE [ViewHandler,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UDP: sending msg to null, src=79e078919f2e-55593, headers are GMS: GmsHeader[VIEW]: view=[79e078919f2e-55593|1] [79e078919f2e-55593, 5f7daed5d167-29951], NAKACK2: [MSG, seqno=7], UDP: [channel_name=ExampleFederation] TRACE [ViewHandler,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UDP: looping back message [dst: , src: 79e078919f2e-55593 (3 headers), size=0 bytes] TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.MERGE3: 79e078919f2e-55593 <-- 79e078919f2e-55593: INFO: view_id=[79e078919f2e-55593|0], logical_name=79e078919f2e-55593, physical_addr=[11.11.11.1:33013], cached views: 1 TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UDP: received [dst: , src: 79e078919f2e-55593 (3 headers), size=0 bytes], headers are GMS: GmsHeader[VIEW]: view=[79e078919f2e-55593|1] [79e078919f2e-55593, 5f7daed5d167-29951], NAKACK2: [MSG, seqno=7], UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.pbcast.NAKACK2: 79e078919f2e-55593: received 79e078919f2e-55593#7 TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.pbcast.NAKACK2: 79e078919f2e-55593: delivering 79e078919f2e-55593#7 DEBUG [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.pbcast.NAKACK2: [79e078919f2e-55593 setDigest()] existing digest: 79e078919f2e-55593: [7 (7)] new digest: 79e078919f2e-55593: [6 (6)], 5f7daed5d167-29951: [0 (0)] resulting digest: 79e078919f2e-55593: [7 (7)], 5f7daed5d167-29951: [0 (0)] DEBUG [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.pbcast.GMS: 79e078919f2e-55593: installing view [79e078919f2e-55593|1] [79e078919f2e-55593, 5f7daed5d167-29951] TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.pbcast.STABLE: 79e078919f2e-55593: resetting digest from NAKACK: 5f7daed5d167-29951: [0], 79e078919f2e-55593: [7] DEBUG [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.pbcast.STABLE: [ergonomics] setting max_bytes to 8MB (2 members) TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UFC: new membership: [79e078919f2e-55593, 5f7daed5d167-29951] TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.MFC: new membership: [79e078919f2e-55593, 5f7daed5d167-29951] DEBUG [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.pbcast.FLUSH: 79e078919f2e-55593: installing view [79e078919f2e-55593|1] [79e078919f2e-55593, 5f7daed5d167-29951] TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UNICAST2: 79e078919f2e-55593 --> DATA(79e078919f2e-55593: #4, conn_id=0) TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UDP: sending msg to 79e078919f2e-55593, src=79e078919f2e-55593, headers are GMS: GmsHeader[VIEW_ACK]: view=null, UNICAST2: DATA, seqno=4, UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UDP: looping back message [dst: 79e078919f2e-55593, src: 79e078919f2e-55593 (3 headers), size=0 bytes, flags=OOB] TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UDP: received [dst: 79e078919f2e-55593, src: 79e078919f2e-55593 (3 headers), size=0 bytes, flags=OOB], headers are GMS: GmsHeader[VIEW_ACK]: view=null, UNICAST2: DATA, seqno=4, UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UNICAST2: 79e078919f2e-55593 <-- DATA(79e078919f2e-55593: #4) TRACE [ViewHandler,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.pbcast.GMS: 79e078919f2e-55593: received all 1 ACKs from members for view [79e078919f2e-55593|1] TRACE [ViewHandler,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UNICAST2: 79e078919f2e-55593: created connection to 5f7daed5d167-29951 (conn_id=1) TRACE [ViewHandler,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UNICAST2: 79e078919f2e-55593 --> DATA(5f7daed5d167-29951: #1, conn_id=1, first) TRACE [ViewHandler,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UDP: sending msg to 5f7daed5d167-29951, src=79e078919f2e-55593, headers are GMS: GmsHeader[JOIN_RSP]: join_rsp=view: [79e078919f2e-55593|1] [79e078919f2e-55593, 5f7daed5d167-29951], digest: 79e078919f2e-55593: [6 (6)], 5f7daed5d167-29951: [0 (0)], UNICAST2: DATA, seqno=1, conn_id=1, first, UDP: [channel_name=ExampleFederation] TRACE [TransferQueueBundler,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.TP$TransferQueueBundler: sending 3 msgs (348 bytes (0.55% of max_bundle_size) to 1 destination(s) TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UDP: received [dst: , src: 79e078919f2e-55593 (3 headers), size=0 bytes], headers are FLUSH: FLUSH[type=START_FLUSH,viewId=0,members=[79e078919f2e-55593]], NAKACK2: [MSG, seqno=6], UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UDP: received [dst: , src: 79e078919f2e-55593 (2 headers), size=0 bytes], headers are MERGE3: INFO: view_id=[79e078919f2e-55593|0], logical_name=79e078919f2e-55593, physical_addr=[11.11.11.1:33013], UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UDP: received [dst: , src: 79e078919f2e-55593 (3 headers), size=0 bytes], headers are GMS: GmsHeader[VIEW]: view=[79e078919f2e-55593|1] [79e078919f2e-55593, 5f7daed5d167-29951], NAKACK2: [MSG, seqno=7], UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UDP: received [dst: 79e078919f2e-55593, src: 5f7daed5d167-29951 (3 headers), size=0 bytes, flags=OOB], headers are GMS: GmsHeader[VIEW_ACK]: view=null, UNICAST2: DATA, seqno=2, UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UNICAST2: 79e078919f2e-55593 <-- DATA(5f7daed5d167-29951: #2) TRACE [ViewHandler,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.pbcast.GMS: 79e078919f2e-55593: received all ACKs (1) from joiners for view [79e078919f2e-55593|1] TRACE [ViewHandler,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.pbcast.STABLE: 79e078919f2e-55593: resetting digest from NAKACK: 5f7daed5d167-29951: [0], 79e078919f2e-55593: [7] DEBUG [ViewHandler,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.pbcast.STABLE: resuming message garbage collection TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UDP: received [dst: 79e078919f2e-55593, src: 5f7daed5d167-29951 (3 headers), size=0 bytes, flags=OOB], headers are GMS: GmsHeader[VIEW_ACK]: view=null, UNICAST2: DATA, seqno=3, UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UNICAST2: 79e078919f2e-55593 <-- DATA(5f7daed5d167-29951: #3) TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UDP: received [dst: 79e078919f2e-55593, src: 5f7daed5d167-29951 (3 headers), size=0 bytes], headers are STATE_TRANSFER: type=STATE_REQ, UNICAST2: DATA, seqno=4, UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UNICAST2: 79e078919f2e-55593 <-- DATA(5f7daed5d167-29951: #4) DEBUG [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.pbcast.STATE_TRANSFER: 79e078919f2e-55593: received state request from 5f7daed5d167-29951 TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.pbcast.STATE_TRANSFER: 79e078919f2e-55593: sending state to 5f7daed5d167-29951 (size=25.64KB) TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UFC: bypassing flow control because of synchronous response Thread[Incoming,ExampleFederation,79e078919f2e-55593,5,JGroups channel] TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UNICAST2: 79e078919f2e-55593 --> DATA(5f7daed5d167-29951: #2, conn_id=1) TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UDP: sending msg to 5f7daed5d167-29951, src=79e078919f2e-55593, headers are STATE_TRANSFER: type=STATE_RSP, UNICAST2: DATA, seqno=2, conn_id=1, UDP: [channel_name=ExampleFederation] TRACE [Incoming] org.jgroups.protocols.UDP: sending msg to null, src=79e078919f2e-55593, headers are FD_ALL: heartbeat, UDP: [channel_name=ExampleFederation] TRACE [Incoming] org.jgroups.protocols.UDP: looping back message [dst: , src: 79e078919f2e-55593 (2 headers), size=0 bytes, flags=OOB] TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UDP: received [dst: , src: 79e078919f2e-55593 (2 headers), size=0 bytes, flags=OOB], headers are FD_ALL: heartbeat, UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UDP: received [dst: , src: 79e078919f2e-55593 (2 headers), size=0 bytes, flags=OOB], headers are FD_ALL: heartbeat, UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UDP: received [dst: , src: 5f7daed5d167-29951 (2 headers), size=0 bytes, flags=OOB], headers are FD_ALL: heartbeat, UDP: [channel_name=ExampleFederation] TRACE [Incoming] org.jgroups.protocols.pbcast.STABLE: 79e078919f2e-55593: setting latest_local_digest from NAKACK: 5f7daed5d167-29951: [0], 79e078919f2e-55593: [7] TRACE [Incoming] org.jgroups.protocols.pbcast.STABLE: 79e078919f2e-55593: sending stable msg 5f7daed5d167-29951: [0], 79e078919f2e-55593: [7] TRACE [Incoming] org.jgroups.protocols.UDP: sending msg to null, src=79e078919f2e-55593, headers are STABLE: [STABLE_GOSSIP]: digest is 79e078919f2e-55593: [7 (7)], 5f7daed5d167-29951: [0 (0)], UDP: [channel_name=ExampleFederation] TRACE [Incoming] org.jgroups.protocols.UDP: looping back message [dst: , src: 79e078919f2e-55593 (2 headers), size=0 bytes, flags=OOB|NO_RELIABILITY] TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UDP: received [dst: , src: 79e078919f2e-55593 (2 headers), size=0 bytes, flags=OOB|NO_RELIABILITY], headers are STABLE: [STABLE_GOSSIP]: digest is 79e078919f2e-55593: [7 (7)], 5f7daed5d167-29951: [0 (0)], UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UDP: received [dst: , src: 79e078919f2e-55593 (2 headers), size=0 bytes, flags=OOB|NO_RELIABILITY], headers are STABLE: [STABLE_GOSSIP]: digest is 79e078919f2e-55593: [7 (7)], 5f7daed5d167-29951: [0 (0)], UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.pbcast.STABLE: 79e078919f2e-55593: handling digest from 79e078919f2e-55593 (0 votes): mine: 5f7daed5d167-29951: [0], 79e078919f2e-55593: [7] other: 5f7daed5d167-29951: [0], 79e078919f2e-55593: [7] result: 5f7daed5d167-29951: [0], 79e078919f2e-55593: [7]

TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UDP: received [dst: , src: 5f7daed5d167-29951 (3 headers), size=0 bytes], headers are FLUSH: FLUSH[type=STOP_FLUSH,viewId=1], NAKACK2: [MSG, seqno=1], UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.pbcast.NAKACK2: 79e078919f2e-55593: received 5f7daed5d167-29951#1 TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.pbcast.NAKACK2: 79e078919f2e-55593: delivering 5f7daed5d167-29951#1 DEBUG [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.pbcast.FLUSH: 79e078919f2e-55593: received STOP_FLUSH, unblocking FLUSH.down() and sending UNBLOCK up TRACE [Incoming] org.jgroups.protocols.UDP: sending msg to null, src=79e078919f2e-55593, headers are FD_ALL: heartbeat, UDP: [channel_name=ExampleFederation] TRACE [Incoming] org.jgroups.protocols.UDP: looping back message [dst: , src: 79e078919f2e-55593 (2 headers), size=0 bytes, flags=OOB] TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UDP: received [dst: , src: 79e078919f2e-55593 (2 headers), size=0 bytes, flags=OOB], headers are FD_ALL: heartbeat, UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UDP: received [dst: , src: 79e078919f2e-55593 (2 headers), size=0 bytes, flags=OOB], headers are FD_ALL: heartbeat, UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UDP: received [dst: , src: 5f7daed5d167-29951 (2 headers), size=0 bytes, flags=OOB], headers are FD_ALL: heartbeat, UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UDP: received [dst: , src: 5f7daed5d167-29951 (3 headers), size=0 bytes, flags=RSVP], headers are RSVP: REQ-ONLY(0), NAKACK2: [MSG, seqno=3], UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.pbcast.NAKACK2: 79e078919f2e-55593: received 5f7daed5d167-29951#3 TRACE [Incoming] org.jgroups.protocols.pbcast.NAKACK2: 79e078919f2e-55593: sending XMIT_REQ ([2]) to 5f7daed5d167-29951 TRACE [Incoming] org.jgroups.protocols.UDP: sending msg to 5f7daed5d167-29951, src=79e078919f2e-55593, headers are NAKACK2: [XMIT_REQ, sender=5f7daed5d167-29951], UDP: [channel_name=ExampleFederation] TRACE [Incoming] org.jgroups.protocols.UDP: sending msg to null, src=79e078919f2e-55593, headers are FD_ALL: heartbeat, UDP: [channel_name=ExampleFederation] TRACE [Incoming] org.jgroups.protocols.UDP: looping back message [dst: , src: 79e078919f2e-55593 (2 headers), size=0 bytes, flags=OOB] TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UDP: received [dst: , src: 79e078919f2e-55593 (2 headers), size=0 bytes, flags=OOB], headers are FD_ALL: heartbeat, UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UDP: received [dst: , src: 79e078919f2e-55593 (2 headers), size=0 bytes, flags=OOB], headers are FD_ALL: heartbeat, UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UDP: received [dst: , src: 5f7daed5d167-29951 (2 headers), size=0 bytes, flags=OOB], headers are FD_ALL: heartbeat, UDP: [channel_name=ExampleFederation] TRACE [Incoming] org.jgroups.protocols.pbcast.NAKACK2: 79e078919f2e-55593: sending XMIT_REQ ([2]) to 5f7daed5d167-29951 TRACE [Incoming] org.jgroups.protocols.UDP: sending msg to 5f7daed5d167-29951, src=79e078919f2e-55593, headers are NAKACK2: [XMIT_REQ, sender=5f7daed5d167-29951], UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UDP: received [dst: , src: 5f7daed5d167-29951 (3 headers), size=0 bytes, flags=RSVP], headers are RSVP: REQ-ONLY(0), NAKACK2: [MSG, seqno=4], UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.pbcast.NAKACK2: 79e078919f2e-55593: received 5f7daed5d167-29951#4 TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UDP: received [dst: , src: 5f7daed5d167-29951 (2 headers), size=0 bytes, flags=OOB|NO_RELIABILITY], headers are STABLE: [STABLE_GOSSIP]: digest is 79e078919f2e-55593: [7 (7)], 5f7daed5d167-29951: [4 (4)], UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.pbcast.STABLE: 79e078919f2e-55593: handling digest from 5f7daed5d167-29951 (1 votes): mine: 5f7daed5d167-29951: [0], 79e078919f2e-55593: [7] other: 5f7daed5d167-29951: [4], 79e078919f2e-55593: [7] result: 5f7daed5d167-29951: [0], 79e078919f2e-55593: [7]

TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.pbcast.STABLE: 79e078919f2e-55593: sending stability msg (in 1555 ms) 5f7daed5d167-29951: [0], 79e078919f2e-55593: [7] TRACE [Incoming] org.jgroups.protocols.pbcast.NAKACK2: 79e078919f2e-55593: sending XMIT_REQ ([2]) to 5f7daed5d167-29951 TRACE [Incoming] org.jgroups.protocols.UDP: sending msg to 5f7daed5d167-29951, src=79e078919f2e-55593, headers are NAKACK2: [XMIT_REQ, sender=5f7daed5d167-29951], UDP: [channel_name=ExampleFederation] TRACE [Incoming] org.jgroups.protocols.pbcast.NAKACK2: 79e078919f2e-55593: sending XMIT_REQ ([2]) to 5f7daed5d167-29951 TRACE [Incoming] org.jgroups.protocols.UDP: sending msg to 5f7daed5d167-29951, src=79e078919f2e-55593, headers are NAKACK2: [XMIT_REQ, sender=5f7daed5d167-29951], UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UDP: received [dst: , src: 5f7daed5d167-29951 (3 headers), size=0 bytes, flags=RSVP], headers are RSVP: REQ-ONLY(0), NAKACK2: [MSG, seqno=5], UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.pbcast.NAKACK2: 79e078919f2e-55593: received 5f7daed5d167-29951#5 TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UDP: received [dst: , src: 5f7daed5d167-29951 (2 headers), size=0 bytes, flags=OOB|NO_RELIABILITY], headers are STABLE: [STABILITY]: digest is 79e078919f2e-55593: [6 (7)], 5f7daed5d167-29951: [0 (4)], UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.pbcast.STABLE: 79e078919f2e-55593: received stability msg from 5f7daed5d167-29951: 5f7daed5d167-29951: [0], 79e078919f2e-55593: [6] TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.pbcast.STABLE: 79e078919f2e-55593: resetting digest from NAKACK: 5f7daed5d167-29951: [1], 79e078919f2e-55593: [7] TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.pbcast.NAKACK2: 79e078919f2e-55593: received stable digest 79e078919f2e-55593: [6 (7)], 5f7daed5d167-29951: [0 (4)] TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.pbcast.NAKACK2: 79e078919f2e-55593: deleting msgs <= 6 from 79e078919f2e-55593 TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.pbcast.NAKACK2: 79e078919f2e-55593: deleting msgs <= 0 from 5f7daed5d167-29951 TRACE [Incoming] org.jgroups.protocols.UDP: sending msg to null, src=79e078919f2e-55593, headers are FD_ALL: heartbeat, UDP: [channel_name=ExampleFederation] TRACE [Incoming] org.jgroups.protocols.UDP: looping back message [dst: , src: 79e078919f2e-55593 (2 headers), size=0 bytes, flags=OOB] TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UDP: received [dst: , src: 79e078919f2e-55593 (2 headers), size=0 bytes, flags=OOB], headers are FD_ALL: heartbeat, UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UDP: received [dst: , src: 79e078919f2e-55593 (2 headers), size=0 bytes, flags=OOB], headers are FD_ALL: heartbeat, UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UDP: received [dst: , src: 5f7daed5d167-29951 (2 headers), size=0 bytes, flags=OOB], headers are FD_ALL: heartbeat, UDP: [channel_name=ExampleFederation] TRACE [Incoming] org.jgroups.protocols.pbcast.NAKACK2: 79e078919f2e-55593: sending XMIT_REQ ([2]) to 5f7daed5d167-29951 TRACE [Incoming] org.jgroups.protocols.UDP: sending msg to 5f7daed5d167-29951, src=79e078919f2e-55593, headers are NAKACK2: [XMIT_REQ, sender=5f7daed5d167-29951], UDP: [channel_name=ExampleFederation] TRACE [Incoming] org.jgroups.protocols.pbcast.NAKACK2: 79e078919f2e-55593: sending XMIT_REQ ([2]) to 5f7daed5d167-29951 TRACE [Incoming] org.jgroups.protocols.UDP: sending msg to 5f7daed5d167-29951, src=79e078919f2e-55593, headers are NAKACK2: [XMIT_REQ, sender=5f7daed5d167-29951], UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UDP: received [dst: , src: 5f7daed5d167-29951 (3 headers), size=0 bytes, flags=RSVP], headers are RSVP: REQ-ONLY(0), NAKACK2: [MSG, seqno=6], UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.pbcast.NAKACK2: 79e078919f2e-55593: received 5f7daed5d167-29951#6 TRACE [Incoming] org.jgroups.protocols.pbcast.NAKACK2: 79e078919f2e-55593: sending XMIT_REQ ([2]) to 5f7daed5d167-29951 TRACE [Incoming] org.jgroups.protocols.UDP: sending msg to 5f7daed5d167-29951, src=79e078919f2e-55593, headers are NAKACK2: [XMIT_REQ, sender=5f7daed5d167-29951], UDP: [channel_name=ExampleFederation] TRACE [Incoming] org.jgroups.protocols.UDP: sending msg to null, src=79e078919f2e-55593, headers are FD_ALL: heartbeat, UDP: [channel_name=ExampleFederation] TRACE [Incoming] org.jgroups.protocols.UDP: looping back message [dst: , src: 79e078919f2e-55593 (2 headers), size=0 bytes, flags=OOB] TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UDP: received [dst: , src: 79e078919f2e-55593 (2 headers), size=0 bytes, flags=OOB], headers are FD_ALL: heartbeat, UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UDP: received [dst: , src: 79e078919f2e-55593 (2 headers), size=0 bytes, flags=OOB], headers are FD_ALL: heartbeat, UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UDP: received [dst: , src: 5f7daed5d167-29951 (2 headers), size=0 bytes, flags=OOB], headers are FD_ALL: heartbeat, UDP: [channel_name=ExampleFederation] TRACE [Incoming] org.jgroups.protocols.pbcast.NAKACK2: 79e078919f2e-55593: sending XMIT_REQ ([2]) to 5f7daed5d167-29951 TRACE [Incoming] org.jgroups.protocols.UDP: sending msg to 5f7daed5d167-29951, src=79e078919f2e-55593, headers are NAKACK2: [XMIT_REQ, sender=5f7daed5d167-29951], UDP: [channel_name=ExampleFederation] TRACE [Incoming] org.jgroups.protocols.pbcast.NAKACK2: 79e078919f2e-55593: sending XMIT_REQ ([2]) to 5f7daed5d167-29951 TRACE [Incoming] org.jgroups.protocols.UDP: sending msg to 5f7daed5d167-29951, src=79e078919f2e-55593, headers are NAKACK2: [XMIT_REQ, sender=5f7daed5d167-29951], UDP: [channel_name=ExampleFederation] TRACE [Incoming] org.jgroups.protocols.pbcast.NAKACK2: 79e078919f2e-55593: sending XMIT_REQ ([2]) to 5f7daed5d167-29951 TRACE [Incoming] org.jgroups.protocols.UDP: sending msg to 5f7daed5d167-29951, src=79e078919f2e-55593, headers are NAKACK2: [XMIT_REQ, sender=5f7daed5d167-29951], UDP: [channel_name=ExampleFederation] TRACE [Incoming] org.jgroups.protocols.UDP: sending msg to null, src=79e078919f2e-55593, headers are FD_ALL: heartbeat, UDP: [channel_name=ExampleFederation] TRACE [Incoming] org.jgroups.protocols.UDP: looping back message [dst: , src: 79e078919f2e-55593 (2 headers), size=0 bytes, flags=OOB] TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UDP: received [dst: , src: 79e078919f2e-55593 (2 headers), size=0 bytes, flags=OOB], headers are FD_ALL: heartbeat, UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UDP: received [dst: , src: 79e078919f2e-55593 (2 headers), size=0 bytes, flags=OOB], headers are FD_ALL: heartbeat, UDP: [channel_name=ExampleFederation] TRACE [Incoming] org.jgroups.protocols.pbcast.NAKACK2: 79e078919f2e-55593: sending XMIT_REQ ([2]) to 5f7daed5d167-29951 TRACE [Incoming] org.jgroups.protocols.UDP: sending msg to 5f7daed5d167-29951, src=79e078919f2e-55593, headers are NAKACK2: [XMIT_REQ, sender=5f7daed5d167-29951], UDP: [channel_name=ExampleFederation] TRACE [Incoming] org.jgroups.protocols.pbcast.NAKACK2: 79e078919f2e-55593: sending XMIT_REQ ([2]) to 5f7daed5d167-29951 TRACE [Incoming] org.jgroups.protocols.UDP: sending msg to 5f7daed5d167-29951, src=79e078919f2e-55593, headers are NAKACK2: [XMIT_REQ, sender=5f7daed5d167-29951], UDP: [channel_name=ExampleFederation] TRACE [Incoming] org.jgroups.protocols.pbcast.NAKACK2: 79e078919f2e-55593: sending XMIT_REQ ([2]) to 5f7daed5d167-29951 TRACE [Incoming] org.jgroups.protocols.UDP: sending msg to 5f7daed5d167-29951, src=79e078919f2e-55593, headers are NAKACK2: [XMIT_REQ, sender=5f7daed5d167-29951], UDP: [channel_name=ExampleFederation] TRACE [Incoming] org.jgroups.protocols.UDP: sending msg to null, src=79e078919f2e-55593, headers are FD_ALL: heartbeat, UDP: [channel_name=ExampleFederation] TRACE [Incoming] org.jgroups.protocols.UDP: looping back message [dst: , src: 79e078919f2e-55593 (2 headers), size=0 bytes, flags=OOB] TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UDP: received [dst: , src: 79e078919f2e-55593 (2 headers), size=0 bytes, flags=OOB], headers are FD_ALL: heartbeat, UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UDP: received [dst: , src: 79e078919f2e-55593 (2 headers), size=0 bytes, flags=OOB], headers are FD_ALL: heartbeat, UDP: [channel_name=ExampleFederation] TRACE [Incoming] org.jgroups.protocols.pbcast.NAKACK2: 79e078919f2e-55593: sending XMIT_REQ ([2]) to 5f7daed5d167-29951 TRACE [Incoming] org.jgroups.protocols.UDP: sending msg to 5f7daed5d167-29951, src=79e078919f2e-55593, headers are NAKACK2: [XMIT_REQ, sender=5f7daed5d167-29951], UDP: [channel_name=ExampleFederation] TRACE [Incoming] org.jgroups.protocols.pbcast.NAKACK2: 79e078919f2e-55593: sending XMIT_REQ ([2]) to 5f7daed5d167-29951 TRACE [Incoming] org.jgroups.protocols.UDP: sending msg to 5f7daed5d167-29951, src=79e078919f2e-55593, headers are NAKACK2: [XMIT_REQ, sender=5f7daed5d167-29951], UDP: [channel_name=ExampleFederation] TRACE [Incoming] org.jgroups.protocols.pbcast.NAKACK2: 79e078919f2e-55593: sending XMIT_REQ ([2]) to 5f7daed5d167-29951 TRACE [Incoming] org.jgroups.protocols.UDP: sending msg to 5f7daed5d167-29951, src=79e078919f2e-55593, headers are NAKACK2: [XMIT_REQ, sender=5f7daed5d167-29951], UDP: [channel_name=ExampleFederation] DEBUG [Incoming] org.jgroups.protocols.FD_ALL: haven't received a heartbeat from 5f7daed5d167-29951 for 8982 ms, adding it to suspect list DEBUG [Incoming] org.jgroups.protocols.FD_ALL: suspecting [5f7daed5d167-29951] TRACE [Incoming] org.jgroups.protocols.VERIFY_SUSPECT: verifying that 5f7daed5d167-29951 is dead TRACE [Incoming] org.jgroups.protocols.UDP: sending msg to 5f7daed5d167-29951, src=79e078919f2e-55593, headers are VERIFY_SUSPECT: [VERIFY_SUSPECT: ARE_YOU_DEAD], UDP: [channel_name=ExampleFederation] TRACE [Incoming] org.jgroups.protocols.UDP: sending msg to null, src=79e078919f2e-55593, headers are FD_ALL: heartbeat, UDP: [channel_name=ExampleFederation] TRACE [Incoming] org.jgroups.protocols.UDP: looping back message [dst: , src: 79e078919f2e-55593 (2 headers), size=0 bytes, flags=OOB] TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UDP: received [dst: , src: 79e078919f2e-55593 (2 headers), size=0 bytes, flags=OOB], headers are FD_ALL: heartbeat, UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UDP: received [dst: , src: 79e078919f2e-55593 (2 headers), size=0 bytes, flags=OOB], headers are FD_ALL: heartbeat, UDP: [channel_name=ExampleFederation] TRACE [Incoming] org.jgroups.protocols.pbcast.NAKACK2: 79e078919f2e-55593: sending XMIT_REQ ([2]) to 5f7daed5d167-29951 TRACE [Incoming] org.jgroups.protocols.UDP: sending msg to 5f7daed5d167-29951, src=79e078919f2e-55593, headers are NAKACK2: [XMIT_REQ, sender=5f7daed5d167-29951], UDP: [channel_name=ExampleFederation] TRACE [Incoming] org.jgroups.protocols.pbcast.NAKACK2: 79e078919f2e-55593: sending XMIT_REQ ([2]) to 5f7daed5d167-29951 TRACE [Incoming] org.jgroups.protocols.UDP: sending msg to 5f7daed5d167-29951, src=79e078919f2e-55593, headers are NAKACK2: [XMIT_REQ, sender=5f7daed5d167-29951], UDP: [channel_name=ExampleFederation] TRACE [VERIFY_SUSPECT.TimerThread,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.VERIFY_SUSPECT: 5f7daed5d167-29951 is dead (passing up SUSPECT event) DEBUG [VERIFY_SUSPECT.TimerThread,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.pbcast.FLUSH: 79e078919f2e-55593: suspect is 5f7daed5d167-29951, completed false, flushOkSet {}, flushMembers [] DEBUG [VERIFY_SUSPECT.TimerThread,ExampleFederation,79e078919f2e-55593] org.jgroups.blocks.RequestCorrelator: suspect=5f7daed5d167-29951 WARN [VERIFY_SUSPECT.TimerThread,ExampleFederation,79e078919f2e-55593] portico.lrc.jgroups: Detected that federate [1] may have crashed, investigating... DEBUG [Incoming] org.jgroups.protocols.FD_ALL: haven't received a heartbeat from 5f7daed5d167-29951 for 10992 ms, adding it to suspect list DEBUG [Incoming] org.jgroups.protocols.FD_ALL: suspecting [5f7daed5d167-29951] TRACE [Incoming] org.jgroups.protocols.VERIFY_SUSPECT: verifying that 5f7daed5d167-29951 is dead TRACE [Incoming] org.jgroups.protocols.UDP: sending msg to 5f7daed5d167-29951, src=79e078919f2e-55593, headers are VERIFY_SUSPECT: [VERIFY_SUSPECT: ARE_YOU_DEAD], UDP: [channel_name=ExampleFederation] TRACE [Incoming] org.jgroups.protocols.pbcast.NAKACK2: 79e078919f2e-55593: sending XMIT_REQ ([2]) to 5f7daed5d167-29951 TRACE [Incoming] org.jgroups.protocols.UDP: sending msg to 5f7daed5d167-29951, src=79e078919f2e-55593, headers are NAKACK2: [XMIT_REQ, sender=5f7daed5d167-29951], UDP: [channel_name=ExampleFederation] TRACE [ViewHandler,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.pbcast.GMS: 79e078919f2e-55593: new members=[], suspected=[5f7daed5d167-29951], leaving=[], new view: [79e078919f2e-55593|2] [79e078919f2e-55593] TRACE [ViewHandler,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.pbcast.NAKACK2: 79e078919f2e-55593 sending 79e078919f2e-55593#8 TRACE [ViewHandler,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UDP: sending msg to null, src=79e078919f2e-55593, headers are FLUSH: FLUSH[type=START_FLUSH,viewId=1,members=[79e078919f2e-55593]], NAKACK2: [MSG, seqno=8], UDP: [channel_name=ExampleFederation] TRACE [ViewHandler,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UDP: looping back message [dst: , src: 79e078919f2e-55593 (3 headers), size=0 bytes] DEBUG [ViewHandler,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.pbcast.FLUSH: 79e078919f2e-55593: flush coordinator is starting FLUSH with participants [79e078919f2e-55593] TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UDP: received [dst: , src: 79e078919f2e-55593 (3 headers), size=0 bytes], headers are FLUSH: FLUSH[type=START_FLUSH,viewId=1,members=[79e078919f2e-55593]], NAKACK2: [MSG, seqno=8], UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.pbcast.NAKACK2: 79e078919f2e-55593: received 79e078919f2e-55593#8 TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.pbcast.NAKACK2: 79e078919f2e-55593: delivering 79e078919f2e-55593#8 TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UNICAST2: 79e078919f2e-55593 --> DATA(79e078919f2e-55593: #5, conn_id=0) TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UDP: sending msg to 79e078919f2e-55593, src=79e078919f2e-55593, headers are FLUSH: FLUSH[type=FLUSH_COMPLETED,viewId=1], UNICAST2: DATA, seqno=5, UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UDP: looping back message [dst: 79e078919f2e-55593, src: 79e078919f2e-55593 (3 headers), size=0 bytes] TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UDP: received [dst: 79e078919f2e-55593, src: 79e078919f2e-55593 (3 headers), size=0 bytes], headers are FLUSH: FLUSH[type=FLUSH_COMPLETED,viewId=1], UNICAST2: DATA, seqno=5, UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UNICAST2: 79e078919f2e-55593 <-- DATA(79e078919f2e-55593: #5) DEBUG [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.pbcast.FLUSH: 79e078919f2e-55593: FLUSH_COMPLETED from 79e078919f2e-55593, completed true, flushMembers [79e078919f2e-55593], flushCompleted [79e078919f2e-55593] DEBUG [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.pbcast.FLUSH: 79e078919f2e-55593: all FLUSH_COMPLETED received DEBUG [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.pbcast.FLUSH: 79e078919f2e-55593: received START_FLUSH, responded with FLUSH_COMPLETED to 79e078919f2e-55593 TRACE [ViewHandler,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.pbcast.GMS: 79e078919f2e-55593: successful GMS flush by coordinator TRACE [ViewHandler,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.pbcast.GMS: 79e078919f2e-55593: mcasting view [79e078919f2e-55593|2] [79e078919f2e-55593](1 mbrs)

TRACE [ViewHandler,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.pbcast.NAKACK2: 79e078919f2e-55593 sending 79e078919f2e-55593#9 TRACE [ViewHandler,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UDP: sending msg to null, src=79e078919f2e-55593, headers are GMS: GmsHeader[VIEW]: view=[79e078919f2e-55593|2] [79e078919f2e-55593], NAKACK2: [MSG, seqno=9], UDP: [channel_name=ExampleFederation] TRACE [ViewHandler,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UDP: looping back message [dst: , src: 79e078919f2e-55593 (3 headers), size=0 bytes] TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UDP: received [dst: , src: 79e078919f2e-55593 (3 headers), size=0 bytes], headers are GMS: GmsHeader[VIEW]: view=[79e078919f2e-55593|2] [79e078919f2e-55593], NAKACK2: [MSG, seqno=9], UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.pbcast.NAKACK2: 79e078919f2e-55593: received 79e078919f2e-55593#9 TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.pbcast.NAKACK2: 79e078919f2e-55593: delivering 79e078919f2e-55593#9 DEBUG [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.pbcast.GMS: 79e078919f2e-55593: installing view [79e078919f2e-55593|2] [79e078919f2e-55593] TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UNICAST2: 79e078919f2e-55593: removing non members [5f7daed5d167-29951] TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UNICAST2: 79e078919f2e-55593 --> STABLE(5f7daed5d167-29951: 4-4, conn_id=0) TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UDP: sending msg to 5f7daed5d167-29951, src=79e078919f2e-55593, headers are UNICAST2: STABLE, seqno=4, UDP: [channel_name=ExampleFederation] TRACE [TransferQueueBundler,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.TP$TransferQueueBundler: sending 2 msgs (190 bytes (0.3% of max_bundle_size) to 1 destination(s) DEBUG [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.pbcast.NAKACK2: removed 5f7daed5d167-29951 from xmit_table (not member anymore) TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UDP: received [dst: , src: 79e078919f2e-55593 (3 headers), size=0 bytes], headers are FLUSH: FLUSH[type=START_FLUSH,viewId=1,members=[79e078919f2e-55593]], NAKACK2: [MSG, seqno=8], UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UDP: received [dst: , src: 79e078919f2e-55593 (3 headers), size=0 bytes], headers are GMS: GmsHeader[VIEW]: view=[79e078919f2e-55593|2] [79e078919f2e-55593], NAKACK2: [MSG, seqno=9], UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.pbcast.STABLE: 79e078919f2e-55593: resetting digest from NAKACK: 79e078919f2e-55593: [9] DEBUG [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.pbcast.STABLE: [ergonomics] setting max_bytes to 4MB (1 members) TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UFC: new membership: [79e078919f2e-55593] TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.MFC: new membership: [79e078919f2e-55593] TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.FRAG2: [VIEW_CHANGE] removed 5f7daed5d167-29951 from fragmentation table DEBUG [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.pbcast.FLUSH: 79e078919f2e-55593: installing view [79e078919f2e-55593|2] [79e078919f2e-55593] TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UNICAST2: 79e078919f2e-55593 --> DATA(79e078919f2e-55593: #6, conn_id=0) TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UDP: sending msg to 79e078919f2e-55593, src=79e078919f2e-55593, headers are GMS: GmsHeader[VIEW_ACK]: view=null, UNICAST2: DATA, seqno=6, UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UDP: looping back message [dst: 79e078919f2e-55593, src: 79e078919f2e-55593 (3 headers), size=0 bytes, flags=OOB] TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UDP: received [dst: 79e078919f2e-55593, src: 79e078919f2e-55593 (3 headers), size=0 bytes, flags=OOB], headers are GMS: GmsHeader[VIEW_ACK]: view=null, UNICAST2: DATA, seqno=6, UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UNICAST2: 79e078919f2e-55593 <-- DATA(79e078919f2e-55593: #6) TRACE [ViewHandler,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.pbcast.GMS: 79e078919f2e-55593: received all 1 ACKs from members for view [79e078919f2e-55593|2] TRACE [ViewHandler,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.pbcast.GMS: 79e078919f2e-55593: sending RESUME event DEBUG [ViewHandler,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.pbcast.FLUSH: 79e078919f2e-55593: received RESUME, sending STOP_FLUSH to all TRACE [ViewHandler,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.pbcast.NAKACK2: 79e078919f2e-55593 sending 79e078919f2e-55593#10 TRACE [ViewHandler,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UDP: sending msg to null, src=79e078919f2e-55593, headers are FLUSH: FLUSH[type=STOP_FLUSH,viewId=2], NAKACK2: [MSG, seqno=10], UDP: [channel_name=ExampleFederation] TRACE [ViewHandler,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UDP: looping back message [dst: , src: 79e078919f2e-55593 (3 headers), size=0 bytes] TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UDP: received [dst: , src: 79e078919f2e-55593 (3 headers), size=0 bytes], headers are FLUSH: FLUSH[type=STOP_FLUSH,viewId=2], NAKACK2: [MSG, seqno=10], UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.pbcast.NAKACK2: 79e078919f2e-55593: received 79e078919f2e-55593#10 TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.pbcast.NAKACK2: 79e078919f2e-55593: delivering 79e078919f2e-55593#10 DEBUG [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.pbcast.FLUSH: 79e078919f2e-55593: received STOP_FLUSH, unblocking FLUSH.down() and sending UNBLOCK up TRACE [TransferQueueBundler,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.TP$TransferQueueBundler: sending 1 msgs (68 bytes (0.11% of max_bundle_size) to 1 destination(s) TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UDP: received [dst: , src: 79e078919f2e-55593 (3 headers), size=0 bytes], headers are FLUSH: FLUSH[type=STOP_FLUSH,viewId=2], NAKACK2: [MSG, seqno=10], UDP: [channel_name=ExampleFederation] TRACE [Incoming] org.jgroups.protocols.UNICAST2: 79e078919f2e-55593 --> STABLE(79e078919f2e-55593: 6-6, conn_id=0) TRACE [Incoming] org.jgroups.protocols.UDP: sending msg to 79e078919f2e-55593, src=79e078919f2e-55593, headers are UNICAST2: STABLE, seqno=6, UDP: [channel_name=ExampleFederation] TRACE [Incoming] org.jgroups.protocols.UDP: looping back message [dst: 79e078919f2e-55593, src: 79e078919f2e-55593 (2 headers), size=0 bytes, flags=OOB] TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UDP: received [dst: 79e078919f2e-55593, src: 79e078919f2e-55593 (2 headers), size=0 bytes, flags=OOB], headers are UNICAST2: STABLE, seqno=6, UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UNICAST2: 79e078919f2e-55593 <-- STABLE(79e078919f2e-55593: 6-6, conn_id=0) TRACE [Incoming] org.jgroups.protocols.UDP: sending msg to null, src=79e078919f2e-55593, headers are MERGE3: INFO: view_id=[79e078919f2e-55593|2], logical_name=79e078919f2e-55593, physical_addr=[11.11.11.1:33013], UDP: [channel_name=ExampleFederation] TRACE [Incoming] org.jgroups.protocols.UDP: looping back message [dst: , src: 79e078919f2e-55593 (2 headers), size=0 bytes] TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UDP: received [dst: , src: 79e078919f2e-55593 (2 headers), size=0 bytes], headers are MERGE3: INFO: view_id=[79e078919f2e-55593|2], logical_name=79e078919f2e-55593, physical_addr=[11.11.11.1:33013], UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.MERGE3: 79e078919f2e-55593 <-- 79e078919f2e-55593: INFO: view_id=[79e078919f2e-55593|2], logical_name=79e078919f2e-55593, physical_addr=[11.11.11.1:33013], cached views: 1 TRACE [TransferQueueBundler,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.TP$TransferQueueBundler: sending 1 msgs (102 bytes (0.16% of max_bundle_size) to 1 destination(s) TRACE [Incoming,ExampleFederation,79e078919f2e-55593] org.jgroups.protocols.UDP: received [dst: , src: 79e078919f2e-55593 (2 headers), size=0 bytes], headers are MERGE3: INFO: view_id=[79e078919f2e-55593|2], logical_name=79e078919f2e-55593, physical_addr=[11.11.11.1:33013], UDP: [channel_name=ExampleFederation]

bergtwvd commented 9 years ago

FEDERATE YYY

DEBUG [main] portico.lrc: Creating new LRC DEBUG [main] portico.lrc: Portico version: 2.0.1 (build 0) DEBUG [main] portico.lrc: Interface: IEEE1516e WARN [main] portico.lrc: MOM support is currently unsupported in IEEE-1516e federations. TRACE [main] portico.lrc: Provided connection implementation is "org.portico.bindings.jgroups.JGroupsConnection" TRACE [main] portico.lrc: Trying to load connection class: org.portico.bindings.jgroups.JGroupsConnection TRACE [main] portico.lrc: ATTEMPT create IConnection, class= class org.portico.bindings.jgroups.JGroupsConnection TRACE [main] portico.lrc: SUCCESS created IConnection, class= class org.portico.bindings.jgroups.JGroupsConnection TRACE [main] portico.lrc: Applying modules using component keyword: lrc1516e TRACE [main] portico.lrc: STARTING Apply module [lrc13-callback] to LRC TRACE [main] portico.lrc: Applied [0/24] handlers TRACE [main] portico.lrc: STARTING Apply module [lrc-base] to LRC TRACE [main] portico.lrc: Applied [82/92] handlers TRACE [main] portico.lrc: STARTING Apply module [lrc1516-callback] to LRC TRACE [main] portico.lrc: Applied [0/11] handlers TRACE [main] portico.lrc: STARTING Apply module [lrc1516e-callback] to LRC TRACE [main] portico.lrc: Applied [24/24] handlers DEBUG [main] portico.lrc: Messaging framework configuration complete INFO [main] portico.lrc: LRC initialized (HLA version: IEEE1516e) INFO [main] portico.lrc: Opening LRC Connection INFO [main] portico.lrc.jgroups: jgroups connection is up and running DEBUG [main] portico.lrc.fom: Parsing FED file (format=ieee1516e): file:/usr/local/portico/portico-2.0.1/examples/java/ieee1516e/foms/RestaurantProcesses.xml DEBUG [main] portico.lrc.fom: Parsing FED file (format=ieee1516e): file:/usr/local/portico/portico-2.0.1/examples/java/ieee1516e/foms/RestaurantFood.xml DEBUG [main] portico.lrc.fom: Parsing FED file (format=ieee1516e): file:/usr/local/portico/portico-2.0.1/examples/java/ieee1516e/foms/RestaurantDrinks.xml DEBUG [main] portico.lrc: Standard MIM not present - adding it DEBUG [main] portico.lrc.fom: Parsing FED file (format=ieee1516e): jar:file:/usr/local/portico/portico-2.0.1/lib/portico.jar!/etc/ieee1516e/HLAstandardMIM.xml TRACE [main] portico.lrc.merger: Beginning merge of 4 FOM models TRACE [main] portico.lrc.merger: Merging [file:/usr/local/portico/portico-2.0.1/examples/java/ieee1516e/foms/RestaurantProcesses.xml] into combined FOM TRACE [main] portico.lrc.merger: -> Inserting class [HLAobjectRoot.Order] TRACE [main] portico.lrc.merger: -> Inserting class [HLAobjectRoot.Employee] TRACE [main] portico.lrc.merger: -> Inserting class [HLAobjectRoot.Employee.Greeter] TRACE [main] portico.lrc.merger: -> Inserting class [HLAobjectRoot.Employee.Waiter] TRACE [main] portico.lrc.merger: -> Inserting class [HLAobjectRoot.Employee.Cashier] TRACE [main] portico.lrc.merger: -> Inserting class [HLAobjectRoot.Employee.Cook] TRACE [main] portico.lrc.merger: -> Inserting class [HLAobjectRoot.Employee.Dishwasher] TRACE [main] portico.lrc.merger: -> Inserting class [HLAobjectRoot.Customer] TRACE [main] portico.lrc.merger: -> Inserting class [HLAinteractionRoot.CustomerTransactions] TRACE [main] portico.lrc.merger: -> Inserting class [HLAinteractionRoot.CustomerTransactions.CustomerSeated] TRACE [main] portico.lrc.merger: -> Inserting class [HLAinteractionRoot.CustomerTransactions.CustomerPays] TRACE [main] portico.lrc.merger: -> Inserting class [HLAinteractionRoot.CustomerTransactions.CustomerPays.ByCash] TRACE [main] portico.lrc.merger: -> Inserting class [HLAinteractionRoot.CustomerTransactions.CustomerPays.ByCreditCard] TRACE [main] portico.lrc.merger: -> Inserting class [HLAinteractionRoot.CustomerTransactions.FoodServed] TRACE [main] portico.lrc.merger: -> Inserting class [HLAinteractionRoot.CustomerTransactions.FoodServed.MainCourseServed] TRACE [main] portico.lrc.merger: -> Inserting class [HLAinteractionRoot.CustomerTransactions.FoodServed.DessertServed] TRACE [main] portico.lrc.merger: -> Inserting class [HLAinteractionRoot.CustomerTransactions.FoodServed.AppetizerServed] TRACE [main] portico.lrc.merger: -> Inserting class [HLAinteractionRoot.CustomerTransactions.FoodServed.DrinkServed] TRACE [main] portico.lrc.merger: -> Inserting class [HLAinteractionRoot.CustomerTransactions.CustomerLeaves] TRACE [main] portico.lrc.merger: -> Inserting class [HLAinteractionRoot.CustomerTransactions.OrderTaken] TRACE [main] portico.lrc.merger: -> Inserting class [HLAinteractionRoot.CustomerTransactions.OrderTaken.FromKidsMenu] TRACE [main] portico.lrc.merger: -> Inserting class [HLAinteractionRoot.CustomerTransactions.OrderTaken.FromAdultMeny] TRACE [main] portico.lrc.merger: Merging [file:/usr/local/portico/portico-2.0.1/examples/java/ieee1516e/foms/RestaurantFood.xml] into combined FOM TRACE [main] portico.lrc.merger: -> Inserting class [HLAobjectRoot.Food] TRACE [main] portico.lrc.merger: -> Inserting class [HLAobjectRoot.Food.Appetizers] TRACE [main] portico.lrc.merger: -> Inserting class [HLAobjectRoot.Food.Appetizers.Soup] TRACE [main] portico.lrc.merger: -> Inserting class [HLAobjectRoot.Food.Appetizers.Nachos] TRACE [main] portico.lrc.merger: -> Inserting class [HLAobjectRoot.Food.Drink] TRACE [main] portico.lrc.merger: -> Inserting class [HLAobjectRoot.Food.Dessert] TRACE [main] portico.lrc.merger: -> Inserting class [HLAobjectRoot.Food.Dessert.IceCream] TRACE [main] portico.lrc.merger: -> Inserting class [HLAobjectRoot.Food.Dessert.IceCream.Chocolate] TRACE [main] portico.lrc.merger: -> Inserting class [HLAobjectRoot.Food.Dessert.IceCream.Vanilla] TRACE [main] portico.lrc.merger: -> Inserting class [HLAobjectRoot.Food.Dessert.Cake] TRACE [main] portico.lrc.merger: -> Inserting class [HLAobjectRoot.Food.MainCourse] TRACE [main] portico.lrc.merger: -> Inserting class [HLAobjectRoot.Food.Entree] TRACE [main] portico.lrc.merger: -> Inserting class [HLAobjectRoot.Food.Entree.Seafood] TRACE [main] portico.lrc.merger: -> Inserting class [HLAobjectRoot.Food.Entree.Seafood.Lobster] TRACE [main] portico.lrc.merger: -> Inserting class [HLAobjectRoot.Food.Entree.Seafood.Shrimp] TRACE [main] portico.lrc.merger: -> Inserting class [HLAobjectRoot.Food.Entree.Seafood.Fish] TRACE [main] portico.lrc.merger: -> Inserting class [HLAobjectRoot.Food.Entree.Pasta] TRACE [main] portico.lrc.merger: -> Inserting class [HLAobjectRoot.Food.Entree.Beef] TRACE [main] portico.lrc.merger: -> Inserting class [HLAobjectRoot.Food.Entree.Chicken] TRACE [main] portico.lrc.merger: -> Inserting class [HLAobjectRoot.Food.SideDish] TRACE [main] portico.lrc.merger: -> Inserting class [HLAobjectRoot.Food.SideDish.Corn] TRACE [main] portico.lrc.merger: -> Inserting class [HLAobjectRoot.Food.SideDish.Broccoli] TRACE [main] portico.lrc.merger: -> Inserting class [HLAobjectRoot.Food.SideDish.BakedPotato] TRACE [main] portico.lrc.merger: Merging [file:/usr/local/portico/portico-2.0.1/examples/java/ieee1516e/foms/RestaurantDrinks.xml] into combined FOM TRACE [main] portico.lrc.merger: -> Inserting class [HLAobjectRoot.Food.Drink.Water] TRACE [main] portico.lrc.merger: -> Inserting class [HLAobjectRoot.Food.Drink.Coffee] TRACE [main] portico.lrc.merger: -> Inserting class [HLAobjectRoot.Food.Drink.Soda] DEBUG [main] portico.lrc: ATTEMPT Create federation execution [ExampleFederation] TRACE [main] portico.lrc.jgroups: ATTEMPT Connecting to channel [ExampleFederation] DEBUG [main] org.jgroups.conf.ClassConfigurator: Using jg-magic-map.xml as magic number file and jg-protocol-ids.xml for protocol IDs DEBUG [main] org.jgroups.stack.Configurator: set property UDP.diagnostics_addr to default value /224.0.75.75 DEBUG [main] org.jgroups.protocols.FRAG2: received CONFIG event: {bind_addr=/11.11.11.2} TRACE [main] org.jgroups.blocks.MessageDispatcher$ProtocolAdapter: setting local_addr (null) to 5f7daed5d167-29951 DEBUG [main] org.jgroups.protocols.FRAG2: received CONFIG event: {flush_supported=true} TRACE [main] org.jgroups.protocols.pbcast.STABLE: stable task started TRACE [main] org.jgroups.protocols.UNICAST2: 5f7daed5d167-29951: stable task started DEBUG [main] org.jgroups.protocols.UDP: sockets will use interface 11.11.11.2 WARN [main] org.jgroups.protocols.UDP: [JGRP00014] the send buffer of socket DatagramSocket was set to 640KB, but the OS only allocated 212.99KB. This might lead to performance problems. Please set your max send buffer in the OS correctly (e.g. net.core.wmem_max on Linux) WARN [main] org.jgroups.protocols.UDP: [JGRP00014] the receive buffer of socket DatagramSocket was set to 8MB, but the OS only allocated 212.99KB. This might lead to performance problems. Please set your max receive buffer in the OS correctly (e.g. net.core.rmem_max on Linux) WARN [main] org.jgroups.protocols.UDP: [JGRP00014] the send buffer of socket MulticastSocket was set to 640KB, but the OS only allocated 212.99KB. This might lead to performance problems. Please set your max send buffer in the OS correctly (e.g. net.core.wmem_max on Linux) WARN [main] org.jgroups.protocols.UDP: [JGRP00014] the receive buffer of socket MulticastSocket was set to 8MB, but the OS only allocated 212.99KB. This might lead to performance problems. Please set your max receive buffer in the OS correctly (e.g. net.core.rmem_max on Linux) DEBUG [main] org.jgroups.protocols.UDP: socket information: , mcast_addr=239.255.20.16:20913, bind_addr=/11.11.11.2, ttl=8 sock: bound to 11.11.11.2:33187, receive buffer size=212992, send buffer size=212992 mcast_sock: bound to 11.11.11.2:20913, send buffer size=212992, receive buffer size=212992 TRACE [main] org.jgroups.protocols.UDP: joined /224.0.75.75:7500 on ethwe TRACE [main] org.jgroups.protocols.UDP: joined /224.0.75.75:7500 on eth0 TRACE [main] org.jgroups.protocols.UDP: joined /224.0.75.75:7500 on lo TRACE [main] org.jgroups.protocols.UDP: sending msg to null, src=5f7daed5d167-29951, headers are PING: [PING: type=GET_MBRS_REQ, cluster=ExampleFederation, arg=5f7daed5d167-29951, view_id=, is_server=false, is_coord=false, logical_name=5f7daed5d167-29951, physical_addrs=11.11.11.2:33187], UDP: [channel_name=ExampleFederation] TRACE [main] org.jgroups.protocols.UDP: looping back message [dst: , src: 5f7daed5d167-29951 (2 headers), size=0 bytes, flags=OOB] TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.UDP: received [dst: , src: 5f7daed5d167-29951 (2 headers), size=0 bytes, flags=OOB], headers are PING: [PING: type=GET_MBRS_REQ, cluster=ExampleFederation, arg=5f7daed5d167-29951, view_id=, is_server=false, is_coord=false, logical_name=5f7daed5d167-29951, physical_addrs=11.11.11.2:33187], UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.UDP: received [dst: , src: 5f7daed5d167-29951 (2 headers), size=0 bytes, flags=OOB], headers are PING: [PING: type=GET_MBRS_REQ, cluster=ExampleFederation, arg=5f7daed5d167-29951, view_id=, is_server=false, is_coord=false, logical_name=5f7daed5d167-29951, physical_addrs=11.11.11.2:33187], UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.UDP: received [dst: 5f7daed5d167-29951, src: 1283d5c5-53a1-e32a-3256-ab0c220cc574 (2 headers), size=0 bytes, flags=OOB], headers are PING: [PING: type=GET_MBRS_RSP, arg=1283d5c5-53a1-e32a-3256-ab0c220cc574, view_id=[1283d5c5-53a1-e32a-3256-ab0c220cc574|0], is_server=true, is_coord=true, logical_name=79e078919f2e-55593, physical_addrs=11.11.11.1:33013], UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.PING: 5f7daed5d167-29951: received GET_MBRS_RSP from 79e078919f2e-55593: 79e078919f2e-55593, view_id=[79e078919f2e-55593|0], is_server=true, is_coord=true, logical_name=79e078919f2e-55593, physical_addrs=11.11.11.1:33013 TRACE [main] org.jgroups.protocols.PING: discovery took 11 ms: responses: 2 total (1 servers (1 coord), 1 clients) TRACE [Incoming] org.jgroups.protocols.PING: 5f7daed5d167-29951: received GET_MBRS_REQ from 5f7daed5d167-29951, sending staggered response [PING: type=GET_MBRS_RSP, arg=5f7daed5d167-29951, view_id=, is_server=false, is_coord=false, logical_name=5f7daed5d167-29951, physical_addrs=11.11.11.2:33187] TRACE [main] org.jgroups.protocols.pbcast.GMS: 5f7daed5d167-29951: initial_mbrs are 79e078919f2e-55593 DEBUG [main] org.jgroups.protocols.pbcast.GMS: election results: {79e078919f2e-55593=1} TRACE [Incoming] org.jgroups.protocols.UDP: sending msg to 5f7daed5d167-29951, src=5f7daed5d167-29951, headers are PING: [PING: type=GET_MBRS_RSP, arg=5f7daed5d167-29951, view_id=, is_server=false, is_coord=false, logical_name=5f7daed5d167-29951, physical_addrs=11.11.11.2:33187], UDP: [channel_name=ExampleFederation] TRACE [Incoming] org.jgroups.protocols.UDP: looping back message [dst: 5f7daed5d167-29951, src: 5f7daed5d167-29951 (2 headers), size=0 bytes, flags=OOB] TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.UDP: received [dst: 5f7daed5d167-29951, src: 5f7daed5d167-29951 (2 headers), size=0 bytes, flags=OOB], headers are PING: [PING: type=GET_MBRS_RSP, arg=5f7daed5d167-29951, view_id=, is_server=false, is_coord=false, logical_name=5f7daed5d167-29951, physical_addrs=11.11.11.2:33187], UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.PING: 5f7daed5d167-29951: received GET_MBRS_RSP from 5f7daed5d167-29951: 5f7daed5d167-29951, view_id=, is_server=false, is_coord=false, logical_name=5f7daed5d167-29951, physical_addrs=11.11.11.2:33187 DEBUG [main] org.jgroups.protocols.pbcast.GMS: sending JOIN(5f7daed5d167-29951) to 79e078919f2e-55593 TRACE [main] org.jgroups.protocols.UNICAST2: 5f7daed5d167-29951: created connection to 79e078919f2e-55593 (conn_id=0) TRACE [main] org.jgroups.protocols.UNICAST2: 5f7daed5d167-29951 --> DATA(79e078919f2e-55593: #1, conn_id=0, first) TRACE [main] org.jgroups.protocols.UDP: sending msg to 79e078919f2e-55593, src=5f7daed5d167-29951, headers are GMS: GmsHeader[JOIN_REQ_WITH_STATE_TRANSFER], UNICAST2: DATA, seqno=1, first, UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.UDP: received [dst: 5f7daed5d167-29951, src: 79e078919f2e-55593 (3 headers), size=0 bytes], headers are GMS: GmsHeader[JOIN_RSP]: join_rsp=view: [79e078919f2e-55593|1] [79e078919f2e-55593, 5f7daed5d167-29951], digest: 79e078919f2e-55593: [6 (6)], 5f7daed5d167-29951: [0 (0)], UNICAST2: DATA, seqno=1, conn_id=1, first, UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.UNICAST2: 5f7daed5d167-29951 <-- DATA(79e078919f2e-55593: #1, conn_id=1, first) TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.UNICAST2: 5f7daed5d167-29951: created receiver window for 79e078919f2e-55593 at seqno=#1 for conn-id=1 TRACE [main] org.jgroups.protocols.pbcast.GMS: 5f7daed5d167-29951: JOIN-RSP=[79e078919f2e-55593|1] [79e078919f2e-55593, 5f7daed5d167-29951] [size=2]

DEBUG [main] org.jgroups.protocols.pbcast.NAKACK2: [5f7daed5d167-29951 setDigest()] existing digest: [] new digest: 79e078919f2e-55593: [6 (6)], 5f7daed5d167-29951: [0 (0)] resulting digest: 79e078919f2e-55593: [6 (6)], 5f7daed5d167-29951: [0 (0)] DEBUG [main] org.jgroups.protocols.pbcast.GMS: 5f7daed5d167-29951: installing view [79e078919f2e-55593|1] [79e078919f2e-55593, 5f7daed5d167-29951] TRACE [main] org.jgroups.protocols.pbcast.STABLE: 5f7daed5d167-29951: resetting digest from NAKACK: 5f7daed5d167-29951: [0], 79e078919f2e-55593: [6] TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.UDP: received [dst: , src: 79e078919f2e-55593 (3 headers), size=0 bytes], headers are FLUSH: FLUSH[type=START_FLUSH,viewId=0,members=[79e078919f2e-55593]], NAKACK2: [MSG, seqno=6], UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.UDP: received [dst: , src: 79e078919f2e-55593 (2 headers), size=0 bytes], headers are MERGE3: INFO: view_id=[79e078919f2e-55593|0], logical_name=79e078919f2e-55593, physical_addr=[11.11.11.1:33013], UDP: [channel_name=ExampleFederation] DEBUG [main] org.jgroups.protocols.pbcast.STABLE: [ergonomics] setting max_bytes to 8MB (2 members) TRACE [main] org.jgroups.protocols.UFC: new membership: [79e078919f2e-55593, 5f7daed5d167-29951] TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.MERGE3: 5f7daed5d167-29951 <-- 79e078919f2e-55593: INFO: view_id=[79e078919f2e-55593|0], logical_name=79e078919f2e-55593, physical_addr=[11.11.11.1:33013], cached views: 1 TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.UDP: received [dst: , src: 79e078919f2e-55593 (3 headers), size=0 bytes], headers are GMS: GmsHeader[VIEW]: view=[79e078919f2e-55593|1] [79e078919f2e-55593, 5f7daed5d167-29951], NAKACK2: [MSG, seqno=7], UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.pbcast.NAKACK2: 5f7daed5d167-29951: received 79e078919f2e-55593#7 TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.pbcast.NAKACK2: 5f7daed5d167-29951: delivering 79e078919f2e-55593#7 TRACE [main] org.jgroups.protocols.MFC: new membership: [79e078919f2e-55593, 5f7daed5d167-29951] TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.UNICAST2: 5f7daed5d167-29951 --> DATA(79e078919f2e-55593: #2, conn_id=0) TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.UDP: sending msg to 79e078919f2e-55593, src=5f7daed5d167-29951, headers are GMS: GmsHeader[VIEW_ACK]: view=null, UNICAST2: DATA, seqno=2, UDP: [channel_name=ExampleFederation] DEBUG [main] org.jgroups.protocols.pbcast.FLUSH: 5f7daed5d167-29951: installing view [79e078919f2e-55593|1] [79e078919f2e-55593, 5f7daed5d167-29951] TRACE [main] org.jgroups.protocols.UNICAST2: 5f7daed5d167-29951 --> DATA(79e078919f2e-55593: #3, conn_id=0) TRACE [main] org.jgroups.protocols.UDP: sending msg to 79e078919f2e-55593, src=5f7daed5d167-29951, headers are GMS: GmsHeader[VIEW_ACK]: view=null, UNICAST2: DATA, seqno=3, UDP: [channel_name=ExampleFederation] DEBUG [main] org.jgroups.protocols.pbcast.STATE_TRANSFER: 5f7daed5d167-29951: asking 79e078919f2e-55593 for state TRACE [main] org.jgroups.protocols.UNICAST2: 5f7daed5d167-29951 --> DATA(79e078919f2e-55593: #4, conn_id=0) TRACE [main] org.jgroups.protocols.UDP: sending msg to 79e078919f2e-55593, src=5f7daed5d167-29951, headers are STATE_TRANSFER: type=STATE_REQ, UNICAST2: DATA, seqno=4, UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.UDP: received [dst: , src: 79e078919f2e-55593 (2 headers), size=0 bytes, flags=OOB], headers are FD_ALL: heartbeat, UDP: [channel_name=ExampleFederation] TRACE [Incoming] org.jgroups.protocols.UDP: sending msg to null, src=5f7daed5d167-29951, headers are FD_ALL: heartbeat, UDP: [channel_name=ExampleFederation] TRACE [Incoming] org.jgroups.protocols.UDP: looping back message [dst: , src: 5f7daed5d167-29951 (2 headers), size=0 bytes, flags=OOB] TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.UDP: received [dst: , src: 5f7daed5d167-29951 (2 headers), size=0 bytes, flags=OOB], headers are FD_ALL: heartbeat, UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.UDP: received [dst: , src: 5f7daed5d167-29951 (2 headers), size=0 bytes, flags=OOB], headers are FD_ALL: heartbeat, UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.UDP: received [dst: , src: 79e078919f2e-55593 (2 headers), size=0 bytes, flags=OOB|NO_RELIABILITY], headers are STABLE: [STABLE_GOSSIP]: digest is 79e078919f2e-55593: [7 (7)], 5f7daed5d167-29951: [0 (0)], UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.pbcast.STABLE: 5f7daed5d167-29951: handling digest from 79e078919f2e-55593 (0 votes): mine: 5f7daed5d167-29951: [0], 79e078919f2e-55593: [6] other: 5f7daed5d167-29951: [0], 79e078919f2e-55593: [7] result: 5f7daed5d167-29951: [0], 79e078919f2e-55593: [6]

DEBUG [main] org.jgroups.protocols.pbcast.FLUSH: 5f7daed5d167-29951: received RESUME, sending STOP_FLUSH to all TRACE [main] org.jgroups.protocols.pbcast.NAKACK2: 5f7daed5d167-29951 sending 5f7daed5d167-29951#1 TRACE [main] org.jgroups.protocols.UDP: sending msg to null, src=5f7daed5d167-29951, headers are FLUSH: FLUSH[type=STOP_FLUSH,viewId=1], NAKACK2: [MSG, seqno=1], UDP: [channel_name=ExampleFederation] TRACE [main] org.jgroups.protocols.UDP: looping back message [dst: , src: 5f7daed5d167-29951 (3 headers), size=0 bytes] DEBUG [main] portico.lrc.jgroups: SUCCESS Connected to channel [ExampleFederation] DEBUG [main] portico.lrc.jgroups: REQUEST createFederation: name=ExampleFederation TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.UDP: received [dst: , src: 5f7daed5d167-29951 (3 headers), size=0 bytes], headers are FLUSH: FLUSH[type=STOP_FLUSH,viewId=1], NAKACK2: [MSG, seqno=1], UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.pbcast.NAKACK2: 5f7daed5d167-29951: received 5f7daed5d167-29951#1 TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.pbcast.NAKACK2: 5f7daed5d167-29951: delivering 5f7daed5d167-29951#1 DEBUG [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.pbcast.FLUSH: 5f7daed5d167-29951: received STOP_FLUSH, unblocking FLUSH.down() and sending UNBLOCK up TRACE [TransferQueueBundler,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.TP$TransferQueueBundler: sending 1 msgs (68 bytes (0.11% of max_bundle_size) to 1 destination(s) TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.UDP: received [dst: , src: 5f7daed5d167-29951 (3 headers), size=0 bytes], headers are FLUSH: FLUSH[type=STOP_FLUSH,viewId=1], NAKACK2: [MSG, seqno=1], UDP: [channel_name=ExampleFederation] TRACE [main] org.jgroups.protocols.RSVP: 5f7daed5d167-29951: REQ --> null TRACE [main] org.jgroups.protocols.pbcast.NAKACK2: 5f7daed5d167-29951 sending 5f7daed5d167-29951#2 TRACE [main] org.jgroups.protocols.UDP: sending msg to null, src=5f7daed5d167-29951, headers are 7777: CreateFederation, RSVP: REQ(0), NAKACK2: [MSG, seqno=2], UDP: [channel_name=ExampleFederation] TRACE [main] org.jgroups.protocols.UDP: looping back message [dst: , src: 5f7daed5d167-29951 (4 headers), size=24647 bytes, flags=OOB|DONT_BUNDLE|NO_FC|RSVP] TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.UDP: received [dst: , src: 5f7daed5d167-29951 (4 headers), size=24647 bytes, flags=OOB|DONT_BUNDLE|NO_FC|RSVP], headers are 7777: CreateFederation, RSVP: REQ(0), NAKACK2: [MSG, seqno=2], UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.pbcast.NAKACK2: 5f7daed5d167-29951: received 5f7daed5d167-29951#2 TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.pbcast.NAKACK2: 5f7daed5d167-29951: delivering 5f7daed5d167-29951#2 TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.UDP: received [dst: , src: 5f7daed5d167-29951 (4 headers), size=24647 bytes, flags=OOB|DONT_BUNDLE|NO_FC|RSVP], headers are 7777: CreateFederation, RSVP: REQ(0), NAKACK2: [MSG, seqno=2], UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.RSVP: 5f7daed5d167-29951: REQ <-- 5f7daed5d167-29951 TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] portico.lrc.jgroups: (incoming) asynchronous, channel=ExampleFederation, size=24647, source=5f7daed5d167-29951 DEBUG [Incoming,ExampleFederation,5f7daed5d167-29951] portico.lrc.jgroups: Received federation creation notification: federation=ExampleFederation, fomSize=24647b, source=5f7daed5d167-29951 INFO [Incoming,ExampleFederation,5f7daed5d167-29951] portico.lrc.jgroups: Federation [ExampleFederation] has been created TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.RSVP: 5f7daed5d167-29951: RSP --> 5f7daed5d167-29951 TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.UNICAST2: 5f7daed5d167-29951: created connection to 5f7daed5d167-29951 (conn_id=1) TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.UNICAST2: 5f7daed5d167-29951 --> DATA(5f7daed5d167-29951: #1, conn_id=1, first) TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.UDP: sending msg to 5f7daed5d167-29951, src=5f7daed5d167-29951, headers are RSVP: RSP(0), UNICAST2: DATA, seqno=1, conn_id=1, first, UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.UDP: looping back message [dst: 5f7daed5d167-29951, src: 5f7daed5d167-29951 (3 headers), size=0 bytes, flags=OOB|RSVP] TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.UDP: received [dst: 5f7daed5d167-29951, src: 5f7daed5d167-29951 (3 headers), size=0 bytes, flags=OOB|RSVP], headers are RSVP: RSP(0), UNICAST2: DATA, seqno=1, conn_id=1, first, UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.UNICAST2: 5f7daed5d167-29951 <-- DATA(5f7daed5d167-29951: #1, conn_id=1, first) TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.UNICAST2: 5f7daed5d167-29951: created receiver window for 5f7daed5d167-29951 at seqno=#1 for conn-id=1 TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.RSVP: 5f7daed5d167-29951: RSP <-- 5f7daed5d167-29951 TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.UDP: received [dst: , src: 79e078919f2e-55593 (2 headers), size=0 bytes, flags=OOB], headers are FD_ALL: heartbeat, UDP: [channel_name=ExampleFederation] TRACE [Incoming] org.jgroups.protocols.UDP: sending msg to null, src=5f7daed5d167-29951, headers are FD_ALL: heartbeat, UDP: [channel_name=ExampleFederation] TRACE [Incoming] org.jgroups.protocols.UDP: looping back message [dst: , src: 5f7daed5d167-29951 (2 headers), size=0 bytes, flags=OOB] TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.UDP: received [dst: , src: 5f7daed5d167-29951 (2 headers), size=0 bytes, flags=OOB], headers are FD_ALL: heartbeat, UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.UDP: received [dst: , src: 5f7daed5d167-29951 (2 headers), size=0 bytes, flags=OOB], headers are FD_ALL: heartbeat, UDP: [channel_name=ExampleFederation] TRACE [Incoming] org.jgroups.protocols.RSVP: 5f7daed5d167-29951: REQ-ONLY --> null TRACE [Incoming] org.jgroups.protocols.pbcast.NAKACK2: 5f7daed5d167-29951 sending 5f7daed5d167-29951#3 TRACE [Incoming] org.jgroups.protocols.UDP: sending msg to null, src=5f7daed5d167-29951, headers are RSVP: REQ-ONLY(0), NAKACK2: [MSG, seqno=3], UDP: [channel_name=ExampleFederation] TRACE [Incoming] org.jgroups.protocols.UDP: looping back message [dst: , src: 5f7daed5d167-29951 (3 headers), size=0 bytes, flags=RSVP] TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.UDP: received [dst: , src: 5f7daed5d167-29951 (3 headers), size=0 bytes, flags=RSVP], headers are RSVP: REQ-ONLY(0), NAKACK2: [MSG, seqno=3], UDP: [channel_name=ExampleFederation] TRACE [TransferQueueBundler,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.TP$TransferQueueBundler: sending 1 msgs (59 bytes (0.09% of max_bundle_size) to 1 destination(s) TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.pbcast.NAKACK2: 5f7daed5d167-29951: received 5f7daed5d167-29951#3 TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.pbcast.NAKACK2: 5f7daed5d167-29951: delivering 5f7daed5d167-29951#3 TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.RSVP: 5f7daed5d167-29951: REQ-ONLY <-- 5f7daed5d167-29951 TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.UDP: received [dst: , src: 5f7daed5d167-29951 (3 headers), size=0 bytes, flags=RSVP], headers are RSVP: REQ-ONLY(0), NAKACK2: [MSG, seqno=3], UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.UDP: received [dst: 5f7daed5d167-29951, src: 79e078919f2e-55593 (2 headers), size=12 bytes, flags=OOB], headers are NAKACK2: [XMIT_REQ, sender=5f7daed5d167-29951], UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.pbcast.NAKACK2: 5f7daed5d167-29951: received xmit request from 79e078919f2e-55593 for 5f7daed5d167-29951[2] TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.pbcast.NAKACK2: 5f7daed5d167-29951: resending 5f7daed5d167-29951::2 TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.UDP: sending msg to null, src=5f7daed5d167-29951, headers are 7777: CreateFederation, RSVP: REQ(0), NAKACK2: [MSG, seqno=2], UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.UDP: looping back message [dst: , src: 5f7daed5d167-29951 (4 headers), size=24647 bytes, flags=OOB|DONT_BUNDLE|NO_FC|RSVP, transient_flags=OOB_DELIVERED] TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.UDP: received [dst: , src: 5f7daed5d167-29951 (4 headers), size=24647 bytes, flags=OOB|DONT_BUNDLE|NO_FC|RSVP, transient_flags=OOB_DELIVERED], headers are 7777: CreateFederation, RSVP: REQ(0), NAKACK2: [MSG, seqno=2], UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.pbcast.NAKACK2: 5f7daed5d167-29951: received 5f7daed5d167-29951#2 TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.UDP: received [dst: , src: 5f7daed5d167-29951 (4 headers), size=24647 bytes, flags=OOB|DONT_BUNDLE|NO_FC|RSVP], headers are 7777: CreateFederation, RSVP: REQ(0), NAKACK2: [MSG, seqno=2], UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.UDP: received [dst: , src: 79e078919f2e-55593 (2 headers), size=0 bytes, flags=OOB], headers are FD_ALL: heartbeat, UDP: [channel_name=ExampleFederation] TRACE [Incoming] org.jgroups.protocols.UDP: sending msg to null, src=5f7daed5d167-29951, headers are FD_ALL: heartbeat, UDP: [channel_name=ExampleFederation] TRACE [Incoming] org.jgroups.protocols.UDP: looping back message [dst: , src: 5f7daed5d167-29951 (2 headers), size=0 bytes, flags=OOB] TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.UDP: received [dst: , src: 5f7daed5d167-29951 (2 headers), size=0 bytes, flags=OOB], headers are FD_ALL: heartbeat, UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.UDP: received [dst: , src: 5f7daed5d167-29951 (2 headers), size=0 bytes, flags=OOB], headers are FD_ALL: heartbeat, UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.UDP: received [dst: 5f7daed5d167-29951, src: 79e078919f2e-55593 (2 headers), size=12 bytes, flags=OOB], headers are NAKACK2: [XMIT_REQ, sender=5f7daed5d167-29951], UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.pbcast.NAKACK2: 5f7daed5d167-29951: received xmit request from 79e078919f2e-55593 for 5f7daed5d167-29951[2] TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.pbcast.NAKACK2: 5f7daed5d167-29951: resending 5f7daed5d167-29951::2 TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.UDP: sending msg to null, src=5f7daed5d167-29951, headers are 7777: CreateFederation, RSVP: REQ(0), NAKACK2: [MSG, seqno=2], UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.UDP: looping back message [dst: , src: 5f7daed5d167-29951 (4 headers), size=24647 bytes, flags=OOB|DONT_BUNDLE|NO_FC|RSVP, transient_flags=OOB_DELIVERED] TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.UDP: received [dst: , src: 5f7daed5d167-29951 (4 headers), size=24647 bytes, flags=OOB|DONT_BUNDLE|NO_FC|RSVP, transient_flags=OOB_DELIVERED], headers are 7777: CreateFederation, RSVP: REQ(0), NAKACK2: [MSG, seqno=2], UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.pbcast.NAKACK2: 5f7daed5d167-29951: received 5f7daed5d167-29951#2 TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.UDP: received [dst: , src: 5f7daed5d167-29951 (4 headers), size=24647 bytes, flags=OOB|DONT_BUNDLE|NO_FC|RSVP], headers are 7777: CreateFederation, RSVP: REQ(0), NAKACK2: [MSG, seqno=2], UDP: [channel_name=ExampleFederation] TRACE [Incoming] org.jgroups.protocols.RSVP: 5f7daed5d167-29951: REQ-ONLY --> null TRACE [Incoming] org.jgroups.protocols.pbcast.NAKACK2: 5f7daed5d167-29951 sending 5f7daed5d167-29951#4 TRACE [Incoming] org.jgroups.protocols.UDP: sending msg to null, src=5f7daed5d167-29951, headers are RSVP: REQ-ONLY(0), NAKACK2: [MSG, seqno=4], UDP: [channel_name=ExampleFederation] TRACE [Incoming] org.jgroups.protocols.UDP: looping back message [dst: , src: 5f7daed5d167-29951 (3 headers), size=0 bytes, flags=RSVP] TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.UDP: received [dst: , src: 5f7daed5d167-29951 (3 headers), size=0 bytes, flags=RSVP], headers are RSVP: REQ-ONLY(0), NAKACK2: [MSG, seqno=4], UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.pbcast.NAKACK2: 5f7daed5d167-29951: received 5f7daed5d167-29951#4 TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.pbcast.NAKACK2: 5f7daed5d167-29951: delivering 5f7daed5d167-29951#4 TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.RSVP: 5f7daed5d167-29951: REQ-ONLY <-- 5f7daed5d167-29951 TRACE [TransferQueueBundler,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.TP$TransferQueueBundler: sending 1 msgs (59 bytes (0.09% of max_bundle_size) to 1 destination(s) TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.UDP: received [dst: , src: 5f7daed5d167-29951 (3 headers), size=0 bytes, flags=RSVP], headers are RSVP: REQ-ONLY(0), NAKACK2: [MSG, seqno=4], UDP: [channel_name=ExampleFederation] TRACE [Incoming] org.jgroups.protocols.pbcast.STABLE: 5f7daed5d167-29951: setting latest_local_digest from NAKACK: 5f7daed5d167-29951: [4], 79e078919f2e-55593: [7] TRACE [Incoming] org.jgroups.protocols.pbcast.STABLE: 5f7daed5d167-29951: sending stable msg 5f7daed5d167-29951: [4], 79e078919f2e-55593: [7] TRACE [Incoming] org.jgroups.protocols.UDP: sending msg to null, src=5f7daed5d167-29951, headers are STABLE: [STABLE_GOSSIP]: digest is 79e078919f2e-55593: [7 (7)], 5f7daed5d167-29951: [4 (4)], UDP: [channel_name=ExampleFederation] TRACE [Incoming] org.jgroups.protocols.UDP: looping back message [dst: , src: 5f7daed5d167-29951 (2 headers), size=0 bytes, flags=OOB|NO_RELIABILITY] TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.UDP: received [dst: , src: 5f7daed5d167-29951 (2 headers), size=0 bytes, flags=OOB|NO_RELIABILITY], headers are STABLE: [STABLE_GOSSIP]: digest is 79e078919f2e-55593: [7 (7)], 5f7daed5d167-29951: [4 (4)], UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.pbcast.STABLE: 5f7daed5d167-29951: handling digest from 5f7daed5d167-29951 (1 votes): mine: 5f7daed5d167-29951: [0], 79e078919f2e-55593: [6] other: 5f7daed5d167-29951: [4], 79e078919f2e-55593: [7] result: 5f7daed5d167-29951: [0], 79e078919f2e-55593: [6]

TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.pbcast.STABLE: 5f7daed5d167-29951: sending stability msg (in 1403 ms) 5f7daed5d167-29951: [0], 79e078919f2e-55593: [6] TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.UDP: received [dst: , src: 5f7daed5d167-29951 (2 headers), size=0 bytes, flags=OOB|NO_RELIABILITY], headers are STABLE: [STABLE_GOSSIP]: digest is 79e078919f2e-55593: [7 (7)], 5f7daed5d167-29951: [4 (4)], UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.UDP: received [dst: 5f7daed5d167-29951, src: 79e078919f2e-55593 (2 headers), size=12 bytes, flags=OOB], headers are NAKACK2: [XMIT_REQ, sender=5f7daed5d167-29951], UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.pbcast.NAKACK2: 5f7daed5d167-29951: received xmit request from 79e078919f2e-55593 for 5f7daed5d167-29951[2] TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.pbcast.NAKACK2: 5f7daed5d167-29951: resending 5f7daed5d167-29951::2 TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.UDP: sending msg to null, src=5f7daed5d167-29951, headers are 7777: CreateFederation, RSVP: REQ(0), NAKACK2: [MSG, seqno=2], UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.UDP: looping back message [dst: , src: 5f7daed5d167-29951 (4 headers), size=24647 bytes, flags=OOB|DONT_BUNDLE|NO_FC|RSVP, transient_flags=OOB_DELIVERED] TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.UDP: received [dst: , src: 5f7daed5d167-29951 (4 headers), size=24647 bytes, flags=OOB|DONT_BUNDLE|NO_FC|RSVP, transient_flags=OOB_DELIVERED], headers are 7777: CreateFederation, RSVP: REQ(0), NAKACK2: [MSG, seqno=2], UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.pbcast.NAKACK2: 5f7daed5d167-29951: received 5f7daed5d167-29951#2 TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.UDP: received [dst: , src: 5f7daed5d167-29951 (4 headers), size=24647 bytes, flags=OOB|DONT_BUNDLE|NO_FC|RSVP], headers are 7777: CreateFederation, RSVP: REQ(0), NAKACK2: [MSG, seqno=2], UDP: [channel_name=ExampleFederation] TRACE [Incoming] org.jgroups.protocols.RSVP: 5f7daed5d167-29951: REQ-ONLY --> null TRACE [Incoming] org.jgroups.protocols.pbcast.NAKACK2: 5f7daed5d167-29951 sending 5f7daed5d167-29951#5 TRACE [Incoming] org.jgroups.protocols.UDP: sending msg to null, src=5f7daed5d167-29951, headers are RSVP: REQ-ONLY(0), NAKACK2: [MSG, seqno=5], UDP: [channel_name=ExampleFederation] TRACE [Incoming] org.jgroups.protocols.UDP: looping back message [dst: , src: 5f7daed5d167-29951 (3 headers), size=0 bytes, flags=RSVP] TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.UDP: received [dst: , src: 5f7daed5d167-29951 (3 headers), size=0 bytes, flags=RSVP], headers are RSVP: REQ-ONLY(0), NAKACK2: [MSG, seqno=5], UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.pbcast.NAKACK2: 5f7daed5d167-29951: received 5f7daed5d167-29951#5 TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.pbcast.NAKACK2: 5f7daed5d167-29951: delivering 5f7daed5d167-29951#5 TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.RSVP: 5f7daed5d167-29951: REQ-ONLY <-- 5f7daed5d167-29951 TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.UDP: received [dst: 5f7daed5d167-29951, src: 79e078919f2e-55593 (2 headers), size=12 bytes, flags=OOB], headers are NAKACK2: [XMIT_REQ, sender=5f7daed5d167-29951], UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.pbcast.NAKACK2: 5f7daed5d167-29951: received xmit request from 79e078919f2e-55593 for 5f7daed5d167-29951[2] TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.pbcast.NAKACK2: 5f7daed5d167-29951: resending 5f7daed5d167-29951::2 TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.UDP: sending msg to null, src=5f7daed5d167-29951, headers are 7777: CreateFederation, RSVP: REQ(0), NAKACK2: [MSG, seqno=2], UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.UDP: looping back message [dst: , src: 5f7daed5d167-29951 (4 headers), size=24647 bytes, flags=OOB|DONT_BUNDLE|NO_FC|RSVP, transient_flags=OOB_DELIVERED] TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.UDP: received [dst: , src: 5f7daed5d167-29951 (4 headers), size=24647 bytes, flags=OOB|DONT_BUNDLE|NO_FC|RSVP, transient_flags=OOB_DELIVERED], headers are 7777: CreateFederation, RSVP: REQ(0), NAKACK2: [MSG, seqno=2], UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.pbcast.NAKACK2: 5f7daed5d167-29951: received 5f7daed5d167-29951#2 TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.UDP: received [dst: , src: 5f7daed5d167-29951 (4 headers), size=24647 bytes, flags=OOB|DONT_BUNDLE|NO_FC|RSVP], headers are 7777: CreateFederation, RSVP: REQ(0), NAKACK2: [MSG, seqno=2], UDP: [channel_name=ExampleFederation] TRACE [TransferQueueBundler,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.TP$TransferQueueBundler: sending 1 msgs (59 bytes (0.09% of max_bundle_size) to 1 destination(s) TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.UDP: received [dst: , src: 5f7daed5d167-29951 (3 headers), size=0 bytes, flags=RSVP], headers are RSVP: REQ-ONLY(0), NAKACK2: [MSG, seqno=5], UDP: [channel_name=ExampleFederation] TRACE [Incoming] org.jgroups.protocols.pbcast.STABLE: 5f7daed5d167-29951: sending stability msg 5f7daed5d167-29951: [0], 79e078919f2e-55593: [6] TRACE [Incoming] org.jgroups.protocols.UDP: sending msg to null, src=5f7daed5d167-29951, headers are STABLE: [STABILITY]: digest is 79e078919f2e-55593: [6 (7)], 5f7daed5d167-29951: [0 (4)], UDP: [channel_name=ExampleFederation] TRACE [Incoming] org.jgroups.protocols.UDP: looping back message [dst: , src: 5f7daed5d167-29951 (2 headers), size=0 bytes, flags=OOB|NO_RELIABILITY] TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.UDP: received [dst: , src: 5f7daed5d167-29951 (2 headers), size=0 bytes, flags=OOB|NO_RELIABILITY], headers are STABLE: [STABILITY]: digest is 79e078919f2e-55593: [6 (7)], 5f7daed5d167-29951: [0 (4)], UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.pbcast.STABLE: 5f7daed5d167-29951: received stability msg from 5f7daed5d167-29951: 5f7daed5d167-29951: [0], 79e078919f2e-55593: [6] TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.pbcast.STABLE: 5f7daed5d167-29951: resetting digest from NAKACK: 5f7daed5d167-29951: [5], 79e078919f2e-55593: [7] TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.pbcast.NAKACK2: 5f7daed5d167-29951: received stable digest 79e078919f2e-55593: [6 (7)], 5f7daed5d167-29951: [0 (4)] TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.UDP: received [dst: , src: 5f7daed5d167-29951 (2 headers), size=0 bytes, flags=OOB|NO_RELIABILITY], headers are STABLE: [STABILITY]: digest is 79e078919f2e-55593: [6 (7)], 5f7daed5d167-29951: [0 (4)], UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.pbcast.NAKACK2: 5f7daed5d167-29951: deleting msgs <= 6 from 79e078919f2e-55593 TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.pbcast.NAKACK2: 5f7daed5d167-29951: deleting msgs <= 0 from 5f7daed5d167-29951 TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.UDP: received [dst: , src: 79e078919f2e-55593 (2 headers), size=0 bytes, flags=OOB], headers are FD_ALL: heartbeat, UDP: [channel_name=ExampleFederation] TRACE [Incoming] org.jgroups.protocols.UDP: sending msg to null, src=5f7daed5d167-29951, headers are FD_ALL: heartbeat, UDP: [channel_name=ExampleFederation] TRACE [Incoming] org.jgroups.protocols.UDP: looping back message [dst: , src: 5f7daed5d167-29951 (2 headers), size=0 bytes, flags=OOB] TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.UDP: received [dst: , src: 5f7daed5d167-29951 (2 headers), size=0 bytes, flags=OOB], headers are FD_ALL: heartbeat, UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.UDP: received [dst: , src: 5f7daed5d167-29951 (2 headers), size=0 bytes, flags=OOB], headers are FD_ALL: heartbeat, UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.UDP: received [dst: 5f7daed5d167-29951, src: 79e078919f2e-55593 (2 headers), size=12 bytes, flags=OOB], headers are NAKACK2: [XMIT_REQ, sender=5f7daed5d167-29951], UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.pbcast.NAKACK2: 5f7daed5d167-29951: received xmit request from 79e078919f2e-55593 for 5f7daed5d167-29951[2] TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.pbcast.NAKACK2: 5f7daed5d167-29951: resending 5f7daed5d167-29951::2 TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.UDP: sending msg to null, src=5f7daed5d167-29951, headers are 7777: CreateFederation, RSVP: REQ(0), NAKACK2: [MSG, seqno=2], UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.UDP: looping back message [dst: , src: 5f7daed5d167-29951 (4 headers), size=24647 bytes, flags=OOB|DONT_BUNDLE|NO_FC|RSVP, transient_flags=OOB_DELIVERED] TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.UDP: received [dst: , src: 5f7daed5d167-29951 (4 headers), size=24647 bytes, flags=OOB|DONT_BUNDLE|NO_FC|RSVP, transient_flags=OOB_DELIVERED], headers are 7777: CreateFederation, RSVP: REQ(0), NAKACK2: [MSG, seqno=2], UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.pbcast.NAKACK2: 5f7daed5d167-29951: received 5f7daed5d167-29951#2 TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.UDP: received [dst: , src: 5f7daed5d167-29951 (4 headers), size=24647 bytes, flags=OOB|DONT_BUNDLE|NO_FC|RSVP], headers are 7777: CreateFederation, RSVP: REQ(0), NAKACK2: [MSG, seqno=2], UDP: [channel_name=ExampleFederation] TRACE [Incoming] org.jgroups.protocols.RSVP: 5f7daed5d167-29951: REQ-ONLY --> null TRACE [Incoming] org.jgroups.protocols.pbcast.NAKACK2: 5f7daed5d167-29951 sending 5f7daed5d167-29951#6 TRACE [Incoming] org.jgroups.protocols.UDP: sending msg to null, src=5f7daed5d167-29951, headers are RSVP: REQ-ONLY(0), NAKACK2: [MSG, seqno=6], UDP: [channel_name=ExampleFederation] TRACE [Incoming] org.jgroups.protocols.UDP: looping back message [dst: , src: 5f7daed5d167-29951 (3 headers), size=0 bytes, flags=RSVP] TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.UDP: received [dst: , src: 5f7daed5d167-29951 (3 headers), size=0 bytes, flags=RSVP], headers are RSVP: REQ-ONLY(0), NAKACK2: [MSG, seqno=6], UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.pbcast.NAKACK2: 5f7daed5d167-29951: received 5f7daed5d167-29951#6 TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.pbcast.NAKACK2: 5f7daed5d167-29951: delivering 5f7daed5d167-29951#6 TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.RSVP: 5f7daed5d167-29951: REQ-ONLY <-- 5f7daed5d167-29951 TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.UDP: received [dst: 5f7daed5d167-29951, src: 79e078919f2e-55593 (2 headers), size=12 bytes, flags=OOB], headers are NAKACK2: [XMIT_REQ, sender=5f7daed5d167-29951], UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.pbcast.NAKACK2: 5f7daed5d167-29951: received xmit request from 79e078919f2e-55593 for 5f7daed5d167-29951[2] TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.pbcast.NAKACK2: 5f7daed5d167-29951: resending 5f7daed5d167-29951::2 TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.UDP: sending msg to null, src=5f7daed5d167-29951, headers are 7777: CreateFederation, RSVP: REQ(0), NAKACK2: [MSG, seqno=2], UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.UDP: looping back message [dst: , src: 5f7daed5d167-29951 (4 headers), size=24647 bytes, flags=OOB|DONT_BUNDLE|NO_FC|RSVP, transient_flags=OOB_DELIVERED] TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.UDP: received [dst: , src: 5f7daed5d167-29951 (4 headers), size=24647 bytes, flags=OOB|DONT_BUNDLE|NO_FC|RSVP, transient_flags=OOB_DELIVERED], headers are 7777: CreateFederation, RSVP: REQ(0), NAKACK2: [MSG, seqno=2], UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.pbcast.NAKACK2: 5f7daed5d167-29951: received 5f7daed5d167-29951#2 TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.UDP: received [dst: , src: 5f7daed5d167-29951 (4 headers), size=24647 bytes, flags=OOB|DONT_BUNDLE|NO_FC|RSVP], headers are 7777: CreateFederation, RSVP: REQ(0), NAKACK2: [MSG, seqno=2], UDP: [channel_name=ExampleFederation] TRACE [TransferQueueBundler,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.TP$TransferQueueBundler: sending 1 msgs (59 bytes (0.09% of max_bundle_size) to 1 destination(s) TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.UDP: received [dst: , src: 5f7daed5d167-29951 (3 headers), size=0 bytes, flags=RSVP], headers are RSVP: REQ-ONLY(0), NAKACK2: [MSG, seqno=6], UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.UDP: received [dst: 5f7daed5d167-29951, src: 79e078919f2e-55593 (2 headers), size=12 bytes, flags=OOB], headers are NAKACK2: [XMIT_REQ, sender=5f7daed5d167-29951], UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.pbcast.NAKACK2: 5f7daed5d167-29951: received xmit request from 79e078919f2e-55593 for 5f7daed5d167-29951[2] TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.pbcast.NAKACK2: 5f7daed5d167-29951: resending 5f7daed5d167-29951::2 TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.UDP: sending msg to null, src=5f7daed5d167-29951, headers are 7777: CreateFederation, RSVP: REQ(0), NAKACK2: [MSG, seqno=2], UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.UDP: looping back message [dst: , src: 5f7daed5d167-29951 (4 headers), size=24647 bytes, flags=OOB|DONT_BUNDLE|NO_FC|RSVP, transient_flags=OOB_DELIVERED] TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.UDP: received [dst: , src: 5f7daed5d167-29951 (4 headers), size=24647 bytes, flags=OOB|DONT_BUNDLE|NO_FC|RSVP, transient_flags=OOB_DELIVERED], headers are 7777: CreateFederation, RSVP: REQ(0), NAKACK2: [MSG, seqno=2], UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.UDP: received [dst: , src: 5f7daed5d167-29951 (4 headers), size=24647 bytes, flags=OOB|DONT_BUNDLE|NO_FC|RSVP], headers are 7777: CreateFederation, RSVP: REQ(0), NAKACK2: [MSG, seqno=2], UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.pbcast.NAKACK2: 5f7daed5d167-29951: received 5f7daed5d167-29951#2 TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.UDP: received [dst: , src: 79e078919f2e-55593 (2 headers), size=0 bytes, flags=OOB], headers are FD_ALL: heartbeat, UDP: [channel_name=ExampleFederation] TRACE [Incoming] org.jgroups.protocols.UDP: sending msg to null, src=5f7daed5d167-29951, headers are FD_ALL: heartbeat, UDP: [channel_name=ExampleFederation] TRACE [Incoming] org.jgroups.protocols.UDP: looping back message [dst: , src: 5f7daed5d167-29951 (2 headers), size=0 bytes, flags=OOB] TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.UDP: received [dst: , src: 5f7daed5d167-29951 (2 headers), size=0 bytes, flags=OOB], headers are FD_ALL: heartbeat, UDP: [channel_name=ExampleFederation] TRACE [Incoming,ExampleFederation,5f7daed5d167-29951] org.jgroups.protocols.UDP: received [dst: , src: 5f7daed5d167-29951 (2 headers), size=0 bytes, flags=OOB], headers are FD_ALL: heartbeat, UDP: [channel_name=ExampleFederation] ERROR [main] portico.lrc: org.jgroups.TimeoutException: TimeoutException

bergtwvd commented 9 years ago

STACKTRACE OF YYY at the end:

ERROR [main] portico.lrc: org.jgroups.TimeoutException: TimeoutException hla.rti1516e.exceptions.RTIinternalError: Unknown exception received from RTI (class org.jgroups.TimeoutException) for createFederationExecution(): TimeoutException at org.portico.impl.hla1516e.Rti1516eAmbassador.logException(Rti1516eAmbassador.java:5588) at org.portico.impl.hla1516e.Rti1516eAmbassador.createFederationExecution(Rti1516eAmbassador.java:349) at ieee1516e.ExampleFederate.runFederate(ExampleFederate.java:206) at ieee1516e.ExampleFederate.main(ExampleFederate.java:560) Caused by: org.jgroups.TimeoutException: TimeoutException at org.jgroups.util.Promise._getResultWithTimeout(Promise.java:145) at org.jgroups.util.Promise.getResultWithTimeout(Promise.java:40) at org.jgroups.util.AckCollector.waitForAllAcks(AckCollector.java:93) at org.jgroups.protocols.RSVP$Entry.block(RSVP.java:287) at org.jgroups.protocols.RSVP.down(RSVP.java:118) at org.jgroups.protocols.pbcast.STABLE.down(STABLE.java:328) at org.jgroups.protocols.pbcast.GMS.down(GMS.java:965) at org.jgroups.protocols.FlowControl.down(FlowControl.java:351) at org.jgroups.protocols.FlowControl.down(FlowControl.java:351) at org.jgroups.protocols.FRAG2.down(FRAG2.java:147) at org.jgroups.protocols.pbcast.STATE_TRANSFER.down(STATE_TRANSFER.java:238) at org.jgroups.protocols.pbcast.FLUSH.down(FLUSH.java:312) at org.jgroups.stack.ProtocolStack.down(ProtocolStack.java:1025) at org.jgroups.JChannel.down(JChannel.java:729) at org.jgroups.JChannel.send(JChannel.java:445) at org.portico.bindings.jgroups.channel.FederationChannel.createFederation(FederationChannel.java:283) at org.portico.bindings.jgroups.JGroupsConnection.createFederation(JGroupsConnection.java:232) at org.portico.lrc.services.federation.handlers.outgoing.CreateFederationHandler.process(CreateFederationHandler.java:79) at org.portico.utils.messaging.MessageSink.process(MessageSink.java:187) at org.portico.impl.hla1516e.Impl1516eHelper.processMessage(Impl1516eHelper.java:99) at org.portico.impl.hla1516e.Rti1516eAmbassador.processMessage(Rti1516eAmbassador.java:5554) at org.portico.impl.hla1516e.Rti1516eAmbassador.createFederationExecution(Rti1516eAmbassador.java:310) ... 2 more

hodiapa commented 8 years ago

Any further updates on this issue? Did you get it resolved @bergtwvd ?

bergtwvd commented 8 years ago

The issue is still open.

hodiapa commented 8 years ago

I was able to run the c++ example federate on different hosts using the weave.

bergtwvd commented 8 years ago

I just checked as well. ubuntu@docker-2A:~$ weave version weave script 1.3.1 weave router 1.3.1 weave proxy 1.3.1

Yes it works now with this version!!! Great, and thanks for letting me know.

(Ps. what got fixed?)