Today something strange started happening to my Galera cluster - nodes would simply terminate with the following error:
141211 21:43:07 [ERROR] WSREP: exception from gcomm, backend must be restarted: evs::proto(16b39b34-817a-11e4-8633-660ab26d996d, GATHER, view_id(REG,16b39b34-817a-11e4-8633-660ab26d996d,152)) failed to form singleton view after exceeding max_install_timeouts 1, giving up (FATAL)
at gcomm/src/evs_proto.cpp:handle_install_timer():577
141211 21:43:07 [Note] WSREP: Received self-leave message.
My cluster:
Galera 25.3.5(r178)
MariaDB 35.2
Percona XtraDB 5.5.38
Running in Docker
And some of the nodes periodically fail to join cluster, ever with Galera cache reset (clean run):
141211 21:15:23 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql/
141211 21:15:23 mysqld_safe WSREP: Running position recovery with --log_error='/var/lib/mysql//wsrep_recovery.Q0gd4E' --pid-file='/var/lib/mysql//MysqlE1-recover.pid'
141211 21:15:26 mysqld_safe WSREP: Recovered position 484f14b8-68c2-11e4-9aef-33c75588e344:921397
141211 21:15:26 [Note] WSREP: wsrep_start_position var submitted: '484f14b8-68c2-11e4-9aef-33c75588e344:921397'
141211 21:15:26 [Note] WSREP: Read nil XID from storage engines, skipping position init
141211 21:15:26 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/galera/libgalera_smm.so'
141211 21:15:26 [Note] WSREP: wsrep_load(): Galera 25.3.5(r178) by Codership Oy <info@codership.com> loaded successfully.
141211 21:15:26 [Note] WSREP: CRC-32C: using hardware acceleration.
141211 21:15:26 [Note] WSREP: Found saved state: 484f14b8-68c2-11e4-9aef-33c75588e344:-1
141211 21:15:26 [Note] WSREP: Passing config to GCS: base_host = 10.1.0.4; base_port = 4567; cert.log_conflicts = no; debug = no; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 1; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.size = 128M; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout = PT3S; pc.checksum = false; pc.ignore_quorum = false; pc.ignore_sb = false; pc.npvo = false; pc.version = 0; pc.wait_prim = true; pc.wait_prim_timeout = P30S; pc.weight = 1; protonet.ba
141211 21:15:26 [Note] WSREP: Service thread queue flushed.
141211 21:15:26 [Note] WSREP: Assign initial position for certification: 921397, protocol version: -1
141211 21:15:26 [Note] WSREP: wsrep_sst_grab()
141211 21:15:26 [Note] WSREP: Start replication
141211 21:15:26 [Note] WSREP: Setting initial position to 484f14b8-68c2-11e4-9aef-33c75588e344:921397
141211 21:15:26 [Note] WSREP: protonet asio version 0
141211 21:15:26 [Note] WSREP: Using CRC-32C (optimized) for message checksums.
141211 21:15:26 [Note] WSREP: backend: asio
141211 21:15:26 [Note] WSREP: GMCast version 0
141211 21:15:26 [Note] WSREP: (d3c54d51-817a-11e4-b47d-f7c6b0d2f080, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567
141211 21:15:26 [Note] WSREP: (d3c54d51-817a-11e4-b47d-f7c6b0d2f080, 'tcp://0.0.0.0:4567') multicast: , ttl: 1
141211 21:15:26 [Note] WSREP: EVS version 0
141211 21:15:26 [Note] WSREP: PC version 0
141211 21:15:26 [Note] WSREP: gcomm: connecting to group 'my_wsrep_cluster', peer '10.0.1.5:,10.0.1.10:,10.0.1.11:,10.0.1.20:,10.1.0.4:'
141211 21:15:26 [Warning] WSREP: (d3c54d51-817a-11e4-b47d-f7c6b0d2f080, 'tcp://0.0.0.0:4567') address 'tcp://10.1.0.4:4567' points to own listening address, blacklisting
141211 21:15:26 [Note] WSREP: (d3c54d51-817a-11e4-b47d-f7c6b0d2f080, 'tcp://0.0.0.0:4567') address 'tcp://10.1.0.4:4567' pointing to uuid d3c54d51-817a-11e4-b47d-f7c6b0d2f080 is blacklisted, skipping
141211 21:15:26 [Note] WSREP: (d3c54d51-817a-11e4-b47d-f7c6b0d2f080, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://10.0.1.21:4567
141211 21:15:26 [Note] WSREP: (d3c54d51-817a-11e4-b47d-f7c6b0d2f080, 'tcp://0.0.0.0:4567') address 'tcp://10.1.0.4:4567' pointing to uuid d3c54d51-817a-11e4-b47d-f7c6b0d2f080 is blacklisted, skipping
141211 21:15:26 [Note] WSREP: (d3c54d51-817a-11e4-b47d-f7c6b0d2f080, 'tcp://0.0.0.0:4567') address 'tcp://10.1.0.4:4567' pointing to uuid d3c54d51-817a-11e4-b47d-f7c6b0d2f080 is blacklisted, skipping
141211 21:15:26 [Note] WSREP: (d3c54d51-817a-11e4-b47d-f7c6b0d2f080, 'tcp://0.0.0.0:4567') address 'tcp://10.1.0.4:4567' pointing to uuid d3c54d51-817a-11e4-b47d-f7c6b0d2f080 is blacklisted, skipping
141211 21:15:26 [Note] WSREP: (d3c54d51-817a-11e4-b47d-f7c6b0d2f080, 'tcp://0.0.0.0:4567') address 'tcp://10.1.0.4:4567' pointing to uuid d3c54d51-817a-11e4-b47d-f7c6b0d2f080 is blacklisted, skipping
141211 21:15:26 [Note] WSREP: (d3c54d51-817a-11e4-b47d-f7c6b0d2f080, 'tcp://0.0.0.0:4567') address 'tcp://10.1.0.4:4567' pointing to uuid d3c54d51-817a-11e4-b47d-f7c6b0d2f080 is blacklisted, skipping
141211 21:15:26 [Note] WSREP: (d3c54d51-817a-11e4-b47d-f7c6b0d2f080, 'tcp://0.0.0.0:4567') address 'tcp://10.1.0.4:4567' pointing to uuid d3c54d51-817a-11e4-b47d-f7c6b0d2f080 is blacklisted, skipping
141211 21:15:26 [Note] WSREP: (d3c54d51-817a-11e4-b47d-f7c6b0d2f080, 'tcp://0.0.0.0:4567') address 'tcp://10.1.0.4:4567' pointing to uuid d3c54d51-817a-11e4-b47d-f7c6b0d2f080 is blacklisted, skipping
141211 21:15:27 [Note] WSREP: (d3c54d51-817a-11e4-b47d-f7c6b0d2f080, 'tcp://0.0.0.0:4567') address 'tcp://10.1.0.4:4567' pointing to uuid d3c54d51-817a-11e4-b47d-f7c6b0d2f080 is blacklisted, skipping
141211 21:15:27 [Note] WSREP: (d3c54d51-817a-11e4-b47d-f7c6b0d2f080, 'tcp://0.0.0.0:4567') address 'tcp://10.1.0.4:4567' pointing to uuid d3c54d51-817a-11e4-b47d-f7c6b0d2f080 is blacklisted, skipping
141211 21:15:27 [Note] WSREP: (d3c54d51-817a-11e4-b47d-f7c6b0d2f080, 'tcp://0.0.0.0:4567') address 'tcp://10.1.0.4:4567' pointing to uuid d3c54d51-817a-11e4-b47d-f7c6b0d2f080 is blacklisted, skipping
141211 21:15:27 [Note] WSREP: (d3c54d51-817a-11e4-b47d-f7c6b0d2f080, 'tcp://0.0.0.0:4567') address 'tcp://10.1.0.4:4567' pointing to uuid d3c54d51-817a-11e4-b47d-f7c6b0d2f080 is blacklisted, skipping
141211 21:15:27 [Note] WSREP: (d3c54d51-817a-11e4-b47d-f7c6b0d2f080, 'tcp://0.0.0.0:4567') address 'tcp://10.1.0.4:4567' pointing to uuid d3c54d51-817a-11e4-b47d-f7c6b0d2f080 is blacklisted, skipping
141211 21:15:27 [Note] WSREP: (d3c54d51-817a-11e4-b47d-f7c6b0d2f080, 'tcp://0.0.0.0:4567') address 'tcp://10.1.0.4:4567' pointing to uuid d3c54d51-817a-11e4-b47d-f7c6b0d2f080 is blacklisted, skipping
141211 21:15:27 [Note] WSREP: (d3c54d51-817a-11e4-b47d-f7c6b0d2f080, 'tcp://0.0.0.0:4567') address 'tcp://10.1.0.4:4567' pointing to uuid d3c54d51-817a-11e4-b47d-f7c6b0d2f080 is blacklisted, skipping
141211 21:15:27 [Note] WSREP: (d3c54d51-817a-11e4-b47d-f7c6b0d2f080, 'tcp://0.0.0.0:4567') cleaning up duplicate 0x7f536143d200 after established 0x7f536143d500
141211 21:15:27 [Note] WSREP: (d3c54d51-817a-11e4-b47d-f7c6b0d2f080, 'tcp://0.0.0.0:4567') address 'tcp://10.1.0.4:4567' pointing to uuid d3c54d51-817a-11e4-b47d-f7c6b0d2f080 is blacklisted, skipping
141211 21:15:27 [Note] WSREP: (d3c54d51-817a-11e4-b47d-f7c6b0d2f080, 'tcp://0.0.0.0:4567') address 'tcp://10.1.0.4:4567' pointing to uuid d3c54d51-817a-11e4-b47d-f7c6b0d2f080 is blacklisted, skipping
141211 21:15:27 [Note] WSREP: (d3c54d51-817a-11e4-b47d-f7c6b0d2f080, 'tcp://0.0.0.0:4567') address 'tcp://10.1.0.4:4567' pointing to uuid d3c54d51-817a-11e4-b47d-f7c6b0d2f080 is blacklisted, skipping
pure virtual method called
terminate called without an active exception
**141211 21:15:27 [ERROR] mysqld got signal 6 ;**
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
To report this bug, see http://kb.askmonty.org/en/reporting-bugs
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Server version: 5.5.40-MariaDB-1~trusty-wsrep-log
key_buffer_size=0
read_buffer_size=131072
max_used_connections=0
max_threads=502
thread_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1101510 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x0 thread_stack 0x48000
/usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x7f53641db0de]
/usr/sbin/mysqld(handle_fatal_signal+0x457)[0x7f5363dc41e7]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x10340)[0x7f5362819340]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x39)[0x7f5361e6fcc9]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x148)[0x7f5361e730d8]
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x155)[0x7f53625656b5]
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x5e836)[0x7f5362563836]
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x5e863)[0x7f5362563863]
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x5f33f)[0x7f536256433f]
/usr/lib/galera/libgalera_smm.so(_Z4sendPN5gcomm6SocketERNS_8DatagramE+0x1f)[0x7f536008d89f]
/usr/lib/galera/libgalera_smm.so(_ZN5gcomm6GMCast11handle_downERNS_8DatagramERKNS_13ProtoDownMetaE+0x2c4)[0x7f536008df94]
/usr/lib/galera/libgalera_smm.so(_ZN5gcomm8Protolay9send_downERNS_8DatagramERKNS_13ProtoDownMetaE+0x40)[0x7f53600868d0]
/usr/lib/galera/libgalera_smm.so(_ZN5gcomm3evs5Proto9send_joinEb+0x174)[0x7f536007c1c4]
/usr/lib/galera/libgalera_smm.so(_ZN5gcomm3evs5Proto14handle_foreignERKNS0_7MessageE+0x126)[0x7f53600811c6]
/usr/lib/galera/libgalera_smm.so(_ZN5gcomm3evs5Proto10handle_msgERKNS0_7MessageERKNS_8DatagramE+0x540)[0x7f5360084b30]
/usr/lib/galera/libgalera_smm.so(_ZN5gcomm3evs5Proto9handle_upEPKvRKNS_8DatagramERKNS_11ProtoUpMetaE+0x2ac)[0x7f536008534c]
/usr/lib/galera/libgalera_smm.so(_ZN5gcomm8Protolay7send_upERKNS_8DatagramERKNS_11ProtoUpMetaE+0x36)[0x7f5360086546]
/usr/lib/galera/libgalera_smm.so(_ZN5gcomm6GMCast9handle_upEPKvRKNS_8DatagramERKNS_11ProtoUpMetaE+0x1f6)[0x7f5360098826]
/usr/lib/galera/libgalera_smm.so(_ZN5gcomm10Protostack8dispatchEPKvRKNS_8DatagramERKNS_11ProtoUpMetaE+0x58)[0x7f53600c1bb8]
/usr/lib/galera/libgalera_smm.so(_ZN5gcomm12AsioProtonet8dispatchERKPKvRKNS_8DatagramERKNS_11ProtoUpMetaE+0x4b)[0x7f53600e7dfb]
/usr/lib/galera/libgalera_smm.so(_ZN5gcomm13AsioTcpSocket12read_handlerERKN4asio10error_codeEm+0x613)[0x7f53600cc503]
/usr/lib/galera/libgalera_smm.so(_ZN4asio6detail7read_opINS_19basic_stream_socketINS_2ip3tcpENS_21stream_socket_serviceIS4_EEEEN5boost5arrayINS_14mutable_bufferELm1EEENS8_3_bi6bind_tImNS8_4_mfi3mf2ImN5gcomm13AsioTcpSocketERKNS_10error_codeEmEENSC_5list3INSC_5valueINS8_10shared_ptrISH_EEEEPFNS8_3argILi1EEEvEPFNSR_ILi2EEEvEEEEENSD_IvNSF_IvSH_SK_mEESY_EEEclESK_mi+0x94)[0x7f53600d9114]
/usr/lib/galera/libgalera_smm.so(_ZN4asio6detail23reactive_socket_recv_opINS0_17consuming_buffersINS_14mutable_bufferEN5boost5arrayIS3_Lm1EEEEENS0_7read_opINS_19basic_stream_socketINS_2ip3tcpENS_21stream_socket_serviceISB_EEEES6_NS4_3_bi6bind_tImNS4_4_mfi3mf2ImN5gcomm13AsioTcpSocketERKNS_10error_codeEmEENSF_5list3INSF_5valueINS4_10shared_ptrISK_EEEEPFNS4_3argILi1EEEvEPFNSU_ILi2EEEvEEEEENSG_IvNSI_IvSK_SN_mEES11_EEEEE11do_completeEPNS0_15task_io_serviceEPNS0_25task_io_service_operationESL_m+0xdd)[0x7f53600d945d]
/usr/lib/galera/libgalera_smm.so(_ZN4asio6detail15task_io_service3runERNS_10error_codeE+0x407)[0x7f53600eb0d7]
/usr/lib/galera/libgalera_smm.so(_ZN5gcomm12AsioProtonet10event_loopERKN2gu8datetime6PeriodE+0x1a2)[0x7f53600e91f2]
/usr/lib/galera/libgalera_smm.so(_ZN5gcomm2PC7connectEb+0x5d5)[0x7f53600ab985]
/usr/lib/galera/libgalera_smm.so(_ZN9GCommConn7connectERKSsb+0x14e)[0x7f5360103fae]
/usr/lib/galera/libgalera_smm.so(+0x192146)[0x7f5360100146]
/usr/lib/galera/libgalera_smm.so(gcs_core_open+0x11e)[0x7f53600f6d9e]
/usr/lib/galera/libgalera_smm.so(gcs_open+0x276)[0x7f53600fa786]
/usr/lib/galera/libgalera_smm.so(_ZN6galera13ReplicatorSMM7connectERKSsS2_S2_b+0x243)[0x7f536014eca3]
/usr/lib/galera/libgalera_smm.so(galera_connect+0x93)[0x7f536015cdc3]
/usr/sbin/mysqld(_Z23wsrep_start_replicationv+0x128)[0x7f5363d6a8b8]
/usr/sbin/mysqld(_Z18wsrep_init_startupb+0x5d)[0x7f5363d6aa4d]
/usr/sbin/mysqld(+0x2f9b23)[0x7f5363bd3b23]
/usr/sbin/mysqld(_Z11mysqld_mainiPPc+0x90a)[0x7f5363bd843a]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5)[0x7f5361e5aec5]
/usr/sbin/mysqld(+0x2f2d26)[0x7f5363bccd26]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
141211 21:15:28 mysqld_safe mysqld from pid file /var/lib/mysql/mysql.pid ended
141211 21:15:40 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql/
141211 21:15:40 mysqld_safe WSREP: Running position recovery with --log_error='/var/lib/mysql//wsrep_recovery.3ZRAO2' --pid-file='/var/lib/mysql//MysqlE1-recover.pid'
141211 21:15:42 mysqld_safe WSREP: Recovered position 484f14b8-68c2-11e4-9aef-33c75588e344:921397
141211 21:15:42 [Note] WSREP: wsrep_start_position var submitted: '484f14b8-68c2-11e4-9aef-33c75588e344:921397'
141211 21:15:42 [Note] WSREP: Read nil XID from storage engines, skipping position init
141211 21:15:42 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/galera/libgalera_smm.so'
141211 21:15:42 [Note] WSREP: wsrep_load(): Galera 25.3.5(r178) by Codership Oy <info@codership.com> loaded successfully.
141211 21:15:42 [Note] WSREP: CRC-32C: using hardware acceleration.
141211 21:15:42 [Note] WSREP: Found saved state: 484f14b8-68c2-11e4-9aef-33c75588e344:-1
141211 21:15:42 [Note] WSREP: Passing config to GCS: base_host = 10.1.0.4; base_port = 4567; cert.log_conflicts = no; debug = no; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 1; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.size = 128M; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout = PT3S; pc.checksum = false; pc.ignore_quorum = false; pc.ignore_sb = false; pc.npvo = false; pc.version = 0; pc.wait_prim = true; pc.wait_prim_timeout = P30S; pc.weight = 1; protonet.ba
141211 21:15:42 [Note] WSREP: Service thread queue flushed.
141211 21:15:42 [Note] WSREP: Assign initial position for certification: 921397, protocol version: -1
141211 21:15:42 [Note] WSREP: wsrep_sst_grab()
141211 21:15:42 [Note] WSREP: Start replication
141211 21:15:42 [Note] WSREP: Setting initial position to 484f14b8-68c2-11e4-9aef-33c75588e344:921397
141211 21:15:42 [Note] WSREP: protonet asio version 0
141211 21:15:42 [Note] WSREP: Using CRC-32C (optimized) for message checksums.
141211 21:15:42 [Note] WSREP: backend: asio
141211 21:15:42 [Note] WSREP: GMCast version 0
141211 21:15:42 [Note] WSREP: (dd54d291-817a-11e4-8e1e-7e40774101bf, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567
141211 21:15:42 [Note] WSREP: (dd54d291-817a-11e4-8e1e-7e40774101bf, 'tcp://0.0.0.0:4567') multicast: , ttl: 1
141211 21:15:42 [Note] WSREP: EVS version 0
141211 21:15:42 [Note] WSREP: PC version 0
141211 21:15:42 [Note] WSREP: gcomm: connecting to group 'my_wsrep_cluster', peer '10.0.1.5:,10.0.1.10:,10.0.1.11:,10.0.1.20:,10.1.0.4:'
141211 21:15:42 [Warning] WSREP: (dd54d291-817a-11e4-8e1e-7e40774101bf, 'tcp://0.0.0.0:4567') address 'tcp://10.1.0.4:4567' points to own listening address, blacklisting
141211 21:15:43 [Note] WSREP: (dd54d291-817a-11e4-8e1e-7e40774101bf, 'tcp://0.0.0.0:4567') address 'tcp://10.1.0.4:4567' pointing to uuid dd54d291-817a-11e4-8e1e-7e40774101bf is blacklisted, skipping
141211 21:15:43 [Note] WSREP: (dd54d291-817a-11e4-8e1e-7e40774101bf, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://10.0.1.21:4567
141211 21:15:43 [Note] WSREP: (dd54d291-817a-11e4-8e1e-7e40774101bf, 'tcp://0.0.0.0:4567') address 'tcp://10.1.0.4:4567' pointing to uuid dd54d291-817a-11e4-8e1e-7e40774101bf is blacklisted, skipping
141211 21:15:43 [Note] WSREP: (dd54d291-817a-11e4-8e1e-7e40774101bf, 'tcp://0.0.0.0:4567') address 'tcp://10.1.0.4:4567' pointing to uuid dd54d291-817a-11e4-8e1e-7e40774101bf is blacklisted, skipping
141211 21:15:43 [Note] WSREP: (dd54d291-817a-11e4-8e1e-7e40774101bf, 'tcp://0.0.0.0:4567') address 'tcp://10.1.0.4:4567' pointing to uuid dd54d291-817a-11e4-8e1e-7e40774101bf is blacklisted, skipping
141211 21:15:43 [Note] WSREP: (dd54d291-817a-11e4-8e1e-7e40774101bf, 'tcp://0.0.0.0:4567') address 'tcp://10.1.0.4:4567' pointing to uuid dd54d291-817a-11e4-8e1e-7e40774101bf is blacklisted, skipping
141211 21:15:43 [Note] WSREP: (dd54d291-817a-11e4-8e1e-7e40774101bf, 'tcp://0.0.0.0:4567') address 'tcp://10.1.0.4:4567' pointing to uuid dd54d291-817a-11e4-8e1e-7e40774101bf is blacklisted, skipping
141211 21:15:43 [Note] WSREP: (dd54d291-817a-11e4-8e1e-7e40774101bf, 'tcp://0.0.0.0:4567') address 'tcp://10.1.0.4:4567' pointing to uuid dd54d291-817a-11e4-8e1e-7e40774101bf is blacklisted, skipping
141211 21:15:43 [Note] WSREP: (dd54d291-817a-11e4-8e1e-7e40774101bf, 'tcp://0.0.0.0:4567') address 'tcp://10.1.0.4:4567' pointing to uuid dd54d291-817a-11e4-8e1e-7e40774101bf is blacklisted, skipping
141211 21:15:43 [Note] WSREP: (dd54d291-817a-11e4-8e1e-7e40774101bf, 'tcp://0.0.0.0:4567') address 'tcp://10.1.0.4:4567' pointing to uuid dd54d291-817a-11e4-8e1e-7e40774101bf is blacklisted, skipping
141211 21:15:43 [Note] WSREP: (dd54d291-817a-11e4-8e1e-7e40774101bf, 'tcp://0.0.0.0:4567') address 'tcp://10.1.0.4:4567' pointing to uuid dd54d291-817a-11e4-8e1e-7e40774101bf is blacklisted, skipping
141211 21:15:43 [Note] WSREP: (dd54d291-817a-11e4-8e1e-7e40774101bf, 'tcp://0.0.0.0:4567') address 'tcp://10.1.0.4:4567' pointing to uuid dd54d291-817a-11e4-8e1e-7e40774101bf is blacklisted, skipping
141211 21:15:43 [Note] WSREP: (dd54d291-817a-11e4-8e1e-7e40774101bf, 'tcp://0.0.0.0:4567') address 'tcp://10.1.0.4:4567' pointing to uuid dd54d291-817a-11e4-8e1e-7e40774101bf is blacklisted, skipping
141211 21:15:43 [Note] WSREP: (dd54d291-817a-11e4-8e1e-7e40774101bf, 'tcp://0.0.0.0:4567') turning message relay requesting off
141211 21:15:43 [Note] WSREP: (dd54d291-817a-11e4-8e1e-7e40774101bf, 'tcp://0.0.0.0:4567') address 'tcp://10.1.0.4:4567' pointing to uuid dd54d291-817a-11e4-8e1e-7e40774101bf is blacklisted, skipping
141211 21:15:43 [Note] WSREP: (dd54d291-817a-11e4-8e1e-7e40774101bf, 'tcp://0.0.0.0:4567') address 'tcp://10.1.0.4:4567' pointing to uuid dd54d291-817a-11e4-8e1e-7e40774101bf is blacklisted, skipping
141211 21:15:43 [Note] WSREP: (dd54d291-817a-11e4-8e1e-7e40774101bf, 'tcp://0.0.0.0:4567') address 'tcp://10.1.0.4:4567' pointing to uuid dd54d291-817a-11e4-8e1e-7e40774101bf is blacklisted, skipping
141211 21:15:43 [Note] WSREP: (dd54d291-817a-11e4-8e1e-7e40774101bf, 'tcp://0.0.0.0:4567') address 'tcp://10.1.0.4:4567' pointing to uuid dd54d291-817a-11e4-8e1e-7e40774101bf is blacklisted, skipping
141211 21:15:45 [Note] WSREP: (dd54d291-817a-11e4-8e1e-7e40774101bf, 'tcp://0.0.0.0:4567') address 'tcp://10.1.0.4:4567' pointing to uuid dd54d291-817a-11e4-8e1e-7e40774101bf is blacklisted, skipping
141211 21:15:45 [Note] WSREP: (dd54d291-817a-11e4-8e1e-7e40774101bf, 'tcp://0.0.0.0:4567') address 'tcp://10.1.0.4:4567' pointing to uuid dd54d291-817a-11e4-8e1e-7e40774101bf is blacklisted, skipping
141211 21:15:45 [Note] WSREP: (dd54d291-817a-11e4-8e1e-7e40774101bf, 'tcp://0.0.0.0:4567') address 'tcp://10.1.0.4:4567' pointing to uuid dd54d291-817a-11e4-8e1e-7e40774101bf is blacklisted, skipping
141211 21:15:45 [Note] WSREP: (dd54d291-817a-11e4-8e1e-7e40774101bf, 'tcp://0.0.0.0:4567') address 'tcp://10.1.0.4:4567' pointing to uuid dd54d291-817a-11e4-8e1e-7e40774101bf is blacklisted, skipping
141211 21:15:45 [Note] WSREP: (dd54d291-817a-11e4-8e1e-7e40774101bf, 'tcp://0.0.0.0:4567') address 'tcp://10.1.0.4:4567' pointing to uuid dd54d291-817a-11e4-8e1e-7e40774101bf is blacklisted, skipping
141211 21:15:45 [Note] WSREP: (dd54d291-817a-11e4-8e1e-7e40774101bf, 'tcp://0.0.0.0:4567') address 'tcp://10.1.0.4:4567' pointing to uuid dd54d291-817a-11e4-8e1e-7e40774101bf is blacklisted, skipping
141211 21:15:45 [Note] WSREP: (dd54d291-817a-11e4-8e1e-7e40774101bf, 'tcp://0.0.0.0:4567') address 'tcp://10.1.0.4:4567' pointing to uuid dd54d291-817a-11e4-8e1e-7e40774101bf is blacklisted, skipping
141211 21:15:45 [Note] WSREP: (dd54d291-817a-11e4-8e1e-7e40774101bf, 'tcp://0.0.0.0:4567') address 'tcp://10.1.0.4:4567' pointing to uuid dd54d291-817a-11e4-8e1e-7e40774101bf is blacklisted, skipping
141211 21:15:45 [Note] WSREP: (dd54d291-817a-11e4-8e1e-7e40774101bf, 'tcp://0.0.0.0:4567') address 'tcp://10.1.0.4:4567' pointing to uuid dd54d291-817a-11e4-8e1e-7e40774101bf is blacklisted, skipping
141211 21:15:45 [Note] WSREP: (dd54d291-817a-11e4-8e1e-7e40774101bf, 'tcp://0.0.0.0:4567') address 'tcp://10.1.0.4:4567' pointing to uuid dd54d291-817a-11e4-8e1e-7e40774101bf is blacklisted, skipping
141211 21:15:45 [Note] WSREP: (dd54d291-817a-11e4-8e1e-7e40774101bf, 'tcp://0.0.0.0:4567') address 'tcp://10.1.0.4:4567' pointing to uuid dd54d291-817a-11e4-8e1e-7e40774101bf is blacklisted, skipping
141211 21:15:45 [Note] WSREP: (dd54d291-817a-11e4-8e1e-7e40774101bf, 'tcp://0.0.0.0:4567') address 'tcp://10.1.0.4:4567' pointing to uuid dd54d291-817a-11e4-8e1e-7e40774101bf is blacklisted, skipping
141211 21:15:45 [Note] WSREP: (dd54d291-817a-11e4-8e1e-7e40774101bf, 'tcp://0.0.0.0:4567') address 'tcp://10.1.0.4:4567' pointing to uuid dd54d291-817a-11e4-8e1e-7e40774101bf is blacklisted, skipping
141211 21:15:45 [Note] WSREP: (dd54d291-817a-11e4-8e1e-7e40774101bf, 'tcp://0.0.0.0:4567') address 'tcp://10.1.0.4:4567' pointing to uuid dd54d291-817a-11e4-8e1e-7e40774101bf is blacklisted, skipping
141211 21:15:45 [Note] WSREP: (dd54d291-817a-11e4-8e1e-7e40774101bf, 'tcp://0.0.0.0:4567') address 'tcp://10.1.0.4:4567' pointing to uuid dd54d291-817a-11e4-8e1e-7e40774101bf is blacklisted, skipping
141211 21:15:45 [Note] WSREP: (dd54d291-817a-11e4-8e1e-7e40774101bf, 'tcp://0.0.0.0:4567') address 'tcp://10.1.0.4:4567' pointing to uuid dd54d291-817a-11e4-8e1e-7e40774101bf is blacklisted, skipping
141211 21:15:45 [Note] WSREP: (dd54d291-817a-11e4-8e1e-7e40774101bf, 'tcp://0.0.0.0:4567') address 'tcp://10.1.0.4:4567' pointing to uuid dd54d291-817a-11e4-8e1e-7e40774101bf is blacklisted, skipping
141211 21:15:45 [Note] WSREP: (dd54d291-817a-11e4-8e1e-7e40774101bf, 'tcp://0.0.0.0:4567') address 'tcp://10.1.0.4:4567' pointing to uuid dd54d291-817a-11e4-8e1e-7e40774101bf is blacklisted, skipping
141211 21:15:45 [Note] WSREP: (dd54d291-817a-11e4-8e1e-7e40774101bf, 'tcp://0.0.0.0:4567') address 'tcp://10.1.0.4:4567' pointing to uuid dd54d291-817a-11e4-8e1e-7e40774101bf is blacklisted, skipping
141211 21:15:45 [Note] WSREP: (dd54d291-817a-11e4-8e1e-7e40774101bf, 'tcp://0.0.0.0:4567') address 'tcp://10.1.0.4:4567' pointing to uuid dd54d291-817a-11e4-8e1e-7e40774101bf is blacklisted, skipping
141211 21:15:48 [Note] WSREP: evs::proto(dd54d291-817a-11e4-8e1e-7e40774101bf, GATHER, view_id(TRANS,dd54d291-817a-11e4-8e1e-7e40774101bf,0)) suspecting node: 16b39b34-817a-11e4-8633-660ab26d996d
141211 21:15:48 [Note] WSREP: evs::proto(dd54d291-817a-11e4-8e1e-7e40774101bf, GATHER, view_id(TRANS,dd54d291-817a-11e4-8e1e-7e40774101bf,0)) suspecting node: d3c54d51-817a-11e4-b47d-f7c6b0d2f080
141211 21:15:48 [Note] WSREP: node d3c54d51-817a-11e4-b47d-f7c6b0d2f080 marked with nil view id and suspected in all present join messages, declaring inactive
141211 21:15:48 [Note] WSREP: node d3c54d51-817a-11e4-b47d-f7c6b0d2f080 marked with nil view id and suspected in all present join messages, declaring inactive
141211 21:15:48 [Note] WSREP: node d3c54d51-817a-11e4-b47d-f7c6b0d2f080 marked with nil view id and suspected in all present join messages, declaring inactive
141211 21:15:48 [Note] WSREP: node d3c54d51-817a-11e4-b47d-f7c6b0d2f080 marked with nil view id and suspected in all present join messages, declaring inactive
141211 21:15:48 [Note] WSREP: node d3c54d51-817a-11e4-b47d-f7c6b0d2f080 marked with nil view id and suspected in all present join messages, declaring inactive
141211 21:15:48 [Note] WSREP: node d3c54d51-817a-11e4-b47d-f7c6b0d2f080 marked with nil view id and suspected in all present join messages, declaring inactive
141211 21:15:48 [Note] WSREP: node d3c54d51-817a-11e4-b47d-f7c6b0d2f080 marked with nil view id and suspected in all present join messages, declaring inactive
141211 21:15:48 [Note] WSREP: node d3c54d51-817a-11e4-b47d-f7c6b0d2f080 marked with nil view id and suspected in all present join messages, declaring inactive
141211 21:15:48 [Note] WSREP: evs::proto(dd54d291-817a-11e4-8e1e-7e40774101bf, GATHER, view_id(TRANS,dd54d291-817a-11e4-8e1e-7e40774101bf,0)) suspecting node: 16b39b34-817a-11e4-8633-660ab26d996d
141211 21:15:49 [Note] WSREP: evs::proto(dd54d291-817a-11e4-8e1e-7e40774101bf, GATHER, view_id(TRANS,dd54d291-817a-11e4-8e1e-7e40774101bf,0)) suspecting node: 16b39b34-817a-11e4-8633-660ab26d996d
141211 21:15:49 [Note] WSREP: node d3c54d51-817a-11e4-b47d-f7c6b0d2f080 marked with nil view id and suspected in all present join messages, declaring inactive
141211 21:15:49 [Note] WSREP: node d3c54d51-817a-11e4-b47d-f7c6b0d2f080 marked with nil view id and suspected in all present join messages, declaring inactive
141211 21:15:49 [Note] WSREP: node d3c54d51-817a-11e4-b47d-f7c6b0d2f080 marked with nil view id and suspected in all present join messages, declaring inactive
141211 21:15:49 [Note] WSREP: node d3c54d51-817a-11e4-b47d-f7c6b0d2f080 marked with nil view id and suspected in all present join messages, declaring inactive
141211 21:15:49 [Note] WSREP: node d3c54d51-817a-11e4-b47d-f7c6b0d2f080 marked with nil view id and suspected in all present join messages, declaring inactive
141211 21:15:49 [Note] WSREP: (dd54d291-817a-11e4-8e1e-7e40774101bf, 'tcp://0.0.0.0:4567') address 'tcp://10.1.0.4:4567' pointing to uuid dd54d291-817a-11e4-8e1e-7e40774101bf is blacklisted, skipping
141211 21:15:49 [Note] WSREP: (dd54d291-817a-11e4-8e1e-7e40774101bf, 'tcp://0.0.0.0:4567') address 'tcp://10.1.0.4:4567' pointing to uuid dd54d291-817a-11e4-8e1e-7e40774101bf is blacklisted, skipping
141211 21:15:49 [Note] WSREP: (dd54d291-817a-11e4-8e1e-7e40774101bf, 'tcp://0.0.0.0:4567') address 'tcp://10.1.0.4:4567' pointing to uuid dd54d291-817a-11e4-8e1e-7e40774101bf is blacklisted, skipping
141211 21:15:49 [Note] WSREP: (dd54d291-817a-11e4-8e1e-7e40774101bf, 'tcp://0.0.0.0:4567') address 'tcp://10.1.0.4:4567' pointing to uuid dd54d291-817a-11e4-8e1e-7e40774101bf is blacklisted, skipping
141211 21:15:50 [Note] WSREP: node d3c54d51-817a-11e4-b47d-f7c6b0d2f080 marked with nil view id and suspected in all present join messages, declaring inactive
141211 21:15:50 [Note] WSREP: node d3c54d51-817a-11e4-b47d-f7c6b0d2f080 marked with nil view id and suspected in all present join messages, declaring inactive
141211 21:15:50 [Note] WSREP: node d3c54d51-817a-11e4-b47d-f7c6b0d2f080 marked with nil view id and suspected in all present join messages, declaring inactive
141211 21:15:50 [Note] WSREP: node d3c54d51-817a-11e4-b47d-f7c6b0d2f080 marked with nil view id and suspected in all present join messages, declaring inactive
141211 21:15:51 [Note] WSREP: node d3c54d51-817a-11e4-b47d-f7c6b0d2f080 marked with nil view id and suspected in all present join messages, declaring inactive
141211 21:15:51 [Note] WSREP: node d3c54d51-817a-11e4-b47d-f7c6b0d2f080 marked with nil view id and suspected in all present join messages, declaring inactive
141211 21:15:51 [Note] WSREP: node d3c54d51-817a-11e4-b47d-f7c6b0d2f080 marked with nil view id and suspected in all present join messages, declaring inactive
141211 21:15:51 [Note] WSREP: node d3c54d51-817a-11e4-b47d-f7c6b0d2f080 marked with nil view id and suspected in all present join messages, declaring inactive
141211 21:15:52 [Note] WSREP: node d3c54d51-817a-11e4-b47d-f7c6b0d2f080 marked with nil view id and suspected in all present join messages, declaring inactive
141211 21:15:52 [Note] WSREP: node d3c54d51-817a-11e4-b47d-f7c6b0d2f080 marked with nil view id and suspected in all present join messages, declaring inactive
141211 21:15:52 [Note] WSREP: node d3c54d51-817a-11e4-b47d-f7c6b0d2f080 marked with nil view id and suspected in all present join messages, declaring inactive
141211 21:15:52 [Note] WSREP: node d3c54d51-817a-11e4-b47d-f7c6b0d2f080 marked with nil view id and suspected in all present join messages, declaring inactive
141211 21:15:53 [Note] WSREP: node d3c54d51-817a-11e4-b47d-f7c6b0d2f080 marked with nil view id and suspected in all present join messages, declaring inactive
141211 21:15:53 [Note] WSREP: node d3c54d51-817a-11e4-b47d-f7c6b0d2f080 marked with nil view id and suspected in all present join messages, declaring inactive
141211 21:15:53 [Note] WSREP: node d3c54d51-817a-11e4-b47d-f7c6b0d2f080 marked with nil view id and suspected in all present join messages, declaring inactive
141211 21:15:53 [Note] WSREP: node d3c54d51-817a-11e4-b47d-f7c6b0d2f080 marked with nil view id and suspected in all present join messages, declaring inactive
141211 21:15:53 [Note] WSREP: (dd54d291-817a-11e4-8e1e-7e40774101bf, 'tcp://0.0.0.0:4567') address 'tcp://10.1.0.4:4567' pointing to uuid dd54d291-817a-11e4-8e1e-7e40774101bf is blacklisted, skipping
141211 21:15:53 [Note] WSREP: (dd54d291-817a-11e4-8e1e-7e40774101bf, 'tcp://0.0.0.0:4567') address 'tcp://10.1.0.4:4567' pointing to uuid dd54d291-817a-11e4-8e1e-7e40774101bf is blacklisted, skipping
141211 21:15:53 [Note] WSREP: (dd54d291-817a-11e4-8e1e-7e40774101bf, 'tcp://0.0.0.0:4567') address 'tcp://10.1.0.4:4567' pointing to uuid dd54d291-817a-11e4-8e1e-7e40774101bf is blacklisted, skipping
141211 21:15:53 [Note] WSREP: (dd54d291-817a-11e4-8e1e-7e40774101bf, 'tcp://0.0.0.0:4567') address 'tcp://10.1.0.4:4567' pointing to uuid dd54d291-817a-11e4-8e1e-7e40774101bf is blacklisted, skipping
141211 21:15:54 [Note] WSREP: node d3c54d51-817a-11e4-b47d-f7c6b0d2f080 marked with nil view id and suspected in all present join messages, declaring inactive
141211 21:15:54 [Note] WSREP: node d3c54d51-817a-11e4-b47d-f7c6b0d2f080 marked with nil view id and suspected in all present join messages, declaring inactive
141211 21:15:54 [Note] WSREP: node d3c54d51-817a-11e4-b47d-f7c6b0d2f080 marked with nil view id and suspected in all present join messages, declaring inactive
141211 21:15:54 [Note] WSREP: node d3c54d51-817a-11e4-b47d-f7c6b0d2f080 marked with nil view id and suspected in all present join messages, declaring inactive
141211 21:15:55 [Note] WSREP: node d3c54d51-817a-11e4-b47d-f7c6b0d2f080 marked with nil view id and suspected in all present join messages, declaring inactive
141211 21:15:55 [Note] WSREP: node d3c54d51-817a-11e4-b47d-f7c6b0d2f080 marked with nil view id and suspected in all present join messages, declaring inactive
141211 21:15:55 [Note] WSREP: node d3c54d51-817a-11e4-b47d-f7c6b0d2f080 marked with nil view id and suspected in all present join messages, declaring inactive
141211 21:15:55 [Note] WSREP: node d3c54d51-817a-11e4-b47d-f7c6b0d2f080 marked with nil view id and suspected in all present join messages, declaring inactive
141211 21:15:56 [Note] WSREP: node d3c54d51-817a-11e4-b47d-f7c6b0d2f080 marked with nil view id and suspected in all present join messages, declaring inactive
141211 21:15:56 [Note] WSREP: node d3c54d51-817a-11e4-b47d-f7c6b0d2f080 marked with nil view id and suspected in all present join messages, declaring inactive
141211 21:15:56 [Note] WSREP: node d3c54d51-817a-11e4-b47d-f7c6b0d2f080 marked with nil view id and suspected in all present join messages, declaring inactive
141211 21:15:56 [Note] WSREP: node d3c54d51-817a-11e4-b47d-f7c6b0d2f080 marked with nil view id and suspected in all present join messages, declaring inactive
141211 21:15:57 [Note] WSREP: node d3c54d51-817a-11e4-b47d-f7c6b0d2f080 marked with nil view id and suspected in all present join messages, declaring inactive
141211 21:15:57 [Note] WSREP: node d3c54d51-817a-11e4-b47d-f7c6b0d2f080 marked with nil view id and suspected in all present join messages, declaring inactive
141211 21:15:57 [Note] WSREP: node d3c54d51-817a-11e4-b47d-f7c6b0d2f080 marked with nil view id and suspected in all present join messages, declaring inactive
141211 21:15:57 [Note] WSREP: node d3c54d51-817a-11e4-b47d-f7c6b0d2f080 marked with nil view id and suspected in all present join messages, declaring inactive
141211 21:15:57 [Note] WSREP: (dd54d291-817a-11e4-8e1e-7e40774101bf, 'tcp://0.0.0.0:4567') address 'tcp://10.1.0.4:4567' pointing to uuid dd54d291-817a-11e4-8e1e-7e40774101bf is blacklisted, skipping
141211 21:15:57 [Note] WSREP: (dd54d291-817a-11e4-8e1e-7e40774101bf, 'tcp://0.0.0.0:4567') address 'tcp://10.1.0.4:4567' pointing to uuid dd54d291-817a-11e4-8e1e-7e40774101bf is blacklisted, skipping
141211 21:15:57 [Note] WSREP: (dd54d291-817a-11e4-8e1e-7e40774101bf, 'tcp://0.0.0.0:4567') address 'tcp://10.1.0.4:4567' pointing to uuid dd54d291-817a-11e4-8e1e-7e40774101bf is blacklisted, skipping
141211 21:15:57 [Note] WSREP: (dd54d291-817a-11e4-8e1e-7e40774101bf, 'tcp://0.0.0.0:4567') address 'tcp://10.1.0.4:4567' pointing to uuid dd54d291-817a-11e4-8e1e-7e40774101bf is blacklisted, skipping
141211 21:15:58 [Warning] WSREP: evs::proto(dd54d291-817a-11e4-8e1e-7e40774101bf, GATHER, view_id(TRANS,dd54d291-817a-11e4-8e1e-7e40774101bf,0)) install timer expired
evs::proto(evs::proto(dd54d291-817a-11e4-8e1e-7e40774101bf, GATHER, view_id(TRANS,dd54d291-817a-11e4-8e1e-7e40774101bf,0)), GATHER) {
current_view=view(view_id(TRANS,dd54d291-817a-11e4-8e1e-7e40774101bf,0) memb {
dd54d291-817a-11e4-8e1e-7e40774101bf,0
} joined {
} left {
} partitioned {
}),
input_map=evs::input_map: {aru_seq=-1,safe_seq=-1,node_index=node: {idx=0,range=[0,-1],safe_seq=-1} },
fifo_seq=31,
last_sent=-1,
known={
16b39b34-817a-11e4-8633-660ab26d996d,evs::node{operational=0,suspected=1,installed=0,fifo_seq=-1,}
2541ca74-817a-11e4-b1f7-d7f0ddaaac29,evs::node{operational=1,suspected=0,installed=0,fifo_seq=3310,join_message=
evs::msg{version=0,type=4,user_type=255,order=1,seq=139,seq_range=-1,aru_seq=139,flags=4,source=2541ca74-817a-11e4-b1f7-d7f0ddaaac29,source_view_id=view_id(REG,16b39b34-817a-11e4-8633-660ab26d996d,34),range_uuid=00000000-0000-0000-0000-000000000000,range=[-1,-1],fifo_seq=3310,node_list=( 16b39b34-817a-11e4-8633-660ab26d996d,node: {operational=0,suspected=1,leave_seq=-1,view_id=view_id(REG,16b39b34-817a-11e4-8633-660ab26d996d,34),safe_seq=139,im_range=[140,139],}
2541ca74-817a-11e4-b1f7-d7f0ddaaac29,node: {operational=1,suspected=0,leave_seq=-1,view_id=view_id(REG,16b39b34-817a-11e4-8633-660ab26d996d,34),safe_seq=139,im_range=[140,139],}
2fc3ecce-817a-11e4-9caa-0705faa8e6f5,node: {operational=1,suspected=0,leave_seq=-1,view_id=view_id(REG,16b39b34-817a-11e4-8633-660ab26d996d,34),safe_seq=139,im_range=[140,139],}
77daa67d-8174-11e4-a3ed-370c72485569,node: {operational=0,suspected=1,leave_seq=-1,view_id=view_id(REG,16b39b34-817a-11e4-8633-660ab26d996d,34),safe_seq=139,im_range=[140,139],}
d3c54d51-817a-11e4-b47d-f7c6b0d2f080,node: {operational=0,suspected=1,leave_seq=-1,view_id=view_id(REG,00000000-0000-0000-0000-000000000000,0),safe_seq=-1,im_range=[-1,-1],}
dd54d291-817a-11e4-8e1e-7e40774101bf,node: {operational=1,suspected=0,leave_seq=-1,view_id=view_id(TRANS,dd54d291-817a-11e4-8e1e-7e40774101bf,0),safe_seq=-1,im_range=[0,-1],}
fa16b484-80f9-11e4-8036-6276f2af149d,node: {operational=0,suspected=1,leave_seq=-1,view_id=view_id(REG,16b39b34-817a-11e4-8633-660ab26d996d,34),safe_seq=139,im_range=[140,139],}
)
},
}
2fc3ecce-817a-11e4-9caa-0705faa8e6f5,evs::node{operational=1,suspected=0,installed=0,fifo_seq=3118,join_message=
evs::msg{version=0,type=4,user_type=255,order=1,seq=139,seq_range=-1,aru_seq=139,flags=4,source=2fc3ecce-817a-11e4-9caa-0705faa8e6f5,source_view_id=view_id(REG,16b39b34-817a-11e4-8633-660ab26d996d,34),range_uuid=00000000-0000-0000-0000-000000000000,range=[-1,-1],fifo_seq=3118,node_list=( 16b39b34-817a-11e4-8633-660ab26d996d,node: {operational=0,suspected=1,leave_seq=-1,view_id=view_id(REG,16b39b34-817a-11e4-8633-660ab26d996d,34),safe_seq=139,im_range=[140,139],}
2541ca74-817a-11e4-b1f7-d7f0ddaaac29,node: {operational=1,suspected=0,leave_seq=-1,view_id=view_id(REG,16b39b34-817a-11e4-8633-660ab26d996d,34),safe_seq=139,im_range=[140,139],}
2fc3ecce-817a-11e4-9caa-0705faa8e6f5,node: {operational=1,suspected=0,leave_seq=-1,view_id=view_id(REG,16b39b34-817a-11e4-8633-660ab26d996d,34),safe_seq=139,im_range=[140,139],}
77daa67d-8174-11e4-a3ed-370c72485569,node: {operational=0,suspected=1,leave_seq=-1,view_id=view_id(REG,16b39b34-817a-11e4-8633-660ab26d996d,34),safe_seq=139,im_range=[140,139],}
d3c54d51-817a-11e4-b47d-f7c6b0d2f080,node: {operational=0,suspected=1,leave_seq=-1,view_id=view_id(REG,00000000-0000-0000-0000-000000000000,0),safe_seq=-1,im_range=[-1,-1],}
dd54d291-817a-11e4-8e1e-7e40774101bf,node: {operational=1,suspected=0,leave_seq=-1,view_id=view_id(TRANS,dd54d291-817a-11e4-8e1e-7e40774101bf,0),safe_seq=-1,im_range=[0,-1],}
fa16b484-80f9-11e4-8036-6276f2af149d,node: {operational=0,suspected=1,leave_seq=-1,view_id=view_id(REG,16b39b34-817a-11e4-8633-660ab26d996d,34),safe_seq=139,im_range=[140,139],}
)
},
}
77daa67d-8174-11e4-a3ed-370c72485569,evs::node{operational=1,suspected=0,installed=0,fifo_seq=12395,join_message=
evs::msg{version=0,type=4,user_type=255,order=1,seq=139,seq_range=-1,aru_seq=139,flags=4,source=77daa67d-8174-11e4-a3ed-370c72485569,source_view_id=view_id(REG,16b39b34-817a-11e4-8633-660ab26d996d,34),range_uuid=00000000-0000-0000-0000-000000000000,range=[-1,-1],fifo_seq=12395,node_list=( 16b39b34-817a-11e4-8633-660ab26d996d,node: {operational=0,suspected=1,leave_seq=-1,view_id=view_id(REG,16b39b34-817a-11e4-8633-660ab26d996d,34),safe_seq=139,im_range=[140,139],}
2541ca74-817a-11e4-b1f7-d7f0ddaaac29,node: {operational=0,suspected=1,leave_seq=-1,view_id=view_id(REG,16b39b34-817a-11e4-8633-660ab26d996d,34),safe_seq=139,im_range=[140,139],}
2fc3ecce-817a-11e4-9caa-0705faa8e6f5,node: {operational=0,suspected=1,leave_seq=-1,view_id=view_id(REG,16b39b34-817a-11e4-8633-660ab26d996d,34),safe_seq=139,im_range=[140,139],}
77daa67d-8174-11e4-a3ed-370c72485569,node: {operational=1,suspected=0,leave_seq=-1,view_id=view_id(REG,16b39b34-817a-11e4-8633-660ab26d996d,34),safe_seq=139,im_range=[140,139],}
d3c54d51-817a-11e4-b47d-f7c6b0d2f080,node: {operational=0,suspected=1,leave_seq=-1,view_id=view_id(REG,00000000-0000-0000-0000-000000000000,0),safe_seq=-1,im_range=[-1,-1],}
dd54d291-817a-11e4-8e1e-7e40774101bf,node: {operational=1,suspected=0,leave_seq=-1,view_id=view_id(TRANS,dd54d291-817a-11e4-8e1e-7e40774101bf,0),safe_seq=-1,im_range=[0,-1],}
fa16b484-80f9-11e4-8036-6276f2af149d,node: {operational=1,suspected=0,leave_seq=-1,view_id=view_id(REG,16b39b34-817a-11e4-8633-660ab26d996d,34),safe_seq=139,im_range=[140,139],}
)
},
}
d3c54d51-817a-11e4-b47d-f7c6b0d2f080,evs::node{operational=0,suspected=1,installed=0,fifo_seq=-1,}
dd54d291-817a-11e4-8e1e-7e40774101bf,evs::node{operational=1,suspected=0,installed=0,fifo_seq=-1,join_message=
evs::msg{version=0,type=4,user_type=255,order=1,seq=-1,seq_range=-1,aru_seq=-1,flags=0,source=dd54d291-817a-11e4-8e1e-7e40774101bf,source_view_id=view_id(TRANS,dd54d291-817a-11e4-8e1e-7e40774101bf,0),range_uuid=00000000-0000-0000-0000-000000000000,range=[-1,-1],fifo_seq=31,node_list=( 16b39b34-817a-11e4-8633-660ab26d996d,node: {operational=0,suspected=1,leave_seq=-1,view_id=view_id(REG,00000000-0000-0000-0000-000000000000,0),safe_seq=-1,im_range=[-1,-1],}
2541ca74-817a-11e4-b1f7-d7f0ddaaac29,node: {operational=1,suspected=0,leave_seq=-1,view_id=view_id(REG,16b39b34-817a-11e4-8633-660ab26d996d,34),safe_seq=139,im_range=[140,139],}
2fc3ecce-817a-11e4-9caa-0705faa8e6f5,node: {operational=1,suspected=0,leave_seq=-1,view_id=view_id(REG,16b39b34-817a-11e4-8633-660ab26d996d,34),safe_seq=139,im_range=[140,139],}
77daa67d-8174-11e4-a3ed-370c72485569,node: {operational=1,suspected=0,leave_seq=-1,view_id=view_id(REG,16b39b34-817a-11e4-8633-660ab26d996d,34),safe_seq=139,im_range=[140,139],}
d3c54d51-817a-11e4-b47d-f7c6b0d2f080,node: {operational=0,suspected=1,leave_seq=-1,view_id=view_id(REG,00000000-0000-0000-0000-000000000000,0),safe_seq=-1,im_range=[-1,-1],}
dd54d291-817a-11e4-8e1e-7e40774101bf,node: {operational=1,suspected=0,leave_seq=-1,view_id=view_id(TRANS,dd54d291-817a-11e4-8e1e-7e40774101bf,0),safe_seq=-1,im_range=[0,-1],}
fa16b484-80f9-11e4-8036-6276f2af149d,node: {operational=1,suspected=0,leave_seq=-1,view_id=view_id(REG,16b39b34-817a-11e4-8633-660ab26d996d,34),safe_seq=139,im_range=[140,139],}
)
},
}
fa16b484-80f9-11e4-8036-6276f2af149d,evs::node{operational=1,suspected=0,installed=0,fifo_seq=64007,join_message=
evs::msg{version=0,type=4,user_type=255,order=1,seq=139,seq_range=-1,aru_seq=139,flags=4,source=fa16b484-80f9-11e4-8036-6276f2af149d,source_view_id=view_id(REG,16b39b34-817a-11e4-8633-660ab26d996d,34),range_uuid=00000000-0000-0000-0000-000000000000,range=[-1,-1],fifo_seq=64007,node_list=( 16b39b34-817a-11e4-8633-660ab26d996d,node: {operational=0,suspected=1,leave_seq=-1,view_id=view_id(REG,16b39b34-817a-11e4-8633-660ab26d996d,34),safe_seq=139,im_range=[140,139],}
2541ca74-817a-11e4-b1f7-d7f0ddaaac29,node: {operational=0,suspected=1,leave_seq=-1,view_id=view_id(REG,16b39b34-817a-11e4-8633-660ab26d996d,34),safe_seq=139,im_range=[140,139],}
2fc3ecce-817a-11e4-9caa-0705faa8e6f5,node: {operational=0,suspected=1,leave_seq=-1,view_id=view_id(REG,16b39b34-817a-11e4-8633-660ab26d996d,34),safe_seq=139,im_range=[140,139],}
77daa67d-8174-11e4-a3ed-370c72485569,node: {operational=1,suspected=0,leave_seq=-1,view_id=view_id(REG,16b39b34-817a-11e4-8633-660ab26d996d,34),safe_seq=139,im_range=[140,139],}
d3c54d51-817a-11e4-b47d-f7c6b0d2f080,node: {operational=0,suspected=1,leave_seq=-1,view_id=view_id(REG,00000000-0000-0000-0000-000000000000,0),safe_seq=-1,im_range=[-1,-1],}
dd54d291-817a-11e4-8e1e-7e40774101bf,node: {operational=1,suspected=0,leave_seq=-1,view_id=view_id(TRANS,dd54d291-817a-11e4-8e1e-7e40774101bf,0),safe_seq=-1,im_range=[0,-1],}
fa16b484-80f9-11e4-8036-6276f2af149d,node: {operational=1,suspected=0,leave_seq=-1,view_id=view_id(REG,16b39b34-817a-11e4-8633-660ab26d996d,34),safe_seq=139,im_range=[140,139],}
)
},
}
}
}
141211 21:15:58 [Note] WSREP: no install message received
141211 21:15:58 [Warning] WSREP: no nodes coming from prim view, prim not possible
141211 21:15:58 [Note] WSREP: view(view_id(NON_PRIM,dd54d291-817a-11e4-8e1e-7e40774101bf,1) memb {
dd54d291-817a-11e4-8e1e-7e40774101bf,0
} joined {
} left {
} partitioned {
})
141211 21:15:58 [Warning] WSREP: evs::proto(dd54d291-817a-11e4-8e1e-7e40774101bf, GATHER, view_id(REG,dd54d291-817a-11e4-8e1e-7e40774101bf,1)) source fa16b484-80f9-11e4-8036-6276f2af149d is not supposed to be representative
141211 21:16:01 [Note] WSREP: (dd54d291-817a-11e4-8e1e-7e40774101bf, 'tcp://0.0.0.0:4567') address 'tcp://10.1.0.4:4567' pointing to uuid dd54d291-817a-11e4-8e1e-7e40774101bf is blacklisted, skipping
141211 21:16:01 [Note] WSREP: (dd54d291-817a-11e4-8e1e-7e40774101bf, 'tcp://0.0.0.0:4567') address 'tcp://10.1.0.4:4567' pointing to uuid dd54d291-817a-11e4-8e1e-7e40774101bf is blacklisted, skipping
141211 21:16:01 [Note] WSREP: (dd54d291-817a-11e4-8e1e-7e40774101bf, 'tcp://0.0.0.0:4567') address 'tcp://10.1.0.4:4567' pointing to uuid dd54d291-817a-11e4-8e1e-7e40774101bf is blacklisted, skipping
141211 21:16:01 [Note] WSREP: (dd54d291-817a-11e4-8e1e-7e40774101bf, 'tcp://0.0.0.0:4567') address 'tcp://10.1.0.4:4567' pointing to uuid dd54d291-817a-11e4-8e1e-7e40774101bf is blacklisted, skipping
141211 21:16:03 [Note] WSREP: evs::proto(dd54d291-817a-11e4-8e1e-7e40774101bf, GATHER, view_id(REG,dd54d291-817a-11e4-8e1e-7e40774101bf,1)) suspecting node: d3c54d51-817a-11e4-b47d-f7c6b0d2f080
141211 21:16:03 [Note] WSREP: node 16b39b34-817a-11e4-8633-660ab26d996d marked with nil view id and suspected in all present join messages, declaring inactive
141211 21:16:03 [Note] WSREP: node d3c54d51-817a-11e4-b47d-f7c6b0d2f080 marked with nil view id and suspected in all present join messages, declaring inactive
141211 21:16:03 [Note] WSREP: node 16b39b34-817a-11e4-8633-660ab26d996d marked with nil view id and suspected in all present join messages, declaring inactive
141211 21:16:03 [Note] WSREP: node d3c54d51-817a-11e4-b47d-f7c6b0d2f080 marked with nil view id and suspected in all present join messages, declaring inactive
141211 21:16:04 [Note] WSREP: node 16b39b34-817a-11e4-8633-660ab26d996d marked with nil view id and suspected in all present join messages, declaring inactive
141211 21:16:04 [Note] WSREP: node d3c54d51-817a-11e4-b47d-f7c6b0d2f080 marked with nil view id and suspected in all present join messages, declaring inactive
141211 21:16:05 [Note] WSREP: node 16b39b34-817a-11e4-8633-660ab26d996d marked with nil view id and suspected in all present join messages, declaring inactive
141211 21:16:05 [Note] WSREP: node d3c54d51-817a-11e4-b47d-f7c6b0d2f080 marked with nil view id and suspected in all present join messages, declaring inactive
141211 21:16:05 [Note] WSREP: (dd54d291-817a-11e4-8e1e-7e40774101bf, 'tcp://0.0.0.0:4567') address 'tcp://10.1.0.4:4567' pointing to uuid dd54d291-817a-11e4-8e1e-7e40774101bf is blacklisted, skipping
141211 21:16:05 [Note] WSREP: (dd54d291-817a-11e4-8e1e-7e40774101bf, 'tcp://0.0.0.0:4567') address 'tcp://10.1.0.4:4567' pointing to uuid dd54d291-817a-11e4-8e1e-7e40774101bf is blacklisted, skipping
141211 21:16:05 [Note] WSREP: (dd54d291-817a-11e4-8e1e-7e40774101bf, 'tcp://0.0.0.0:4567') address 'tcp://10.1.0.4:4567' pointing to uuid dd54d291-817a-11e4-8e1e-7e40774101bf is blacklisted, skipping
141211 21:16:05 [Note] WSREP: (dd54d291-817a-11e4-8e1e-7e40774101bf, 'tcp://0.0.0.0:4567') address 'tcp://10.1.0.4:4567' pointing to uuid dd54d291-817a-11e4-8e1e-7e40774101bf is blacklisted, skipping
141211 21:16:06 [Note] WSREP: node 16b39b34-817a-11e4-8633-660ab26d996d marked with nil view id and suspected in all present join messages, declaring inactive
141211 21:16:06 [Note] WSREP: node d3c54d51-817a-11e4-b47d-f7c6b0d2f080 marked with nil view id and suspected in all present join messages, declaring inactive
141211 21:16:07 [Note] WSREP: node 16b39b34-817a-11e4-8633-660ab26d996d marked with nil view id and suspected in all present join messages, declaring inactive
141211 21:16:07 [Note] WSREP: node d3c54d51-817a-11e4-b47d-f7c6b0d2f080 marked with nil view id and suspected in all present join messages, declaring inactive
141211 21:16:08 [Note] WSREP: node 16b39b34-817a-11e4-8633-660ab26d996d marked with nil view id and suspected in all present join messages, declaring inactive
141211 21:16:08 [Note] WSREP: node d3c54d51-817a-11e4-b47d-f7c6b0d2f080 marked with nil view id and suspected in all present join messages, declaring inactive
141211 21:16:08 [Note] WSREP: (dd54d291-817a-11e4-8e1e-7e40774101bf, 'tcp://0.0.0.0:4567') address 'tcp://10.1.0.4:4567' pointing to uuid dd54d291-817a-11e4-8e1e-7e40774101bf is blacklisted, skipping
141211 21:16:08 [Note] WSREP: (dd54d291-817a-11e4-8e1e-7e40774101bf, 'tcp://0.0.0.0:4567') address 'tcp://10.1.0.4:4567' pointing to uuid dd54d291-817a-11e4-8e1e-7e40774101bf is blacklisted, skipping
141211 21:16:08 [Note] WSREP: (dd54d291-817a-11e4-8e1e-7e40774101bf, 'tcp://0.0.0.0:4567') address 'tcp://10.1.0.4:4567' pointing to uuid dd54d291-817a-11e4-8e1e-7e40774101bf is blacklisted, skipping
141211 21:16:08 [Note] WSREP: (dd54d291-817a-11e4-8e1e-7e40774101bf, 'tcp://0.0.0.0:4567') address 'tcp://10.1.0.4:4567' pointing to uuid dd54d291-817a-11e4-8e1e-7e40774101bf is blacklisted, skipping
141211 21:16:08 [Note] WSREP: (dd54d291-817a-11e4-8e1e-7e40774101bf, 'tcp://0.0.0.0:4567') address 'tcp://10.1.0.4:4567' pointing to uuid dd54d291-817a-11e4-8e1e-7e40774101bf is blacklisted, skipping
141211 21:16:08 [Note] WSREP: (dd54d291-817a-11e4-8e1e-7e40774101bf, 'tcp://0.0.0.0:4567') address 'tcp://10.1.0.4:4567' pointing to uuid dd54d291-817a-11e4-8e1e-7e40774101bf is blacklisted, skipping
141211 21:16:08 [Note] WSREP: (dd54d291-817a-11e4-8e1e-7e40774101bf, 'tcp://0.0.0.0:4567') address 'tcp://10.1.0.4:4567' pointing to uuid dd54d291-817a-11e4-8e1e-7e40774101bf is blacklisted, skipping
141211 21:16:08 [Note] WSREP: (dd54d291-817a-11e4-8e1e-7e40774101bf, 'tcp://0.0.0.0:4567') address 'tcp://10.1.0.4:4567' pointing to uuid dd54d291-817a-11e4-8e1e-7e40774101bf is blacklisted, skipping
141211 21:16:08 [Note] WSREP: (dd54d291-817a-11e4-8e1e-7e40774101bf, 'tcp://0.0.0.0:4567') address 'tcp://10.1.0.4:4567' pointing to uuid dd54d291-817a-11e4-8e1e-7e40774101bf is blacklisted, skipping
141211 21:16:08 [Note] WSREP: (dd54d291-817a-11e4-8e1e-7e40774101bf, 'tcp://0.0.0.0:4567') address 'tcp://10.1.0.4:4567' pointing to uuid dd54d291-817a-11e4-8e1e-7e40774101bf is blacklisted, skipping
141211 21:16:08 [Note] WSREP: (dd54d291-817a-11e4-8e1e-7e40774101bf, 'tcp://0.0.0.0:4567') address 'tcp://10.1.0.4:4567' pointing to uuid dd54d291-817a-11e4-8e1e-7e40774101bf is blacklisted, skipping
141211 21:16:08 [Note] WSREP: (dd54d291-817a-11e4-8e1e-7e40774101bf, 'tcp://0.0.0.0:4567') address 'tcp://10.1.0.4:4567' pointing to uuid dd54d291-817a-11e4-8e1e-7e40774101bf is blacklisted, skipping
141211 21:16:08 [Note] WSREP: (dd54d291-817a-11e4-8e1e-7e40774101bf, 'tcp://0.0.0.0:4567') address 'tcp://10.1.0.4:4567' pointing to uuid dd54d291-817a-11e4-8e1e-7e40774101bf is blacklisted, skipping
141211 21:16:08 [Note] WSREP: (dd54d291-817a-11e4-8e1e-7e40774101bf, 'tcp://0.0.0.0:4567') address 'tcp://10.1.0.4:4567' pointing to uuid dd54d291-817a-11e4-8e1e-7e40774101bf is blacklisted, skipping
141211 21:16:08 [Note] WSREP: (dd54d291-817a-11e4-8e1e-7e40774101bf, 'tcp://0.0.0.0:4567') address 'tcp://10.1.0.4:4567' pointing to uuid dd54d291-817a-11e4-8e1e-7e40774101bf is blacklisted, skipping
141211 21:16:08 [Note] WSREP: (dd54d291-817a-11e4-8e1e-7e40774101bf, 'tcp://0.0.0.0:4567') address 'tcp://10.1.0.4:4567' pointing to uuid dd54d291-817a-11e4-8e1e-7e40774101bf is blacklisted, skipping
141211 21:16:08 [Note] WSREP: (dd54d291-817a-11e4-8e1e-7e40774101bf, 'tcp://0.0.0.0:4567') address 'tcp://10.1.0.4:4567' pointing to uuid dd54d291-817a-11e4-8e1e-7e40774101bf is blacklisted, skipping
141211 21:16:08 [Note] WSREP: (dd54d291-817a-11e4-8e1e-7e40774101bf, 'tcp://0.0.0.0:4567') address 'tcp://10.1.0.4:4567' pointing to uuid dd54d291-817a-11e4-8e1e-7e40774101bf is blacklisted, skipping
141211 21:16:08 [Note] WSREP: (dd54d291-817a-11e4-8e1e-7e40774101bf, 'tcp://0.0.0.0:4567') address 'tcp://10.1.0.4:4567' pointing to uuid dd54d291-817a-11e4-8e1e-7e40774101bf is blacklisted, skipping
141211 21:16:08 [Note] WSREP: (dd54d291-817a-11e4-8e1e-7e40774101bf, 'tcp://0.0.0.0:4567') address 'tcp://10.1.0.4:4567' pointing to uuid dd54d291-817a-11e4-8e1e-7e40774101bf is blacklisted, skipping
141211 21:16:08 [Note] WSREP: (dd54d291-817a-11e4-8e1e-7e40774101bf, 'tcp://0.0.0.0:4567') cleaning up duplicate 0x7f140043d680 after established 0x7f140043cc00
141211 21:16:08 [Note] WSREP: (dd54d291-817a-11e4-8e1e-7e40774101bf, 'tcp://0.0.0.0:4567') address 'tcp://10.1.0.4:4567' pointing to uuid dd54d291-817a-11e4-8e1e-7e40774101bf is blacklisted, skipping
141211 21:16:08 [Note] WSREP: (dd54d291-817a-11e4-8e1e-7e40774101bf, 'tcp://0.0.0.0:4567') address 'tcp://10.1.0.4:4567' pointing to uuid dd54d291-817a-11e4-8e1e-7e40774101bf is blacklisted, skipping
141211 21:16:08 [Note] WSREP: (dd54d291-817a-11e4-8e1e-7e40774101bf, 'tcp://0.0.0.0:4567') address 'tcp://10.1.0.4:4567' pointing to uuid dd54d291-817a-11e4-8e1e-7e40774101bf is blacklisted, skipping
141211 21:16:08 [Note] WSREP: (dd54d291-817a-11e4-8e1e-7e40774101bf, 'tcp://0.0.0.0:4567') address 'tcp://10.1.0.4:4567' pointing to uuid dd54d291-817a-11e4-8e1e-7e40774101bf is blacklisted, skipping
141211 21:16:08 [Note] WSREP: (dd54d291-817a-11e4-8e1e-7e40774101bf, 'tcp://0.0.0.0:4567') address 'tcp://10.1.0.4:4567' pointing to uuid dd54d291-817a-11e4-8e1e-7e40774101bf is blacklisted, skipping
141211 21:16:08 [Note] WSREP: (dd54d291-817a-11e4-8e1e-7e40774101bf, 'tcp://0.0.0.0:4567') address 'tcp://10.1.0.4:4567' pointing to uuid dd54d291-817a-11e4-8e1e-7e40774101bf is blacklisted, skipping
141211 21:16:08 [Note] WSREP: (dd54d291-817a-11e4-8e1e-7e40774101bf, 'tcp://0.0.0.0:4567') address 'tcp://10.1.0.4:4567' pointing to uuid dd54d291-817a-11e4-8e1e-7e40774101bf is blacklisted, skipping
141211 21:16:08 [Note] WSREP: (dd54d291-817a-11e4-8e1e-7e40774101bf, 'tcp://0.0.0.0:4567') address 'tcp://10.1.0.4:4567' pointing to uuid dd54d291-817a-11e4-8e1e-7e40774101bf is blacklisted, skipping
141211 21:16:08 [Note] WSREP: (dd54d291-817a-11e4-8e1e-7e40774101bf, 'tcp://0.0.0.0:4567') address 'tcp://10.1.0.4:4567' pointing to uuid dd54d291-817a-11e4-8e1e-7e40774101bf is blacklisted, skipping
141211 21:16:09 [Note] WSREP: node 16b39b34-817a-11e4-8633-660ab26d996d marked with nil view id and suspected in all present join messages, declaring inactive
141211 21:16:09 [Note] WSREP: node d3c54d51-817a-11e4-b47d-f7c6b0d2f080 marked with nil view id and suspected in all present join messages, declaring inactive
141211 21:16:10 [Note] WSREP: node 16b39b34-817a-11e4-8633-660ab26d996d marked with nil view id and suspected in all present join messages, declaring inactive
141211 21:16:10 [Note] WSREP: node d3c54d51-817a-11e4-b47d-f7c6b0d2f080 marked with nil view id and suspected in all present join messages, declaring inactive
141211 21:16:11 [Note] WSREP: node 16b39b34-817a-11e4-8633-660ab26d996d marked with nil view id and suspected in all present join messages, declaring inactive
141211 21:16:11 [Note] WSREP: node d3c54d51-817a-11e4-b47d-f7c6b0d2f080 marked with nil view id and suspected in all present join messages, declaring inactive
141211 21:16:12 [Note] WSREP: node 16b39b34-817a-11e4-8633-660ab26d996d marked with nil view id and suspected in all present join messages, declaring inactive
141211 21:16:12 [Note] WSREP: node d3c54d51-817a-11e4-b47d-f7c6b0d2f080 marked with nil view id and suspected in all present join messages, declaring inactive
141211 21:16:13 [Warning] WSREP: evs::proto(dd54d291-817a-11e4-8e1e-7e40774101bf, GATHER, view_id(REG,dd54d291-817a-11e4-8e1e-7e40774101bf,1)) install timer expired
evs::proto(evs::proto(dd54d291-817a-11e4-8e1e-7e40774101bf, GATHER, view_id(REG,dd54d291-817a-11e4-8e1e-7e40774101bf,1)), GATHER) {
current_view=view(view_id(REG,dd54d291-817a-11e4-8e1e-7e40774101bf,1) memb {
dd54d291-817a-11e4-8e1e-7e40774101bf,0
} joined {
} left {
} partitioned {
}),
input_map=evs::input_map: {aru_seq=0,safe_seq=0,node_index=node: {idx=0,range=[1,0],safe_seq=0} },
fifo_seq=63,
last_sent=0,
known={
16b39b34-817a-11e4-8633-660ab26d996d,evs::node{operational=0,suspected=1,installed=0,fifo_seq=-1,}
2541ca74-817a-11e4-b1f7-d7f0ddaaac29,evs::node{operational=0,suspected=1,installed=0,fifo_seq=-1,}
2fc3ecce-817a-11e4-9caa-0705faa8e6f5,evs::node{operational=0,suspected=1,installed=0,fifo_seq=-1,}
77daa67d-8174-11e4-a3ed-370c72485569,evs::node{operational=0,suspected=1,installed=0,fifo_seq=-1,}
d3c54d51-817a-11e4-b47d-f7c6b0d2f080,evs::node{operational=0,suspected=1,installed=0,fifo_seq=-1,}
dd54d291-817a-11e4-8e1e-7e40774101bf,evs::node{operational=1,suspected=0,installed=0,fifo_seq=-1,join_message=
evs::msg{version=0,type=4,user_type=255,order=1,seq=0,seq_range=-1,aru_seq=0,flags=0,source=dd54d291-817a-11e4-8e1e-7e40774101bf,source_view_id=view_id(REG,dd54d291-817a-11e4-8e1e-7e40774101bf,1),range_uuid=00000000-0000-0000-0000-000000000000,range=[-1,-1],fifo_seq=63,node_list=( 16b39b34-817a-11e4-8633-660ab26d996d,node: {operational=0,suspected=1,leave_seq=-1,view_id=view_id(REG,00000000-0000-0000-0000-000000000000,0),safe_seq=-1,im_range=[-1,-1],}
2541ca74-817a-11e4-b1f7-d7f0ddaaac29,node: {operational=0,suspected=1,leave_seq=-1,view_id=view_id(REG,00000000-0000-0000-0000-000000000000,0),safe_seq=-1,im_range=[-1,-1],}
2fc3ecce-817a-11e4-9caa-0705faa8e6f5,node: {operational=0,suspected=1,leave_seq=-1,view_id=view_id(REG,00000000-0000-0000-0000-000000000000,0),safe_seq=-1,im_range=[-1,-1],}
77daa67d-8174-11e4-a3ed-370c72485569,node: {operational=0,suspected=1,leave_seq=-1,view_id=view_id(REG,00000000-0000-0000-0000-000000000000,0),safe_seq=-1,im_range=[-1,-1],}
d3c54d51-817a-11e4-b47d-f7c6b0d2f080,node: {operational=0,suspected=1,leave_seq=-1,view_id=view_id(REG,00000000-0000-0000-0000-000000000000,0),safe_seq=-1,im_range=[-1,-1],}
dd54d291-817a-11e4-8e1e-7e40774101bf,node: {operational=1,suspected=0,leave_seq=-1,view_id=view_id(REG,dd54d291-817a-11e4-8e1e-7e40774101bf,1),safe_seq=0,im_range=[1,0],}
fa16b484-80f9-11e4-8036-6276f2af149d,node: {operational=1,suspected=0,leave_seq=-1,view_id=view_id(REG,2541ca74-817a-11e4-b1f7-d7f0ddaaac29,36),safe_seq=0,im_range=[1,0],}
)
},
}
fa16b484-80f9-11e4-8036-6276f2af149d,evs::node{operational=1,suspected=0,installed=0,fifo_seq=64051,join_message=
evs::msg{version=0,type=4,user_type=255,order=1,seq=0,seq_range=-1,aru_seq=0,flags=4,source=fa16b484-80f9-11e4-8036-6276f2af149d,source_view_id=view_id(REG,2541ca74-817a-11e4-b1f7-d7f0ddaaac29,36),range_uuid=00000000-0000-0000-0000-000000000000,range=[-1,-1],fifo_seq=64051,node_list=( 16b39b34-817a-11e4-8633-660ab26d996d,node: {operational=0,suspected=1,leave_seq=-1,view_id=view_id(REG,00000000-0000-0000-0000-000000000000,0),safe_seq=-1,im_range=[-1,-1],}
2541ca74-817a-11e4-b1f7-d7f0ddaaac29,node: {operational=1,suspected=0,leave_seq=-1,view_id=view_id(REG,2541ca74-817a-11e4-b1f7-d7f0ddaaac29,36),safe_seq=0,im_range=[1,0],}
2fc3ecce-817a-11e4-9caa-0705faa8e6f5,node: {operational=1,suspected=0,leave_seq=-1,view_id=view_id(REG,2541ca74-817a-11e4-b1f7-d7f0ddaaac29,36),safe_seq=0,im_range=[1,0],}
77daa67d-8174-11e4-a3ed-370c72485569,node: {operational=1,suspected=0,leave_seq=-1,view_id=view_id(REG,2541ca74-817a-11e4-b1f7-d7f0ddaaac29,36),safe_seq=0,im_range=[1,0],}
d3c54d51-817a-11e4-b47d-f7c6b0d2f080,node: {operational=0,suspected=1,leave_seq=-1,view_id=view_id(REG,00000000-0000-0000-0000-000000000000,0),safe_seq=-1,im_range=[-1,-1],}
dd54d291-817a-11e4-8e1e-7e40774101bf,node: {operational=1,suspected=0,leave_seq=-1,view_id=view_id(REG,dd54d291-817a-11e4-8e1e-7e40774101bf,1),safe_seq=0,im_range=[1,0],}
fa16b484-80f9-11e4-8036-6276f2af149d,node: {operational=1,suspected=0,leave_seq=-1,view_id=view_id(REG,2541ca74-817a-11e4-b1f7-d7f0ddaaac29,36),safe_seq=0,im_range=[1,0],}
)
},
}
}
}
141211 21:16:13 [Note] WSREP: no install message received
141211 21:16:13 [Warning] WSREP: subsequent views have same members, prev view view(view_id(REG,dd54d291-817a-11e4-8e1e-7e40774101bf,1) memb {
dd54d291-817a-11e4-8e1e-7e40774101bf,0
} joined {
} left {
} partitioned {
}) current view view(view_id(REG,dd54d291-817a-11e4-8e1e-7e40774101bf,2) memb {
dd54d291-817a-11e4-8e1e-7e40774101bf,0
} joined {
} left {
} partitioned {
})
141211 21:16:13 [Warning] WSREP: no nodes coming from prim view, prim not possible
141211 21:16:13 [Note] WSREP: view(view_id(NON_PRIM,dd54d291-817a-11e4-8e1e-7e40774101bf,2) memb {
dd54d291-817a-11e4-8e1e-7e40774101bf,0
} joined {
} left {
} partitioned {
})
141211 21:16:14 [ERROR] WSREP: failed to open gcomm backend connection: 110: failed to reach primary view: 110 (Connection timed out)
at gcomm/src/pc.cpp:connect():141
141211 21:16:14 [ERROR] WSREP: gcs/src/gcs_core.c:gcs_core_open():202: Failed to open backend connection: -110 (Connection timed out)
141211 21:16:14 [ERROR] WSREP: gcs/src/gcs.c:gcs_open():1291: Failed to open channel 'my_wsrep_cluster' at 'gcomm://10.0.1.5,10.0.1.10,10.0.1.11,10.0.1.20,10.1.0.4': -110 (Connection timed out)
141211 21:16:14 [ERROR] WSREP: gcs connect failed: Connection timed out
141211 21:16:14 [ERROR] WSREP: wsrep::connect() failed: 7
141211 21:16:14 [ERROR] Aborting
141211 21:16:14 [Note] WSREP: Service disconnected.
141211 21:16:15 [Note] WSREP: Some threads may fail to exit.
141211 21:16:15 [Note] /usr/sbin/mysqld: Shutdown complete
Hi,
Today something strange started happening to my Galera cluster - nodes would simply terminate with the following error:
My cluster:
Galera 25.3.5(r178) MariaDB 35.2 Percona XtraDB 5.5.38 Running in Docker
And some of the nodes periodically fail to join cluster, ever with Galera cache reset (clean run):
Extended log:
I would also get errors like this:
If I restart cluster, it would work for around 20 mins, after which a similar error occurs.
And, more logs from another node: