Closed kaojunsong closed 3 years ago
hasancelik
commented
3 years ago Hi @kaojunsong,
Could you share application/hazelcast logs? Also could you give more information about your setup like hazelcast config, framework(spring, quarkus etc), deployment(helm charts or handcrafted yamls)?
kaojunsong
commented
3 years ago {"@timestamp":"2020-11-04T05:18:31.377+00:00","message":"[192.168.1.92]:5701 [girl] [3.11.5] Cannot suspect [192.168.1.88]:5701, since it's not a member.","logger_name":"c.h.internal.cluster.ClusterService","thread_name":"hz._hzInstance_1_girl.cached.thread-5","level":"DEBUG","caller_class_name":"com.hazelcast.logging.Slf4jFactory$Slf4jLogger","caller_method_name":"log","caller_file_name":"Slf4jFactory.java","caller_line_number":63,"service":{"name":"space-ingest-ingest"},"trace":"","span":"","parent":"","pid":"1"}
{"@timestamp":"2020-11-04T05:18:31.378+00:00","message":"[192.168.1.92]:5701 [girl] [3.11.5] Connecting to /192.168.1.86:5701, timeout: 0, bind-any: true","logger_name":"com.hazelcast.nio.tcp.TcpIpConnector","thread_name":"hz._hzInstance_1_girl.cached.thread-5","level":"INFO","caller_class_name":"com.hazelcast.logging.Slf4jFactory$Slf4jLogger","caller_method_name":"log","caller_file_name":"Slf4jFactory.java","caller_line_number":65,"service":{"name":"space-ingest-ingest"},"trace":"","span":"","parent":"","pid":"1"}
{"@timestamp":"2020-11-04T05:18:31.380+00:00","message":"[192.168.1.92]:5701 [girl] [3.11.5] Could not connect to: /192.168.1.86:5701. Reason: SocketException[Connection refused to address /192.168.1.86:5701]","logger_name":"com.hazelcast.nio.tcp.TcpIpConnector","thread_name":"hz._hzInstance_1_girl.cached.thread-5","level":"INFO","caller_class_name":"com.hazelcast.logging.Slf4jFactory$Slf4jLogger","caller_method_name":"log","caller_file_name":"Slf4jFactory.java","caller_line_number":65,"service":{"name":"space-ingest-ingest"},"trace":"","span":"","parent":"","pid":"1"}
{"@timestamp":"2020-11-04T05:18:31.383+00:00","message":"[192.168.1.92]:5701 [girl] [3.11.5] Received heartbeat from Member [192.168.2.63]:5701 - f9b01bd0-4a9c-4b05-91d5-8a477d371dd9 (now: 2020-11-04 05:18:31.380, timestamp: 2020-11-04 05:18:31.319)","logger_name":"c.h.i.c.impl.ClusterHeartbeatManager","thread_name":"hz._hzInstance_1_girl.generic-operation.thread-1","level":"DEBUG","caller_class_name":"com.hazelcast.logging.Slf4jFactory$Slf4jLogger","caller_method_name":"log","caller_file_name":"Slf4jFactory.java","caller_line_number":63,"service":{"name":"space-ingest-ingest"},"trace":"","span":"","parent":"","pid":"1"}
{"@timestamp":"2020-11-04T05:18:31.388+00:00","message":"[192.168.1.92]:5701 [girl] [3.11.5] Connecting to /192.168.1.86:5701, timeout: 0, bind-any: true","logger_name":"com.hazelcast.nio.tcp.TcpIpConnector","thread_name":"hz._hzInstance_1_girl.cached.thread-11","level":"INFO","caller_class_name":"com.hazelcast.logging.Slf4jFactory$Slf4jLogger","caller_method_name":"log","caller_file_name":"Slf4jFactory.java","caller_line_number":65,"service":{"name":"space-ingest-ingest"},"trace":"","span":"","parent":"","pid":"1"}
{"@timestamp":"2020-11-04T05:18:31.388+00:00","message":"[192.168.1.92]:5701 [girl] [3.11.5] Could not connect to: /192.168.1.86:5701. Reason: SocketException[Connection refused to address /192.168.1.86:5701]","logger_name":"com.hazelcast.nio.tcp.TcpIpConnector","thread_name":"hz._hzInstance_1_girl.cached.thread-11","level":"INFO","caller_class_name":"com.hazelcast.logging.Slf4jFactory$Slf4jLogger","caller_method_name":"log","caller_file_name":"Slf4jFactory.java","caller_line_number":65,"service":{"name":"space-ingest-ingest"},"trace":"","span":"","parent":"","pid":"1"}
{"@timestamp":"2020-11-04T05:18:31.392+00:00","message":"[192.168.1.92]:5701 [girl] [3.11.5] Connecting to /192.168.1.86:5701, timeout: 0, bind-any: true","logger_name":"com.hazelcast.nio.tcp.TcpIpConnector","thread_name":"hz._hzInstance_1_girl.cached.thread-6","level":"INFO","caller_class_name":"com.hazelcast.logging.Slf4jFactory$Slf4jLogger","caller_method_name":"log","caller_file_name":"Slf4jFactory.java","caller_line_number":65,"service":{"name":"space-ingest-ingest"},"trace":"","span":"","parent":"","pid":"1"}
{"@timestamp":"2020-11-04T05:18:31.393+00:00","message":"[192.168.1.92]:5701 [girl] [3.11.5] Could not connect to: /192.168.1.86:5701. Reason: SocketException[Connection refused to address /192.168.1.86:5701]","logger_name":"com.hazelcast.nio.tcp.TcpIpConnector","thread_name":"hz._hzInstance_1_girl.cached.thread-6","level":"INFO","caller_class_name":"com.hazelcast.logging.Slf4jFactory$Slf4jLogger","caller_method_name":"log","caller_file_name":"Slf4jFactory.java","caller_line_number":65,"service":{"name":"space-ingest-ingest"},"trace":"","span":"","parent":"","pid":"1"}
{"@timestamp":"2020-11-04T05:18:31.406+00:00","message":"[192.168.1.92]:5701 [girl] [3.11.5] Connecting to /192.168.1.86:5701, timeout: 0, bind-any: true","logger_name":"com.hazelcast.nio.tcp.TcpIpConnector","thread_name":"hz._hzInstance_1_girl.cached.thread-12","level":"INFO","caller_class_name":"com.hazelcast.logging.Slf4jFactory$Slf4jLogger","caller_method_name":"log","caller_file_name":"Slf4jFactory.java","caller_line_number":65,"service":{"name":"space-ingest-ingest"},"trace":"","span":"","parent":"","pid":"1"}
{"@timestamp":"2020-11-04T05:18:31.407+00:00","message":"[192.168.1.92]:5701 [girl] [3.11.5] Could not connect to: /192.168.1.86:5701. Reason: SocketException[Connection refused to address /192.168.1.86:5701]","logger_name":"com.hazelcast.nio.tcp.TcpIpConnector","thread_name":"hz._hzInstance_1_girl.cached.thread-12","level":"INFO","caller_class_name":"com.hazelcast.logging.Slf4jFactory$Slf4jLogger","caller_method_name":"log","caller_file_name":"Slf4jFactory.java","caller_line_number":65,"service":{"name":"space-ingest-ingest"},"trace":"","span":"","parent":"","pid":"1"}
{"@timestamp":"2020-11-04T05:18:31.415+00:00","message":"[192.168.1.92]:5701 [girl] [3.11.5] Received heartbeat from Member [192.168.1.87]:5701 - b85f7b70-b84d-4120-888f-52db008a2c35 (now: 2020-11-04 05:18:31.412, timestamp: 2020-11-04 05:18:31.408)","logger_name":"c.h.i.c.impl.ClusterHeartbeatManager","thread_name":"hz._hzInstance_1_girl.generic-operation.thread-0","level":"DEBUG","caller_class_name":"com.hazelcast.logging.Slf4jFactory$Slf4jLogger","caller_method_name":"log","caller_file_name":"Slf4jFactory.java","caller_line_number":63,"service":{"name":"space-ingest-ingest"},"trace":"","span":"","parent":"","pid":"1"}
{"@timestamp":"2020-11-04T05:18:31.420+00:00","message":"[192.168.1.92]:5701 [girl] [3.11.5] Committing/rolling-back live transactions of [192.168.1.88]:5701, UUID: 877c3592-fa57-48df-8981-61d79a2d4278","logger_name":"c.h.t.TransactionManagerService","thread_name":"hz._hzInstance_1_girl.cached.thread-8","level":"INFO","caller_class_name":"com.hazelcast.logging.Slf4jFactory$Slf4jLogger","caller_method_name":"log","caller_file_name":"Slf4jFactory.java","caller_line_number":65,"service":{"name":"space-ingest-ingest"},"trace":"","span":"","parent":"","pid":"1"}
{"@timestamp":"2020-11-04T05:18:31.428+00:00","message":"[192.168.1.92]:5701 [girl] [3.11.5] Connecting to /192.168.1.86:5701, timeout: 0, bind-any: true","logger_name":"com.hazelcast.nio.tcp.TcpIpConnector","thread_name":"hz._hzInstance_1_girl.cached.thread-12","level":"INFO","caller_class_name":"com.hazelcast.logging.Slf4jFactory$Slf4jLogger","caller_method_name":"log","caller_file_name":"Slf4jFactory.java","caller_line_number":65,"service":{"name":"space-ingest-ingest"},"trace":"","span":"","parent":"","pid":"1"}
{"@timestamp":"2020-11-04T05:18:31.430+00:00","message":"[192.168.1.92]:5701 [girl] [3.11.5] Could not connect to: /192.168.1.86:5701. Reason: SocketException[Connection refused to address /192.168.1.86:5701]","logger_name":"com.hazelcast.nio.tcp.TcpIpConnector","thread_name":"hz._hzInstance_1_girl.cached.thread-12","level":"INFO","caller_class_name":"com.hazelcast.logging.Slf4jFactory$Slf4jLogger","caller_method_name":"log","caller_file_name":"Slf4jFactory.java","caller_line_number":65,"service":{"name":"space-ingest-ingest"},"trace":"","span":"","parent":"","pid":"1"}
{"@timestamp":"2020-11-04T05:18:31.431+00:00","message":"[192.168.1.92]:5701 [girl] [3.11.5] Removing connection to endpoint [192.168.1.86]:5701 Cause => java.net.SocketException {Connection refused to address /192.168.1.86:5701}, Error-Count: 5","logger_name":"c.h.nio.tcp.TcpIpConnectionErrorHandler","thread_name":"hz._hzInstance_1_girl.cached.thread-12","level":"WARN","caller_class_name":"com.hazelcast.logging.Slf4jFactory$Slf4jLogger","caller_method_name":"log","caller_file_name":"Slf4jFactory.java","caller_line_number":67,"service":{"name":"space-ingest-ingest"},"trace":"","span":"","parent":"","pid":"1"}
{"@timestamp":"2020-11-04T05:18:31.431+00:00","message":"[192.168.1.92]:5701 [girl] [3.11.5] Member [192.168.1.86]:5701 - d3bd085a-c842-4a9d-88dc-255dc3baabec is suspected to be dead for reason: No connection","logger_name":"c.h.i.cluster.impl.MembershipManager","thread_name":"hz._hzInstance_1_girl.cached.thread-12","level":"WARN","caller_class_name":"com.hazelcast.logging.Slf4jFactory$Slf4jLogger","caller_method_name":"log","caller_file_name":"Slf4jFactory.java","caller_line_number":67,"service":{"name":"space-ingest-ingest"},"trace":"","span":"","parent":"","pid":"1"}
{"@timestamp":"2020-11-04T05:18:31.432+00:00","message":"[192.168.1.92]:5701 [girl] [3.11.5] Connecting to /192.168.1.86:5701, timeout: 0, bind-any: true","logger_name":"com.hazelcast.nio.tcp.TcpIpConnector","thread_name":"hz._hzInstance_1_girl.cached.thread-13","level":"INFO","caller_class_name":"com.hazelcast.logging.Slf4jFactory$Slf4jLogger","caller_method_name":"log","caller_file_name":"Slf4jFactory.java","caller_line_number":65,"service":{"name":"space-ingest-ingest"},"trace":"","span":"","parent":"","pid":"1"}
{"@timestamp":"2020-11-04T05:18:31.436+00:00","message":"[192.168.1.92]:5701 [girl] [3.11.5] Could not connect to: /192.168.1.86:5701. Reason: SocketException[Connection refused to address /192.168.1.86:5701]","logger_name":"com.hazelcast.nio.tcp.TcpIpConnector","thread_name":"hz._hzInstance_1_girl.cached.thread-13","level":"INFO","caller_class_name":"com.hazelcast.logging.Slf4jFactory$Slf4jLogger","caller_method_name":"log","caller_file_name":"Slf4jFactory.java","caller_line_number":65,"service":{"name":"space-ingest-ingest"},"trace":"","span":"","parent":"","pid":"1"}
{"@timestamp":"2020-11-04T05:18:31.469+00:00","message":"[192.168.1.92]:5701 [girl] [3.11.5] Connecting to /192.168.1.86:5701, timeout: 0, bind-any: true","logger_name":"com.hazelcast.nio.tcp.TcpIpConnector","thread_name":"hz._hzInstance_1_girl.cached.thread-5","level":"INFO","caller_class_name":"com.hazelcast.logging.Slf4jFactory$Slf4jLogger","caller_method_name":"log","caller_file_name":"Slf4jFactory.java","caller_line_number":65,"service":{"name":"space-ingest-ingest"},"trace":"","span":"","parent":"","pid":"1"}
{"@timestamp":"2020-11-04T05:18:31.471+00:00","message":"[192.168.1.92]:5701 [girl] [3.11.5] Could not connect to: /192.168.1.86:5701. Reason: SocketException[Connection refused to address /192.168.1.86:5701]","logger_name":"com.hazelcast.nio.tcp.TcpIpConnector","thread_name":"hz._hzInstance_1_girl.cached.thread-5","level":"INFO","caller_class_name":"com.hazelcast.logging.Slf4jFactory$Slf4jLogger","caller_method_name":"log","caller_file_name":"Slf4jFactory.java","caller_line_number":65,"service":{"name":"space-ingest-ingest"},"trace":"","span":"","parent":"","pid":"1"}
{"@timestamp":"2020-11-04T05:18:31.475+00:00","message":"[192.168.1.92]:5701 [girl] [3.11.5] Setting members [Member [192.168.1.85]:5701 - aea7d31f-e1b7-421e-a18e-7ac97b8456c0, Member [192.168.2.57]:5701 - 7369eaee-cac5-4055-af06-78c2479f3b4f, Member [192.168.2.61]:5701 - b1db58f9-02c3-4b4c-bcae-7119e1bb1b64, Member [192.168.2.63]:5701 - f9b01bd0-4a9c-4b05-91d5-8a477d371dd9, Member [192.168.2.60]:5701 - 619ccee8-3314-4a8d-aa97-8b2a651e029a, Member [192.168.2.65]:5701 - 546ea0f3-2afd-4218-8011-2a229a6a8e5c, Member [192.168.2.56]:5701 - 3147e528-d699-4b4b-95ed-5ee3b5cc6336, Member [192.168.2.62]:5701 - d29a4bf1-6bee-4780-b737-86211cdb23df, Member [192.168.2.58]:5701 - 0be947d8-8ab3-4f5a-bc1a-92ee5e407148, Member [192.168.2.64]:5701 - 66dcfb4f-630e-42a2-b6fa-01b34cc161ea, Member [192.168.2.55]:5701 - bab0860e-c47b-4017-b14f-f34cacc56c17, Member [192.168.1.83]:5701 - c2a3e4a8-5f6b-4b50-b545-b668acf713e1, Member [192.168.1.91]:5701 - 0a9550ce-65e9-4499-8280-3a8a9a1aa83f, Member [192.168.1.82]:5701 - a99b0e9d-391e-4c36-b42e-ca54997fe4ca, Member [192.168.1.87]:5701 - b85f7b70-b84d-4120-888f-52db008a2c35, Member [192.168.1.84]:5701 - 54def7b7-aa66-4e07-96a9-01f89721158c, Member [192.168.1.92]:5701 - 65330a23-c7e8-4a22-bf8b-f7ad11327aa3 this, Member [192.168.1.90]:5701 - 7f37ac8e-210e-4d51-a603-4a879da94d4a, Member [192.168.1.1]:5703 - 68a415fc-1207-4f3c-906b-f0ba47371585, Member [192.168.1.1]:5702 - 31076b6b-fa13-4021-ac13-95b6cd209736, Member [192.168.1.1]:5704 - 9f118b1f-769b-41f0-b08e-32e1966bc5ca], version: 241","logger_name":"c.h.i.cluster.impl.MembershipManager","thread_name":"hz._hzInstance_1_girl.generic-operation.thread-1","level":"DEBUG","caller_class_name":"com.hazelcast.logging.Slf4jFactory$Slf4jLogger","caller_method_name":"log","caller_file_name":"Slf4jFactory.java","caller_line_number":63,"service":{"name":"space-ingest-ingest"},"trace":"","span":"","parent":"","pid":"1"}
{"@timestamp":"2020-11-04T05:18:31.475+00:00","message":"[192.168.1.92]:5701 [girl] [3.11.5] Removing suspected address [192.168.1.86]:5701, it's no longer a member.","logger_name":"c.h.i.cluster.impl.MembershipManager","thread_name":"hz._hzInstance_1_girl.generic-operation.thread-1","level":"DEBUG","caller_class_name":"com.hazelcast.logging.Slf4jFactory$Slf4jLogger","caller_method_name":"log","caller_file_name":"Slf4jFactory.java","caller_line_number":63,"service":{"name":"space-ingest-ingest"},"trace":"","span":"","parent":"","pid":"1"}
{"@timestamp":"2020-11-04T05:18:31.475+00:00","message":"[192.168.1.92]:5701 [girl] [3.11.5] Removing Member [192.168.1.86]:5701 - d3bd085a-c842-4a9d-88dc-255dc3baabec","logger_name":"c.h.i.partition.InternalPartitionService","thread_name":"hz._hzInstance_1_girl.generic-operation.thread-1","level":"DEBUG","caller_class_name":"com.hazelcast.logging.Slf4jFactory$Slf4jLogger","caller_method_name":"log","caller_file_name":"Slf4jFactory.java","caller_line_number":63,"service":{"name":"space-ingest-ingest"},"trace":"","span":"","parent":"","pid":"1"}
{"@timestamp":"2020-11-04T05:18:31.476+00:00","message":"[192.168.1.92]:5701 [girl] [3.11.5] \n\nMembers {size:21, ver:241} [\n\tMember [192.168.1.85]:5701 - aea7d31f-e1b7-421e-a18e-7ac97b8456c0\n\tMember [192.168.2.57]:5701 - 7369eaee-cac5-4055-af06-78c2479f3b4f\n\tMember [192.168.2.61]:5701 - b1db58f9-02c3-4b4c-bcae-7119e1bb1b64\n\tMember [192.168.2.63]:5701 - f9b01bd0-4a9c-4b05-91d5-8a477d371dd9\n\tMember [192.168.2.60]:5701 - 619ccee8-3314-4a8d-aa97-8b2a651e029a\n\tMember [192.168.2.65]:5701 - 546ea0f3-2afd-4218-8011-2a229a6a8e5c\n\tMember [192.168.2.56]:5701 - 3147e528-d699-4b4b-95ed-5ee3b5cc6336\n\tMember [192.168.2.62]:5701 - d29a4bf1-6bee-4780-b737-86211cdb23df\n\tMember [192.168.2.58]:5701 - 0be947d8-8ab3-4f5a-bc1a-92ee5e407148\n\tMember [192.168.2.64]:5701 - 66dcfb4f-630e-42a2-b6fa-01b34cc161ea\n\tMember [192.168.2.55]:5701 - bab0860e-c47b-4017-b14f-f34cacc56c17\n\tMember [192.168.1.83]:5701 - c2a3e4a8-5f6b-4b50-b545-b668acf713e1\n\tMember [192.168.1.91]:5701 - 0a9550ce-65e9-4499-8280-3a8a9a1aa83f\n\tMember [192.168.1.82]:5701 - a99b0e9d-391e-4c36-b42e-ca54997fe4ca\n\tMember [192.168.1.87]:5701 - b85f7b70-b84d-4120-888f-52db008a2c35\n\tMember [192.168.1.84]:5701 - 54def7b7-aa66-4e07-96a9-01f89721158c\n\tMember [192.168.1.92]:5701 - 65330a23-c7e8-4a22-bf8b-f7ad11327aa3 this\n\tMember [192.168.1.90]:5701 - 7f37ac8e-210e-4d51-a603-4a879da94d4a\n\tMember [192.168.1.1]:5703 - 68a415fc-1207-4f3c-906b-f0ba47371585\n\tMember [192.168.1.1]:5702 - 31076b6b-fa13-4021-ac13-95b6cd209736\n\tMember [192.168.1.1]:5704 - 9f118b1f-769b-41f0-b08e-32e1966bc5ca\n]\n","logger_name":"c.h.internal.cluster.ClusterService","thread_name":"hz._hzInstance_1_girl.generic-operation.thread-1","level":"INFO","caller_class_name":"com.hazelcast.logging.Slf4jFactory$Slf4jLogger","caller_method_name":"log","caller_file_name":"Slf4jFactory.java","caller_line_number":65,"service":{"name":"space-ingest-ingest"},"trace":"","span":"","parent":"","pid":"1"}
{"@timestamp":"2020-11-04T05:18:31.476+00:00","message":"[192.168.1.92]:5701 [girl] [3.11.5] Committing/rolling-back live transactions of [192.168.1.86]:5701, UUID: d3bd085a-c842-4a9d-88dc-255dc3baabec","logger_name":"c.h.t.TransactionManagerService","thread_name":"hz._hzInstance_1_girl.cached.thread-11","level":"INFO","caller_class_name":"com.hazelcast.logging.Slf4jFactory$Slf4jLogger","caller_method_name":"log","caller_file_name":"Slf4jFactory.java","caller_line_number":65,"service":{"name":"space-ingest-ingest"},"trace":"","span":"","parent":"","pid":"1"}
{"@timestamp":"2020-11-04T05:18:31.476+00:00","message":"[192.168.1.92]:5701 [girl] [3.11.5] Following unknown addresses are found in partition table sent from master[[192.168.1.85]:5701]. (Probably they have recently joined or left the cluster.) {\n\t[192.168.1.86]:5701\n}","logger_name":"c.h.i.partition.InternalPartitionService","thread_name":"hz._hzInstance_1_girl.priority-generic-operation.thread-0","level":"WARN","caller_class_name":"com.hazelcast.logging.Slf4jFactory$Slf4jLogger","caller_method_name":"log","caller_file_name":"Slf4jFactory.java","caller_line_number":67,"service":{"name":"space-ingest-ingest"},"trace":"","span":"","parent":"","pid":"1"}
{"@timestamp":"2020-11-04T05:18:31.477+00:00","message":"[192.168.1.92]:5701 [girl] [3.11.5] Received heartbeat from Member [192.168.1.1]:5704 - 9f118b1f-769b-41f0-b08e-32e1966bc5ca (now: 2020-11-04 05:18:31.474, timestamp: 2020-11-04 05:18:31.470)","logger_name":"c.h.i.c.impl.ClusterHeartbeatManager","thread_name":"hz._hzInstance_1_girl.generic-operation.thread-0","level":"DEBUG","caller_class_name":"com.hazelcast.logging.Slf4jFactory$Slf4jLogger","caller_method_name":"log","caller_file_name":"Slf4jFactory.java","caller_line_number":63,"service":{"name":"space-ingest-ingest"},"trace":"","span":"","parent":"","pid":"1"}
{"@timestamp":"2020-11-04T05:18:31.477+00:00","message":"[192.168.1.92]:5701 [girl] [3.11.5] Applied new partition state. Version: 2231, caller: [192.168.1.85]:5701","logger_name":"c.h.i.p.o.PartitionStateOperation","thread_name":"hz._hzInstance_1_girl.priority-generic-operation.thread-0","level":"DEBUG","caller_class_name":"com.hazelcast.logging.Slf4jFactory$Slf4jLogger","caller_method_name":"log","caller_file_name":"Slf4jFactory.java","caller_line_number":63,"service":{"name":"space-ingest-ingest"},"trace":"","span":"","parent":"","pid":"1"}
{"@timestamp":"2020-11-04T05:18:31.483+00:00","message":"[192.168.1.92]:5701 [girl] [3.11.5] Received heartbeat from Member [192.168.1.1]:5702 - 31076b6b-fa13-4021-ac13-95b6cd209736 (now: 2020-11-04 05:18:31.480, timestamp: 2020-11-04 05:18:31.469)","logger_name":"c.h.i.c.impl.ClusterHeartbeatManager","thread_name":"hz._hzInstance_1_girl.generic-operation.thread-0","level":"DEBUG","caller_class_name":"com.hazelcast.logging.Slf4jFactory$Slf4jLogger","caller_method_name":"log","caller_file_name":"Slf4jFactory.java","caller_line_number":63,"service":{"name":"space-ingest-ingest"},"trace":"","span":"","parent":"","pid":"1"}
{"@timestamp":"2020-11-04T05:18:31.490+00:00","message":"[192.168.1.92]:5701 [girl] [3.11.5] Received heartbeat from Member [192.168.1.1]:5703 - 68a415fc-1207-4f3c-906b-f0ba47371585 (now: 2020-11-04 05:18:31.487, timestamp: 2020-11-04 05:18:31.471)","logger_name":"c.h.i.c.impl.ClusterHeartbeatManager","thread_name":"hz._hzInstance_1_girl.generic-operation.thread-1","level":"DEBUG","caller_class_name":"com.hazelcast.logging.Slf4jFactory$Slf4jLogger","caller_method_name":"log","caller_file_name":"Slf4jFactory.java","caller_line_number":63,"service":{"name":"space-ingest-ingest"},"trace":"","span":"","parent":"","pid":"1"}
{"@timestamp":"2020-11-04T05:18:31.504+00:00","message":"[192.168.1.92]:5701 [girl] [3.11.5] Received heartbeat from Member [192.168.2.60]:5701 - 619ccee8-3314-4a8d-aa97-8b2a651e029a (now: 2020-11-04 05:18:31.500, timestamp: 2020-11-04 05:18:31.478)","logger_name":"c.h.i.c.impl.ClusterHeartbeatManager","thread_name":"hz._hzInstance_1_girl.generic-operation.thread-0","level":"DEBUG","caller_class_name":"com.hazelcast.logging.Slf4jFactory$Slf4jLogger","caller_method_name":"log","caller_file_name":"Slf4jFactory.java","caller_line_number":63,"service":{"name":"space-ingest-ingest"},"trace":"","span":"","parent":"","pid":"1"}
{"@timestamp":"2020-11-04T05:18:31.504+00:00","message":"[192.168.1.92]:5701 [girl] [3.11.5] Received heartbeat from Member [192.168.2.61]:5701 - b1db58f9-02c3-4b4c-bcae-7119e1bb1b64 (now: 2020-11-04 05:18:31.501, timestamp: 2020-11-04 05:18:31.480)","logger_name":"c.h.i.c.impl.ClusterHeartbeatManager","thread_name":"hz._hzInstance_1_girl.generic-operation.thread-0","level":"DEBUG","caller_class_name":"com.hazelcast.logging.Slf4jFactory$Slf4jLogger","caller_method_name":"log","caller_file_name":"Slf4jFactory.java","caller_line_number":63,"service":{"name":"space-ingest-ingest"},"trace":"","span":"","parent":"","pid":"1"}
{"@timestamp":"2020-11-04T05:18:31.504+00:00","message":"[192.168.1.92]:5701 [girl] [3.11.5] Received heartbeat from Member [192.168.1.82]:5701 - a99b0e9d-391e-4c36-b42e-ca54997fe4ca (now: 2020-11-04 05:18:31.501, timestamp: 2020-11-04 05:18:31.480)","logger_name":"c.h.i.c.impl.ClusterHeartbeatManager","thread_name":"hz._hzInstance_1_girl.generic-operation.thread-0","level":"DEBUG","caller_class_name":"com.hazelcast.logging.Slf4jFactory$Slf4jLogger","caller_method_name":"log","caller_file_name":"Slf4jFactory.java","caller_line_number":63,"service":{"name":"space-ingest-ingest"},"trace":"","span":"","parent":"","pid":"1"}
{"@timestamp":"2020-11-04T05:18:31.504+00:00","message":"[192.168.1.92]:5701 [girl] [3.11.5] Received heartbeat from Member [192.168.2.56]:5701 - 3147e528-d699-4b4b-95ed-5ee3b5cc6336 (now: 2020-11-04 05:18:31.501, timestamp: 2020-11-04 05:18:31.496)","logger_name":"c.h.i.c.impl.ClusterHeartbeatManager","thread_name":"hz._hzInstance_1_girl.generic-operation.thread-0","level":"DEBUG","caller_class_name":"com.hazelcast.logging.Slf4jFactory$Slf4jLogger","caller_method_name":"log","caller_file_name":"Slf4jFactory.java","caller_line_number":63,"service":{"name":"space-ingest-ingest"},"trace":"","span":"","parent":"","pid":"1"}
{"@timestamp":"2020-11-04T05:18:31.504+00:00","message":"[192.168.1.92]:5701 [girl] [3.11.5] Received a periodic member update, ignoring... Version: 241","logger_name":"c.h.internal.cluster.ClusterService","thread_name":"hz._hzInstance_1_girl.generic-operation.thread-0","level":"DEBUG","caller_class_name":"com.hazelcast.logging.Slf4jFactory$Slf4jLogger","caller_method_name":"log","caller_file_name":"Slf4jFactory.java","caller_line_number":63,"service":{"name":"space-ingest-ingest"},"trace":"","span":"","parent":"","pid":"1"}
{"@timestamp":"2020-11-04T05:18:31.504+00:00","message":"[192.168.1.92]:5701 [girl] [3.11.5] Received heartbeat from Member [192.168.1.84]:5701 - 54def7b7-aa66-4e07-96a9-01f89721158c (now: 2020-11-04 05:18:31.501, timestamp: 2020-11-04 05:18:31.485)","logger_name":"c.h.i.c.impl.ClusterHeartbeatManager","thread_name":"hz._hzInstance_1_girl.generic-operation.thread-0","level":"DEBUG","caller_class_name":"com.hazelcast.logging.Slf4jFactory$Slf4jLogger","caller_method_name":"log","caller_file_name":"Slf4jFactory.java","caller_line_number":63,"service":{"name":"space-ingest-ingest"},"trace":"","span":"","parent":"","pid":"1"}
{"@timestamp":"2020-11-04T05:18:31.504+00:00","message":"[192.168.1.92]:5701 [girl] [3.11.5] Received heartbeat from Member [192.168.2.55]:5701 - bab0860e-c47b-4017-b14f-f34cacc56c17 (now: 2020-11-04 05:18:31.501, timestamp: 2020-11-04 05:18:31.490)","logger_name":"c.h.i.c.impl.ClusterHeartbeatManager","thread_name":"hz._hzInstance_1_girl.generic-operation.thread-0","level":"DEBUG","caller_class_name":"com.hazelcast.logging.Slf4jFactory$Slf4jLogger","caller_method_name":"log","caller_file_name":"Slf4jFactory.java","caller_line_number":63,"service":{"name":"space-ingest-ingest"},"trace":"","span":"","parent":"","pid":"1"}
{"@timestamp":"2020-11-04T05:18:31.505+00:00","message":"[192.168.1.92]:5701 [girl] [3.11.5] Received heartbeat from Member [192.168.1.83]:5701 - c2a3e4a8-5f6b-4b50-b545-b668acf713e1 (now: 2020-11-04 05:18:31.502, timestamp: 2020-11-04 05:18:31.482)","logger_name":"c.h.i.c.impl.ClusterHeartbeatManager","thread_name":"hz._hzInstance_1_girl.generic-operation.thread-0","level":"DEBUG","caller_class_name":"com.hazelcast.logging.Slf4jFactory$Slf4jLogger","caller_method_name":"log","caller_file_name":"Slf4jFactory.java","caller_line_number":63,"service":{"name":"space-ingest-ingest"},"trace":"","span":"","parent":"","pid":"1"}
{"@timestamp":"2020-11-04T05:18:31.505+00:00","message":"[192.168.1.92]:5701 [girl] [3.11.5] Received heartbeat from Member [192.168.2.62]:5701 - d29a4bf1-6bee-4780-b737-86211cdb23df (now: 2020-11-04 05:18:31.502, timestamp: 2020-11-04 05:18:31.485)","logger_name":"c.h.i.c.impl.ClusterHeartbeatManager","thread_name":"hz._hzInstance_1_girl.generic-operation.thread-0","level":"DEBUG","caller_class_name":"com.hazelcast.logging.Slf4jFactory$Slf4jLogger","caller_method_name":"log","caller_file_name":"Slf4jFactory.java","caller_line_number":63,"service":{"name":"space-ingest-ingest"},"trace":"","span":"","parent":"","pid":"1"}
{"@timestamp":"2020-11-04T05:18:31.505+00:00","message":"[192.168.1.92]:5701 [girl] [3.11.5] Received heartbeat from Member [192.168.1.87]:5701 - b85f7b70-b84d-4120-888f-52db008a2c35 (now: 2020-11-04 05:18:31.502, timestamp: 2020-11-04 05:18:31.477)","logger_name":"c.h.i.c.impl.ClusterHeartbeatManager","thread_name":"hz._hzInstance_1_girl.generic-operation.thread-0","level":"DEBUG","caller_class_name":"com.hazelcast.logging.Slf4jFactory$Slf4jLogger","caller_method_name":"log","caller_file_name":"Slf4jFactory.java","caller_line_number":63,"service":{"name":"space-ingest-ingest"},"trace":"","span":"","parent":"","pid":"1"}
{"@timestamp":"2020-11-04T05:18:31.505+00:00","message":"[192.168.1.92]:5701 [girl] [3.11.5] Received heartbeat from Member [192.168.2.57]:5701 - 7369eaee-cac5-4055-af06-78c2479f3b4f (now: 2020-11-04 05:18:31.502, timestamp: 2020-11-04 05:18:31.498)","logger_name":"c.h.i.c.impl.ClusterHeartbeatManager","thread_name":"hz._hzInstance_1_girl.generic-operation.thread-0","level":"DEBUG","caller_class_name":"com.hazelcast.logging.Slf4jFactory$Slf4jLogger","caller_method_name":"log","caller_file_name":"Slf4jFactory.java","caller_line_number":63,"service":{"name":"space-ingest-ingest"},"trace":"","span":"","parent":"","pid":"1"}
{"@timestamp":"2020-11-04T05:18:31.516+00:00","message":"[192.168.1.92]:5701 [girl] [3.11.5] Received heartbeat from Member [192.168.2.64]:5701 - 66dcfb4f-630e-42a2-b6fa-01b34cc161ea (now: 2020-11-04 05:18:31.513, timestamp: 2020-11-04 05:18:31.511)","logger_name":"c.h.i.c.impl.ClusterHeartbeatManager","thread_name":"hz._hzInstance_1_girl.generic-operation.thread-0","level":"DEBUG","caller_class_name":"com.hazelcast.logging.Slf4jFactory$Slf4jLogger","caller_method_name":"log","caller_file_name":"Slf4jFactory.java","caller_line_number":63,"service":{"name":"space-ingest-ingest"},"trace":"","span":"","parent":"","pid":"1"}
{"@timestamp":"2020-11-04T05:18:31.517+00:00","message":"[192.168.1.92]:5701 [girl] [3.11.5] Received heartbeat from Member [192.168.1.90]:5701 - 7f37ac8e-210e-4d51-a603-4a879da94d4a (now: 2020-11-04 05:18:31.514, timestamp: 2020-11-04 05:18:31.481)","logger_name":"c.h.i.c.impl.ClusterHeartbeatManager","thread_name":"hz._hzInstance_1_girl.generic-operation.thread-0","level":"DEBUG","caller_class_name":"com.hazelcast.logging.Slf4jFactory$Slf4jLogger","caller_method_name":"log","caller_file_name":"Slf4jFactory.java","caller_line_number":63,"service":{"name":"space-ingest-ingest"},"trace":"","span":"","parent":"","pid":"1"}
{"@timestamp":"2020-11-04T05:18:31.517+00:00","message":"[192.168.1.92]:5701 [girl] [3.11.5] Received heartbeat from Member [192.168.2.58]:5701 - 0be947d8-8ab3-4f5a-bc1a-92ee5e407148 (now: 2020-11-04 05:18:31.514, timestamp: 2020-11-04 05:18:31.472)","logger_name":"c.h.i.c.impl.ClusterHeartbeatManager","thread_name":"hz._hzInstance_1_girl.generic-operation.thread-1","level":"DEBUG","caller_class_name":"com.hazelcast.logging.Slf4jFactory$Slf4jLogger","caller_method_name":"log","caller_file_name":"Slf4jFactory.java","caller_line_number":63,"service":{"name":"space-ingest-ingest"},"trace":"","span":"","parent":"","pid":"1"}
{"@timestamp":"2020-11-04T05:18:31.517+00:00","message":"[192.168.1.92]:5701 [girl] [3.11.5] Received heartbeat from Member [192.168.2.65]:5701 - 546ea0f3-2afd-4218-8011-2a229a6a8e5c (now: 2020-11-04 05:18:31.514, timestamp: 2020-11-04 05:18:31.502)","logger_name":"c.h.i.c.impl.ClusterHeartbeatManager","thread_name":"hz._hzInstance_1_girl.priority-generic-operation.thread-0","level":"DEBUG","caller_class_name":"com.hazelcast.logging.Slf4jFactory$Slf4jLogger","caller_method_name":"log","caller_file_name":"Slf4jFactory.java","caller_line_number":63,"service":{"name":"space-ingest-ingest"},"trace":"","span":"","parent":"","pid":"1"}
{"@timestamp":"2020-11-04T05:18:31.560+00:00","message":"[192.168.1.92]:5701 [girl] [3.11.5] Received heartbeat from Member [192.168.2.63]:5701 - f9b01bd0-4a9c-4b05-91d5-8a477d371dd9 (now: 2020-11-04 05:18:31.557, timestamp: 2020-11-04 05:18:31.542)","logger_name":"c.h.i.c.impl.ClusterHeartbeatManager","thread_name":"hz._hzInstance_1_girl.priority-generic-operation.thread-0","level":"DEBUG","caller_class_name":"com.hazelcast.logging.Slf4jFactory$Slf4jLogger","caller_method_name":"log","caller_file_name":"Slf4jFactory.java","caller_line_number":63,"service":{"name":"space-ingest-ingest"},"trace":"","span":"","parent":"","pid":"1"}
{"@timestamp":"2020-11-04T05:18:31.762+00:00","message":"[192.168.1.92]:5701 [girl] [3.11.5] Received heartbeat from Member [192.168.2.61]:5701 - b1db58f9-02c3-4b4c-bcae-7119e1bb1b64 (now: 2020-11-04 05:18:31.759, timestamp: 2020-11-04 05:18:31.760)","logger_name":"c.h.i.c.impl.ClusterHeartbeatManager","thread_name":"hz._hzInstance_1_girl.generic-operation.thread-1","level":"DEBUG","caller_class_name":"com.hazelcast.logging.Slf4jFactory$Slf4jLogger","caller_method_name":"log","caller_file_name":"Slf4jFactory.java","caller_line_number":63,"service":{"name":"space-ingest-ingest"},"trace":"","span":"","parent":"","pid":"1"}
{"@timestamp":"2020-11-04T05:18:31.846+00:00","message":"[192.168.1.92]:5701 [girl] [3.11.5] Received heartbeat from Member [192.168.2.60]:5701 - 619ccee8-3314-4a8d-aa97-8b2a651e029a (now: 2020-11-04 05:18:31.843, timestamp: 2020-11-04 05:18:31.843)","logger_name":"c.h.i.c.impl.ClusterHeartbeatManager","thread_name":"hz._hzInstance_1_girl.priority-generic-operation.thread-0","level":"DEBUG","caller_class_name":"com.hazelcast.logging.Slf4jFactory$Slf4jLogger","caller_method_name":"log","caller_file_name":"Slf4jFactory.java","caller_line_number":63,"service":{"name":"space-ingest-ingest"},"trace":"","span":"","parent":"","pid":"1"}
{"@timestamp":"2020-11-04T05:18:31.853+00:00","message":"[192.168.1.92]:5701 [girl] [3.11.5] Invocations:1 timeouts:0 backup-timeouts:0","logger_name":"c.h.s.i.o.impl.InvocationMonitor","thread_name":"hz._hzInstance_1_girl.InvocationMonitorThread","level":"DEBUG","caller_class_name":"com.hazelcast.logging.Slf4jFactory$Slf4jLogger","caller_method_name":"log","caller_file_name":"Slf4jFactory.java","caller_line_number":63,"service":{"name":"space-ingest-ingest"},"trace":"","span":"","parent":"","pid":"1"}
{"@timestamp":"2020-11-04T05:18:31.857+00:00","message":"[192.168.1.92]:5701 [girl] [3.11.5] Received heartbeat from Member [192.168.2.57]:5701 - 7369eaee-cac5-4055-af06-78c2479f3b4f (now: 2020-11-04 05:18:31.854, timestamp: 2020-11-04 05:18:31.851)","logger_name":"c.h.i.c.impl.ClusterHeartbeatManager","thread_name":"hz._hzInstance_1_girl.generic-operation.thread-1","level":"DEBUG","caller_class_name":"com.hazelcast.logging.Slf4jFactory$Slf4jLogger","caller_method_name":"log","caller_file_name":"Slf4jFactory.java","caller_line_number":63,"service":{"name":"space-ingest-ingest"},"trace":"","span":"","parent":"","pid":"1"}
{"@timestamp":"2020-11-04T05:18:32.071+00:00","message":"[192.168.1.92]:5701 [girl] [3.11.5] Received heartbeat from Member [192.168.2.62]:5701 - d29a4bf1-6bee-4780-b737-86211cdb23df (now: 2020-11-04 05:18:32.067, timestamp: 2020-11-04 05:18:32.069)","logger_name":"c.h.i.c.impl.ClusterHeartbeatManager","thread_name":"hz._hzInstance_1_girl.priority-generic-operation.thread-0","level":"DEBUG","caller_class_name":"com.hazelcast.logging.Slf4jFactory$Slf4jLogger","caller_method_name":"log","caller_file_name":"Slf4jFactory.java","caller_line_number":63,"service":{"name":"space-ingest-ingest"},"trace":"","span":"","parent":"","pid":"1"}
{"@timestamp":"2020-11-04T05:18:32.150+00:00","message":"[192.168.1.92]:5701 [girl] [3.11.5] Received heartbeat from Member [192.168.1.1]:5703 - 68a415fc-1207-4f3c-906b-f0ba47371585 (now: 2020-11-04 05:18:32.147, timestamp: 2020-11-04 05:18:32.147)","logger_name":"c.h.i.c.impl.ClusterHeartbeatManager","thread_name":"hz._hzInstance_1_girl.generic-operation.thread-1","level":"DEBUG","caller_class_name":"com.hazelcast.logging.Slf4jFactory$Slf4jLogger","caller_method_name":"log","caller_file_name":"Slf4jFactory.java","caller_line_number":63,"service":{"name":"space-ingest-ingest"},"trace":"","span":"","parent":"","pid":"1"}
{"@timestamp":"2020-11-04T05:18:32.549+00:00","message":"[192.168.1.92]:5701 [girl] [3.11.5] Received heartbeat from Member [192.168.2.65]:5701 - 546ea0f3-2afd-4218-8011-2a229a6a8e5c (now: 2020-11-04 05:18:32.546, timestamp: 2020-11-04 05:18:32.546)","logger_name":"c.h.i.c.impl.ClusterHeartbeatManager","thread_name":"hz._hzInstance_1_girl.priority-generic-operation.thread-0","level":"DEBUG","caller_class_name":"com.hazelcast.logging.Slf4jFactory$Slf4jLogger","caller_method_name":"log","caller_file_name":"Slf4jFactory.java","caller_line_number":63,"service":{"name":"space-ingest-ingest"},"trace":"","span":"","parent":"","pid":"1"}
{"@timestamp":"2020-11-04T05:18:32.853+00:00","message":"[192.168.1.92]:5701 [girl] [3.11.5] Invocations:1 timeouts:0 backup-timeouts:0","logger_name":"c.h.s.i.o.impl.InvocationMonitor","thread_name":"hz._hzInstance_1_girl.InvocationMonitorThread","level":"DEBUG","caller_class_name":"com.hazelcast.logging.Slf4jFactory$Slf4jLogger","caller_method_name":"log","caller_file_name":"Slf4jFactory.java","caller_line_number":63,"service":{"name":"space-ingest-ingest"},"trace":"","span":"","parent":"","pid":"1"}
{"@timestamp":"2020-11-04T05:18:33.011+00:00","message":"[192.168.1.92]:5701 [girl] [3.11.5] Received heartbeat from Member [192.168.2.58]:5701 - 0be947d8-8ab3-4f5a-bc1a-92ee5e407148 (now: 2020-11-04 05:18:33.008, timestamp: 2020-11-04 05:18:33.007)","logger_name":"c.h.i.c.impl.ClusterHeartbeatManager","thread_name":"hz._hzInstance_1_girl.priority-generic-operation.thread-0","level":"DEBUG","caller_class_name":"com.hazelcast.logging.Slf4jFactory$Slf4jLogger","caller_method_name":"log","caller_file_name":"Slf4jFactory.java","caller_line_number":63,"service":{"name":"space-ingest-ingest"},"trace":"","span":"","parent":"","pid":"1"}
{"@timestamp":"2020-11-04T05:18:33.265+00:00","message":"[192.168.1.92]:5701 [girl] [3.11.5] Received heartbeat from Member [192.168.2.63]:5701 - f9b01bd0-4a9c-4b05-91d5-8a477d371dd9 (now: 2020-11-04 05:18:33.262, timestamp: 2020-11-04 05:18:33.262)","logger_name":"c.h.i.c.impl.ClusterHeartbeatManager","thread_name":"hz._hzInstance_1_girl.generic-operation.thread-0","level":"DEBUG","caller_class_name":"com.hazelcast.logging.Slf4jFactory$Slf4jLogger","caller_method_name":"log","caller_file_name":"Slf4jFactory.java","caller_line_number":63,"service":{"name":"space-ingest-ingest"},"trace":"","span":"","parent":"","pid":"1"}
{"@timestamp":"2020-11-04T05:18:33.423+00:00","message":"[192.168.1.92]:5701 [girl] [3.11.5] Received heartbeat from Member [192.168.1.83]:5701 - c2a3e4a8-5f6b-4b50-b545-b668acf713e1 (now: 2020-11-04 05:18:33.419, timestamp: 2020-11-04 05:18:33.419)","logger_name":"c.h.i.c.impl.ClusterHeartbeatManager","thread_name":"hz._hzInstance_1_girl.generic-operation.thread-1","level":"DEBUG","caller_class_name":"com.hazelcast.logging.Slf4jFactory$Slf4jLogger","caller_method_name":"log","caller_file_name":"Slf4jFactory.java","caller_line_number":63,"service":{"name":"space-ingest-ingest"},"trace":"","span":"","parent":"","pid":"1"}
{"@timestamp":"2020-11-04T05:18:33.853+00:00","message":"[192.168.1.92]:5701 [girl] [3.11.5] Invocations:1 timeouts:0 backup-timeouts:0","logger_name":"c.h.s.i.o.impl.InvocationMonitor","thread_name":"hz._hzInstance_1_girl.InvocationMonitorThread","level":"DEBUG","caller_class_name":"com.hazelcast.logging.Slf4jFactory$Slf4jLogger","caller_method_name":"log","caller_file_name":"Slf4jFactory.java","caller_line_number":63,"service":{"name":"space-ingest-ingest"},"trace":"","span":"","parent":"","pid":"1"}
{"@timestamp":"2020-11-04T05:18:34.035+00:00","message":"[192.168.1.92]:5701 [girl] [3.11.5] Received heartbeat from Member [192.168.1.1]:5704 - 9f118b1f-769b-41f0-b08e-32e1966bc5ca (now: 2020-11-04 05:18:34.032, timestamp: 2020-11-04 05:18:34.031)","logger_name":"c.h.i.c.impl.ClusterHeartbeatManager","thread_name":"hz._hzInstance_1_girl.priority-generic-operation.thread-0","level":"DEBUG","caller_class_name":"com.hazelcast.logging.Slf4jFactory$Slf4jLogger","caller_method_name":"log","caller_file_name":"Slf4jFactory.java","caller_line_number":63,"service":{"name":"space-ingest-ingest"},"trace":"","span":"","parent":"","pid":"1"}
{"@timestamp":"2020-11-04T05:18:34.347+00:00","message":"Saw local status change event StatusChangeEvent [timestamp=1604467114347, current=DOWN, previous=UP]","logger_name":"com.netflix.discovery.DiscoveryClient","thread_name":"Thread-11","level":"WARN","caller_class_name":"com.netflix.discovery.DiscoveryClient$3","caller_method_name":"notify","caller_file_name":"DiscoveryClient.java","caller_line_number":1321,"service":{"name":"space-ingest-ingest"},"trace":"","span":"","parent":"","pid":"1"}
{"@timestamp":"2020-11-04T05:18:34.392+00:00","message":"[192.168.1.92]:5701 [girl] [3.11.5] Running shutdown hook... Current state: ACTIVE","logger_name":"com.hazelcast.instance.Node","thread_name":"hz.ShutdownThread","level":"INFO","caller_class_name":"com.hazelcast.logging.Slf4jFactory$Slf4jLogger","caller_method_name":"log","caller_file_name":"Slf4jFactory.java","caller_line_number":65,"service":{"name":"space-ingest-ingest"},"trace":"","span":"","parent":"","pid":"1"}
{"@timestamp":"2020-11-04T05:18:34.392+00:00","message":"[192.168.1.92]:5701 [girl] [3.11.5] [192.168.1.92]:5701 is SHUTTING_DOWN","logger_name":"com.hazelcast.core.LifecycleService","thread_name":"hz.ShutdownThread","level":"INFO","caller_class_name":"com.hazelcast.logging.Slf4jFactory$Slf4jLogger","caller_method_name":"log","caller_file_name":"Slf4jFactory.java","caller_line_number":65,"service":{"name":"space-ingest-ingest"},"trace":"","span":"","parent":"","pid":"1"}
{"@timestamp":"2020-11-04T05:18:34.394+00:00","message":"[192.168.1.92]:5701 [girl] [3.11.5] Terminating forcefully...","logger_name":"com.hazelcast.instance.Node","thread_name":"hz.ShutdownThread","level":"WARN","caller_class_name":"com.hazelcast.logging.Slf4jFactory$Slf4jLogger","caller_method_name":"log","caller_file_name":"Slf4jFactory.java","caller_line_number":67,"service":{"name":"space-ingest-ingest"},"trace":"","span":"","parent":"","pid":"1"}
{"@timestamp":"2020-11-04T05:18:34.394+00:00","message":"[192.168.1.92]:5701 [girl] [3.11.5] Setting master address to null","logger_name":"c.h.internal.cluster.ClusterService","thread_name":"hz.ShutdownThread","level":"DEBUG","caller_class_name":"com.hazelcast.logging.Slf4jFactory$Slf4jLogger","caller_method_name":"log","caller_file_name":"Slf4jFactory.java","caller_line_number":63,"service":{"name":"space-ingest-ingest"},"trace":"","span":"","parent":"","pid":"1"}
{"@timestamp":"2020-11-04T05:18:34.394+00:00","message":"[192.168.1.92]:5701 [girl] [3.11.5] Shutting down connection manager...","logger_name":"com.hazelcast.instance.Node","thread_name":"hz.ShutdownThread","level":"INFO","caller_class_name":"com.hazelcast.logging.Slf4jFactory$Slf4jLogger","caller_method_name":"log","caller_file_name":"Slf4jFactory.java","caller_line_number":65,"service":{"name":"space-ingest-ingest"},"trace":"","span":"","parent":"","pid":"1"}
{"@timestamp":"2020-11-04T05:18:34.394+00:00","message":"[192.168.1.92]:5701 [girl] [3.11.5] Connection[id=2, /192.168.1.92:5701->/192.168.1.91:39822, endpoint=[192.168.1.91]:5701, alive=false, type=MEMBER] closed. Reason: TcpIpConnectionManager is stopping","logger_name":"com.hazelcast.nio.tcp.TcpIpConnection","thread_name":"hz.ShutdownThread","level":"INFO","caller_class_name":"com.hazelcast.logging.Slf4jFactory$Slf4jLogger","caller_method_name":"log","caller_file_name":"Slf4jFactory.java","caller_line_number":65,"service":{"name":"space-ingest-ingest"},"trace":"","span":"","parent":"","pid":"1"}
{"@timestamp":"2020-11-04T05:18:34.397+00:00","message":"[192.168.1.92]:5701 [girl] [3.11.5] Connection[id=13, /192.168.1.92:53842->/192.168.1.90:5701, endpoint=[192.168.1.90]:5701, alive=false, type=MEMBER] closed. Reason: TcpIpConnectionManager is stopping","logger_name":"com.hazelcast.nio.tcp.TcpIpConnection","thread_name":"hz.ShutdownThread","level":"INFO","caller_class_name":"com.hazelcast.logging.Slf4jFactory$Slf4jLogger","caller_method_name":"log","caller_file_name":"Slf4jFactory.java","caller_line_number":65,"service":{"name":"space-ingest-ingest"},"trace":"","span":"","parent":"","pid":"1"}
{"@timestamp":"2020-11-04T05:18:34.399+00:00","message":"[192.168.1.92]:5701 [girl] [3.11.5] Connection[id=19, /192.168.1.92:48680->/192.168.2.64:5701, endpoint=[192.168.2.64]:5701, alive=false, type=MEMBER] closed. Reason: TcpIpConnectionManager is stopping","logger_name":"com.hazelcast.nio.tcp.TcpIpConnection","thread_name":"hz.ShutdownThread","level":"INFO","caller_class_name":"com.hazelcast.logging.Slf4jFactory$Slf4jLogger","caller_method_name":"log","caller_file_name":"Slf4jFactory.java","caller_line_number":65,"service":{"name":"space-ingest-ingest"},"trace":"","span":"","parent":"","pid":"1"}
{"@timestamp":"2020-11-04T05:18:34.399+00:00","message":"[192.168.1.92]:5701 [girl] [3.11.5] Connection[id=12, /192.168.1.92:59668->/192.168.2.65:5701, endpoint=[192.168.2.65]:5701, alive=false, type=MEMBER] closed. Reason: TcpIpConnectionManager is stopping","logger_name":"com.hazelcast.nio.tcp.TcpIpConnection","thread_name":"hz.ShutdownThread","level":"INFO","caller_class_name":"com.hazelcast.logging.Slf4jFactory$Slf4jLogger","caller_method_name":"log","caller_file_name":"Slf4jFactory.java","caller_line_number":65,"service":{"name":"space-ingest-ingest"},"trace":"","span":"","parent":"","pid":"1"}
{"@timestamp":"2020-11-04T05:18:34.399+00:00","message":"[192.168.1.92]:5701 [girl] [3.11.5] Connection[id=16, /192.168.1.92:52559->/192.168.2.60:5701, endpoint=[192.168.2.60]:5701, alive=false, type=MEMBER] closed. Reason: TcpIpConnectionManager is stopping","logger_name":"com.hazelcast.nio.tcp.TcpIpConnection","thread_name":"hz.ShutdownThread","level":"INFO","caller_class_name":"com.hazelcast.logging.Slf4jFactory$Slf4jLogger","caller_method_name":"log","caller_file_name":"Slf4jFactory.java","caller_line_number":65,"service":{"name":"space-ingest-ingest"},"trace":"","span":"","parent":"","pid":"1"}
{"@timestamp":"2020-11-04T05:18:34.400+00:00","message":"[192.168.1.92]:5701 [girl] [3.11.5] Connection[id=8, /192.168.1.92:52616->/192.168.2.61:5701, endpoint=[192.168.2.61]:5701, alive=false, type=MEMBER] closed. Reason: TcpIpConnectionManager is stopping","logger_name":"com.hazelcast.nio.tcp.TcpIpConnection","thread_name":"hz.ShutdownThread","level":"INFO","caller_class_name":"com.hazelcast.logging.Slf4jFactory$Slf4jLogger","caller_method_name":"log","caller_file_name":"Slf4jFactory.java","caller_line_number":65,"service":{"name":"space-ingest-ingest"},"trace":"","span":"","parent":"","pid":"1"}
{"@timestamp":"2020-11-04T05:18:34.400+00:00","message":"[192.168.1.92]:5701 [girl] [3.11.5] Connection[id=6, /192.168.1.92:44655->/192.168.2.62:5701, endpoint=[192.168.2.62]:5701, alive=false, type=MEMBER] closed. Reason: TcpIpConnectionManager is stopping","logger_name":"com.hazelcast.nio.tcp.TcpIpConnection","thread_name":"hz.ShutdownThread","level":"INFO","caller_class_name":"com.hazelcast.logging.Slf4jFactory$Slf4jLogger","caller_method_name":"log","caller_file_name":"Slf4jFactory.java","caller_line_number":65,"service":{"name":"space-ingest-ingest"},"trace":"","span":"","parent":"","pid":"1"}
{"@timestamp":"2020-11-04T05:18:34.400+00:00","message":"[192.168.1.92]:5701 [girl] [3.11.5] Connection[id=9, /192.168.1.92:38702->/192.168.2.63:5701, endpoint=[192.168.2.63]:5701, alive=false, type=MEMBER] closed. Reason: TcpIpConnectionManager is stopping","logger_name":"com.hazelcast.nio.tcp.TcpIpConnection","thread_name":"hz.ShutdownThread","level":"INFO","caller_class_name":"com.hazelcast.logging.Slf4jFactory$Slf4jLogger","caller_method_name":"log","caller_file_name":"Slf4jFactory.java","caller_line_number":65,"service":{"name":"space-ingest-ingest"},"trace":"","span":"","parent":"","pid":"1"}
{"@timestamp":"2020-11-04T05:18:34.401+00:00","message":"[192.168.1.92]:5701 [girl] [3.11.5] Connection[id=14, /192.168.1.92:51068->/192.168.2.57:5701, endpoint=[192.168.2.57]:5701, alive=false, type=MEMBER] closed. Reason: TcpIpConnectionManager is stopping","logger_name":"com.hazelcast.nio.tcp.TcpIpConnection","thread_name":"hz.ShutdownThread","level":"INFO","caller_class_name":"com.hazelcast.logging.Slf4jFactory$Slf4jLogger","caller_method_name":"log","caller_file_name":"Slf4jFactory.java","caller_line_number":65,"service":{"name":"space-ingest-ingest"},"trace":"","span":"","parent":"","pid":"1"}
{"@timestamp":"2020-11-04T05:18:34.401+00:00","message":"[192.168.1.92]:5701 [girl] [3.11.5] Connection[id=11, /192.168.1.92:42822->/192.168.2.58:5701, endpoint=[192.168.2.58]:5701, alive=false, type=MEMBER] closed. Reason: TcpIpConnectionManager is stopping","logger_name":"com.hazelcast.nio.tcp.TcpIpConnection","thread_name":"hz.ShutdownThread","level":"INFO","caller_class_name":"com.hazelcast.logging.Slf4jFactory$Slf4jLogger","caller_method_name":"log","caller_file_name":"Slf4jFactory.java","caller_line_number":65,"service":{"name":"space-ingest-ingest"},"trace":"","span":"","parent":"","pid":"1"}
{"@timestamp":"2020-11-04T05:18:34.401+00:00","message":"[192.168.1.92]:5701 [girl] [3.11.5] Connection[id=24, /192.168.1.92:5701->/192.168.1.1:45878, endpoint=[192.168.1.1]:5704, alive=false, type=MEMBER] closed. Reason: TcpIpConnectionManager is stopping","logger_name":"com.hazelcast.nio.tcp.TcpIpConnection","thread_name":"hz.ShutdownThread","level":"INFO","caller_class_name":"com.hazelcast.logging.Slf4jFactory$Slf4jLogger","caller_method_name":"log","caller_file_name":"Slf4jFactory.java","caller_line_number":65,"service":{"name":"space-ingest-ingest"},"trace":"","span":"","parent":"","pid":"1"}
{"@timestamp":"2020-11-04T05:18:34.401+00:00","message":"[192.168.1.92]:5701 [girl] [3.11.5] Connection[id=5, /192.168.1.92:5701->/192.168.1.84:48520, endpoint=[192.168.1.84]:5701, alive=false, type=MEMBER] closed. Reason: TcpIpConnectionManager is stopping","logger_name":"com.hazelcast.nio.tcp.TcpIpConnection","thread_name":"hz.ShutdownThread","level":"INFO","caller_class_name":"com.hazelcast.logging.Slf4jFactory$Slf4jLogger","caller_method_name":"log","caller_file_name":"Slf4jFactory.java","caller_line_number":65,"service":{"name":"space-ingest-ingest"},"trace":"","span":"","parent":"","pid":"1"}
{"@timestamp":"2020-11-04T05:18:34.401+00:00","message":"[192.168.1.92]:5701 [girl] [3.11.5] Connection[id=1, /192.168.1.92:5701->/192.168.1.83:59128, endpoint=[192.168.1.83]:5701, alive=false, type=MEMBER] closed. Reason: TcpIpConnectionManager is stopping","logger_name":"com.hazelcast.nio.tcp.TcpIpConnection","thread_name":"hz.ShutdownThread","level":"INFO","caller_class_name":"com.hazelcast.logging.Slf4jFactory$Slf4jLogger","caller_method_name":"log","caller_file_name":"Slf4jFactory.java","caller_line_number":65,"service":{"name":"space-ingest-ingest"},"trace":"","span":"","parent":"","pid":"1"}
{"@timestamp":"2020-11-04T05:18:34.401+00:00","message":"[192.168.1.92]:5701 [girl] [3.11.5] Connection[id=3, /192.168.1.92:5701->/192.168.1.82:35939, endpoint=[192.168.1.82]:5701, alive=false, type=MEMBER] closed. Reason: TcpIpConnectionManager is stopping","logger_name":"com.hazelcast.nio.tcp.TcpIpConnection","thread_name":"hz.ShutdownThread","level":"INFO","caller_class_name":"com.hazelcast.logging.Slf4jFactory$Slf4jLogger","caller_method_name":"log","caller_file_name":"Slf4jFactory.java","caller_line_number":65,"service":{"name":"space-ingest-ingest"},"trace":"","span":"","parent":"","pid":"1"}
{"@timestamp":"2020-11-04T05:18:34.401+00:00","message":"[192.168.1.92]:5701 [girl] [3.11.5] Connection[id=10, /192.168.1.92:52480->/192.168.2.55:5701, endpoint=[192.168.2.55]:5701, alive=false, type=MEMBER] closed. Reason: TcpIpConnectionManager is stopping","logger_name":"com.hazelcast.nio.tcp.TcpIpConnection","thread_name":"hz.ShutdownThread","level":"INFO","caller_class_name":"com.hazelcast.logging.Slf4jFactory$Slf4jLogger","caller_method_name":"log","caller_file_name":"Slf4jFactory.java","caller_line_number":65,"service":{"name":"space-ingest-ingest"},"trace":"","span":"","parent":"","pid":"1"}
{"@timestamp":"2020-11-04T05:18:34.401+00:00","message":"[192.168.1.92]:5701 [girl] [3.11.5] Connection[id=17, /192.168.1.92:56021->/192.168.2.56:5701, endpoint=[192.168.2.56]:5701, alive=false, type=MEMBER] closed. Reason: TcpIpConnectionManager is stopping","logger_name":"com.hazelcast.nio.tcp.TcpIpConnection","thread_name":"hz.ShutdownThread","level":"INFO","caller_class_name":"com.hazelcast.logging.Slf4jFactory$Slf4jLogger","caller_method_name":"log","caller_file_name":"Slf4jFactory.java","caller_line_number":65,"service":{"name":"space-ingest-ingest"},"trace":"","span":"","parent":"","pid":"1"}
{"@timestamp":"2020-11-04T05:18:34.402+00:00","message":"[192.168.1.92]:5701 [girl] [3.11.5] Connection[id=4, /192.168.1.92:5701->/192.168.1.87:40491, endpoint=[192.168.1.87]:5701, alive=false, type=MEMBER] closed. Reason: TcpIpConnectionManager is stopping","logger_name":"com.hazelcast.nio.tcp.TcpIpConnection","thread_name":"hz.ShutdownThread","level":"INFO","caller_class_name":"com.hazelcast.logging.Slf4jFactory$Slf4jLogger","caller_method_name":"log","caller_file_name":"Slf4jFactory.java","caller_line_number":65,"service":{"name":"space-ingest-ingest"},"trace":"","span":"","parent":"","pid":"1"}
{"@timestamp":"2020-11-04T05:18:34.402+00:00","message":"[192.168.1.92]:5701 [girl] [3.11.5] Connection[id=15, /192.168.1.92:39972->/192.168.1.85:5701, endpoint=[192.168.1.85]:5701, alive=false, type=MEMBER] closed. Reason: TcpIpConnectionManager is stopping","logger_name":"com.hazelcast.nio.tcp.TcpIpConnection","thread_name":"hz.ShutdownThread","level":"INFO","caller_class_name":"com.hazelcast.logging.Slf4jFactory$Slf4jLogger","caller_method_name":"log","caller_file_name":"Slf4jFactory.java","caller_line_number":65,"service":{"name":"space-ingest-ingest"},"trace":"","span":"","parent":"","pid":"1"}
{"@timestamp":"2020-11-04T05:18:34.402+00:00","message":"[192.168.1.92]:5701 [girl] [3.11.5] Connection[id=23, /192.168.1.92:54239->/192.168.1.1:5702, endpoint=[192.168.1.1]:5702, alive=false, type=MEMBER] closed. Reason: TcpIpConnectionManager is stopping","logger_name":"com.hazelcast.nio.tcp.TcpIpConnection","thread_name":"hz.ShutdownThread","level":"INFO","caller_class_name":"com.hazelcast.logging.Slf4jFactory$Slf4jLogger","caller_method_name":"log","caller_file_name":"Slf4jFactory.java","caller_line_number":65,"service":{"name":"space-ingest-ingest"},"trace":"","span":"","parent":"","pid":"1"}
{"@timestamp":"2020-11-04T05:18:34.402+00:00","message":"[192.168.1.92]:5701 [girl] [3.11.5] Connection[id=20, /192.168.1.92:5701->/192.168.1.1:58368, endpoint=[192.168.1.1]:5703, alive=false, type=MEMBER] closed. Reason: TcpIpConnectionManager is stopping","logger_name":"com.hazelcast.nio.tcp.TcpIpConnection","thread_name":"hz.ShutdownThread","level":"INFO","caller_class_name":"com.hazelcast.logging.Slf4jFactory$Slf4jLogger","caller_method_name":"log","caller_file_name":"Slf4jFactory.java","caller_line_number":65,"service":{"name":"space-ingest-ingest"},"trace":"","span":"","parent":"","pid":"1"}
{"@timestamp":"2020-11-04T05:18:34.402+00:00","message":"[192.168.1.92]:5701 [girl] [3.11.5] Connection[id=21, /192.168.1.92:51154->/192.168.1.1:5704, endpoint=[192.168.1.1]:5704, alive=false, type=MEMBER] closed. Reason: TcpIpConnectionManager is stopping","logger_name":"com.hazelcast.nio.tcp.TcpIpConnection","thread_name":"hz.ShutdownThread","level":"INFO","caller_class_name":"com.hazelcast.logging.Slf4jFactory$Slf4jLogger","caller_method_name":"log","caller_file_name":"Slf4jFactory.java","caller_line_number":65,"service":{"name":"space-ingest-ingest"},"trace":"","span":"","parent":"","pid":"1"}
{"@timestamp":"2020-11-04T05:18:34.421+00:00","message":"[192.168.1.92]:5701 [girl] [3.11.5] Connection[id=22, /192.168.1.92:36742->/192.168.1.1:5703, endpoint=[192.168.1.1]:5703, alive=false, type=MEMBER] closed. Reason: TcpIpConnectionManager is stopping","logger_name":"com.hazelcast.nio.tcp.TcpIpConnection","thread_name":"hz.ShutdownThread","level":"INFO","caller_class_name":"com.hazelcast.logging.Slf4jFactory$Slf4jLogger","caller_method_name":"log","caller_file_name":"Slf4jFactory.java","caller_line_number":65,"service":{"name":"space-ingest-ingest"},"trace":"","span":"","parent":"","pid":"1"}
{"@timestamp":"2020-11-04T05:18:34.422+00:00","message":"[192.168.1.92]:5701 [girl] [3.11.5] Connection[id=25, /192.168.1.92:5701->/192.168.1.1:39775, endpoint=[192.168.1.1]:5702, alive=false, type=MEMBER] closed. Reason: TcpIpConnectionManager is stopping","logger_name":"com.hazelcast.nio.tcp.TcpIpConnection","thread_name":"hz.ShutdownThread","level":"INFO","caller_class_name":"com.hazelcast.logging.Slf4jFactory$Slf4jLogger","caller_method_name":"log","caller_file_name":"Slf4jFactory.java","caller_line_number":65,"service":{"name":"space-ingest-ingest"},"trace":"","span":"","parent":"","pid":"1"}
{"@timestamp":"2020-11-04T05:18:34.422+00:00","message":"[192.168.1.92]:5701 [girl] [3.11.5] Shutting down node engine...","logger_name":"com.hazelcast.instance.Node","thread_name":"hz.ShutdownThread","level":"INFO","caller_class_name":"com.hazelcast.logging.Slf4jFactory$Slf4jLogger","caller_method_name":"log","caller_file_name":"Slf4jFactory.java","caller_line_number":65,"service":{"name":"space-ingest-ingest"},"trace":"","span":"","parent":"","pid":"1"}
{"@timestamp":"2020-11-04T05:18:34.432+00:00","message":"[192.168.1.92]:5701 [girl] [3.11.5] Setting master address to null","logger_name":"c.h.internal.cluster.ClusterService","thread_name":"hz.ShutdownThread","level":"DEBUG","caller_class_name":"com.hazelcast.logging.Slf4jFactory$Slf4jLogger","caller_method_name":"log","caller_file_name":"Slf4jFactory.java","caller_line_number":63,"service":{"name":"space-ingest-ingest"},"trace":"","span":"","parent":"","pid":"1"}
{"@timestamp":"2020-11-04T05:18:34.444+00:00","message":"[192.168.1.92]:5701 [girl] [3.11.5] com.hazelcast.core.HazelcastInstanceNotActiveException: Hazelcast instance is not active!","logger_name":"com.hazelcast.spi.OperationService","thread_name":"http-nio-9043-exec-3","level":"WARN","caller_class_name":"com.hazelcast.logging.Slf4jFactory$Slf4jLogger","caller_method_name":"log","caller_file_name":"Slf4jFactory.java","caller_line_number":67,"service":{"name":"space-ingest-ingest"},"trace":"07f50513d21be9ba","span":"07f50513d21be9ba","parent":"","pid":"1"}
{"@timestamp":"2020-11-04T05:18:34.448+00:00","message":"[192.168.1.92]:5701 [girl] [3.11.5] com.hazelcast.spi.exception.TargetNotMemberException: Not Member! target: [192.168.1.88]:5701, partitionId: -1, operation: com.hazelcast.spi.impl.operationservice.impl.operations.PartitionIteratingOperation, service: hz:impl:mapService","logger_name":"com.hazelcast.spi.OperationService","thread_name":"http-nio-9043-exec-3","level":"WARN","caller_class_name":"com.hazelcast.logging.Slf4jFactory$Slf4jLogger","caller_method_name":"log","caller_file_name":"Slf4jFactory.java","caller_line_number":67,"service":{"name":"space-ingest-ingest"},"trace":"07f50513d21be9ba","span":"07f50513d21be9ba","parent":"","pid":"1"}
{"@timestamp":"2020-11-04T05:18:34.450+00:00","message":"[192.168.1.92]:5701 [girl] [3.11.5] com.hazelcast.spi.exception.RetryableIOException: Packet not sent to -> [192.168.1.86]:5701 over null","logger_name":"com.hazelcast.spi.OperationService","thread_name":"http-nio-9043-exec-3","level":"WARN","caller_class_name":"com.hazelcast.logging.Slf4jFactory$Slf4jLogger","caller_method_name":"log","caller_file_name":"Slf4jFactory.java","caller_line_number":67,"service":{"name":"space-ingest-ingest"},"trace":"07f50513d21be9ba","span":"07f50513d21be9ba","parent":"","pid":"1"}
{"@timestamp":"2020-11-04T05:18:34.458+00:00","message":"[192.168.1.92]:5701 [girl] [3.11.5] Destroying node NodeExtension.","logger_name":"com.hazelcast.instance.NodeExtension","thread_name":"hz.ShutdownThread","level":"INFO","caller_class_name":"com.hazelcast.logging.Slf4jFactory$Slf4jLogger","caller_method_name":"log","caller_file_name":"Slf4jFactory.java","caller_line_number":65,"service":{"name":"space-ingest-ingest"},"trace":"","span":"","parent":"","pid":"1"}
{"@timestamp":"2020-11-04T05:18:34.459+00:00","message":"[192.168.1.92]:5701 [girl] [3.11.5] Hazelcast Shutdown is completed in 65 ms.","logger_name":"com.hazelcast.instance.Node","thread_name":"hz.ShutdownThread","level":"INFO","caller_class_name":"com.hazelcast.logging.Slf4jFactory$Slf4jLogger","caller_method_name":"log","caller_file_name":"Slf4jFactory.java","caller_line_number":65,"service":{"name":"space-ingest-ingest"},"trace":"","span":"","parent":"","pid":"1"}
{"stack_trace":"<#81507f7a> com.hazelcast.core.HazelcastInstanceNotActiveException: Hazelcast instance is not active!\n\tat com.hazelcast.spi.impl.operationservice.impl.InvocationRegistry.register(InvocationRegistry.java:128)\n\tat com.hazelcast.spi.impl.operationservice.impl.Invocation.doInvoke(Invocation.java:559)\n\tat com.hazelcast.spi.impl.operationservice.impl.Invocation.invoke0(Invocation.java:530)\n\tat com.hazelcast.spi.impl.operationservice.impl.Invocation.invoke(Invocation.java:220)\n\tat com.hazelcast.spi.impl.operationservice.impl.InvocationBuilderImpl.invoke(InvocationBuilderImpl.java:60)\n\tat com.hazelcast.spi.impl.operationservice.impl.InvokeOnPartitions.retryFailedPartitions(InvokeOnPartitions.java:142)\n\tat com.hazelcast.spi.impl.operationservice.impl.InvokeOnPartitions.invoke(InvokeOnPartitions.java:78)\n\tat com.hazelcast.spi.impl.operationservice.impl.OperationServiceImpl.invokeOnAllPartitions(OperationServiceImpl.java:385)\n\tat com.hazelcast.map.impl.proxy.MapProxySupport.size(MapProxySupport.java:719)\n\tat com.hazelcast.map.impl.proxy.MapProxyImpl.size(MapProxyImpl.java:109)\n\tat io.zontal.space.common.restservice.config.HazelcastInfoContributor.contribute(HazelcastInfoContributor.java:140)\n\tat org.springframework.boot.actuate.info.InfoEndpoint.info(InfoEndpoint.java:52)\n\tat sun.reflect.GeneratedMethodAccessor315.invoke(Unknown Source)\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.lang.reflect.Method.invoke(Method.java:498)\n\tat org.springframework.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:282)\n\tat org.springframework.boot.actuate.endpoint.invoke.reflect.ReflectiveOperationInvoker.invoke(ReflectiveOperationInvoker.java:76)\n\tat org.springframework.boot.actuate.endpoint.annotation.AbstractDiscoveredOperation.invoke(AbstractDiscoveredOperation.java:60)\n\tat o.s.b.actuate.endpoint.web.servlet.AbstractWebMvcEndpointHandlerMapping$ServletWebOperationAdapter.handle(AbstractWebMvcEndpointHandlerMapping.java:278)\n\tat o.s.boot.actuate.endpoint.web.servlet.AbstractWebMvcEndpointHandlerMapping$OperationHandler.handle(AbstractWebMvcEndpointHandlerMapping.java:334)\n\tat sun.reflect.GeneratedMethodAccessor211.invoke(Unknown Source)\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.lang.reflect.Method.invoke(Method.java:498)\n\tat org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:190)\n\tat org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:138)\n\tat org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:105)\n\tat org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:893)\n\tat org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:798)\n\tat org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)\n\tat org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1040)\n\tat org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:943)\n\tat org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)\n\tat org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:898)\n\tat javax.servlet.http.HttpServlet.service(HttpServlet.java:634)\n\tat org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)\n\tat javax.servlet.http.HttpServlet.service(HttpServlet.java:741)\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)\n\tat org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)\n\tat brave.servlet.TracingFilter.doFilter(TracingFilter.java:65)\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)\n\tat io.zontal.space.common.restservice.audit.AccessLogFilter.doFilterInternal(AccessLogFilter.java:114)\n\tat org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)\n\tat org.zalando.logbook.servlet.NormalStrategy.doFilter(NormalStrategy.java:41)\n\tat org.zalando.logbook.servlet.LogbookFilter.doFilter(LogbookFilter.java:33)\n\tat...","@timestamp":"2020-11-04T05:18:34.463+00:00","message":"Hazelcast instance is not active!","logger_name":"i.z.s.c.r.handler.GlobalExceptionHandler","thread_name":"http-nio-9043-exec-3","level":"ERROR","caller_class_name":"io.zontal.space.common.restservice.handler.GlobalExceptionHandler","caller_method_name":"getError","caller_file_name":"GlobalExceptionHandler.java","caller_line_number":227,"stack_hash":"81507f7a","throwable_class":"HazelcastInstanceNotActiveException","throwable_root_cause_class":"com.hazelcast.core.HazelcastInstanceNotActiveException","service":{"name":"space-ingest-ingest"},"trace":"07f50513d21be9ba","span":"07f50513d21be9ba","parent":"","pid":"1"}
{"stack_trace":"<#a9e0e99c> java.io.IOException: Broken pipe\n\tat sun.nio.ch.FileDispatcherImpl.write0(FileDispatcherImpl.java)\n\tat sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)\n\tat sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)\n\tat sun.nio.ch.IOUtil.write(IOUtil.java:65)\n\tat sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471)\n\tat org.apache.tomcat.util.net.NioChannel.write(NioChannel.java:138)\n\tat org.apache.tomcat.util.net.NioBlockingSelector.write(NioBlockingSelector.java:101)\n\tat org.apache.tomcat.util.net.NioSelectorPool.write(NioSelectorPool.java:152)\n\tat org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.doWrite(NioEndpoint.java:1251)\n\tat org.apache.tomcat.util.net.SocketWrapperBase.doWrite(SocketWrapperBase.java:801)\n\tat org.apache.tomcat.util.net.SocketWrapperBase.writeBlocking(SocketWrapperBase.java:571)\n\tat org.apache.tomcat.util.net.SocketWrapperBase.write(SocketWrapperBase.java:509)\n\tat org.apache.coyote.http11.Http11OutputBuffer$SocketOutputBuffer.doWrite(Http11OutputBuffer.java:538)\n\tat org.apache.coyote.http11.filters.ChunkedOutputFilter.doWrite(ChunkedOutputFilter.java:112)\n\tat org.apache.coyote.http11.Http11OutputBuffer.doWrite(Http11OutputBuffer.java:190)\n\tat org.apache.coyote.Response.doWrite(Response.java:601)\n\tat org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:339)\n\t... 114 common frames omitted\nWrapped by: <#4e72292a> org.apache.catalina.connector.ClientAbortException: java.io.IOException: Broken pipe\n\tat org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:351)\n\tat org.apache.catalina.connector.OutputBuffer.flushByteBuffer(OutputBuffer.java:776)\n\tat org.apache.catalina.connector.OutputBuffer.append(OutputBuffer.java:681)\n\tat org.apache.catalina.connector.OutputBuffer.writeBytes(OutputBuffer.java:386)\n\tat org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:364)\n\tat org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:96)\n\tat org.zalando.logbook.servlet.LocalResponse$TeeServletOutputStream.write(LocalResponse.java:161)\n\tat com.fasterxml.jackson.core.json.UTF8JsonGenerator._flushBuffer(UTF8JsonGenerator.java:2085)\n\tat com.fasterxml.jackson.core.json.UTF8JsonGenerator._writeStringSegment2(UTF8JsonGenerator.java:1400)\n\tat com.fasterxml.jackson.core.json.UTF8JsonGenerator._writeStringSegment(UTF8JsonGenerator.java:1347)\n\tat com.fasterxml.jackson.core.json.UTF8JsonGenerator._writeStringSegments(UTF8JsonGenerator.java:1228)\n\tat com.fasterxml.jackson.core.json.UTF8JsonGenerator.writeString(UTF8JsonGenerator.java:453)\n\tat com.fasterxml.jackson.databind.ser.std.StringSerializer.serialize(StringSerializer.java:41)\n\tat com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:727)\n\tat com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:719)\n\tat com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:155)\n\tat com.fasterxml.jackson.databind.ser.DefaultSerializerProvider._serialize(DefaultSerializerProvider.java:480)\n\tat com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:319)\n\tat com.fasterxml.jackson.databind.ObjectWriter$Prefetch.serialize(ObjectWriter.java:1396)\n\tat com.fasterxml.jackson.databind.ObjectWriter.writeValue(ObjectWriter.java:913)\n\tat org.springframework.http.converter.json.AbstractJackson2HttpMessageConverter.writeInternal(AbstractJackson2HttpMessageConverter.java:287)\n\tat org.springframework.http.converter.AbstractGenericHttpMessageConverter.write(AbstractGenericHttpMessageConverter.java:103)\n\tat org.springframework.web.servlet.mvc.method.annotation.AbstractMessageConverterMethodProcessor.writeWithMessageConverters(AbstractMessageConverterMethodProcessor.java:291)\n\tat org.springframework.web.servlet.mvc.method.annotation.HttpEntityMethodProcessor.handleReturnValue(HttpEntityMethodProcessor.java:226)\n\tat org.springframework.web.method.support.HandlerMethodReturnValueHandlerComposite.handleReturnValue(HandlerMethodReturnValueHandlerComposite.java:82)\n\tat org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:123)\n\tat org.springframework.web.servlet.mvc.method.annotation.ExceptionHandlerExceptionResolver.doResolveHandlerMethodException(ExceptionHandlerExceptionResolver.java:413)\n\tat org.springframework.web.servlet.handler.AbstractHandlerMethodExceptionResolver.doResolveException(AbstractHandlerMethodExceptionResolver.java:61)\n\tat org.springframework.web.servlet.handler.AbstractHandlerExceptionResolver.resolveException(AbstractHandlerExceptionResolver.java:141)\n\tat org.springframework.web.servlet.handler.HandlerExceptionResolverComposite.resolveException(HandlerExceptionResolverComposite.java:80)\n\tat org.springframework.web.servlet.DispatcherServlet.processHandlerException(DispatcherServlet.java:1299)\n\tat org.springframework.web.serv...","@timestamp":"2020-11-04T05:18:34.468+00:00","message":"Failure in @ExceptionHandler public org.springframework.http.ResponseEntity
kaojunsong
commented
3 years ago @Bean
@ConditionalOnProperty(name = "xxxx.cache.hazelcast.enabled")
public Config hazelcastConfig(List
if (hazelcastConfiguration.getKubernetes().isEnabled()) {
logger.info("Hazelcast Kubernetes discovery activated for namespace <{}>, using API server <{}>",
hazelcastConfiguration.getKubernetes().getNamespace(),
hazelcastConfiguration.getKubernetes().getAPIServer());
networkConfig.getJoin().getKubernetesConfig()
.setProperty(NAMESPACE_PROPERTY, hazelcastConfiguration.getKubernetes().getNamespace())
.setProperty(POD_LABEL_NAME_PROPERTY, HAZELCAST_LABEL).setProperty(POD_LABEL_VALUE_PROPERTY, "true")
.setProperty(RESOLVE_NOT_READY_ADDRESSES_PROPERTY, "true");
if (Objects.nonNull(hazelcastConfiguration.getKubernetes().getAPIServer())) {
networkConfig.getJoin().getKubernetesConfig().setProperty(KUBERNETES_MASTER_PROPERTY,
hazelcastConfiguration.getKubernetes().getAPIServer());
}
}
config.setNetworkConfig(networkConfig);
for (MapConfig mapConfig : mapConfigSuppliers) {
config.addMapConfig(mapConfig);
}
return config;
}We use Spring, and do not use helm charts @hasancelik thank you
hasancelik
commented
3 years ago @kaojunsong,
As far as I understand from above redacted logs, your member received the heartbeat from other members but they can not form the cluster, right?
What is your hazelcast-kubernetes plugin version btw? Could you also share your k8s yamls?
You are using old version of hazelcast. Do you consider to use one of newer hazelcast and kubernetes discovery plugin version? It would be great if you try to reproduce same issue with 4.x hazelcast and 2.x k8s discovery plugin.
kaojunsong
commented
3 years ago @hasancelik Thank you very much for your reply. I think I have found the cause: After the load of one pod get high, hazelcast becomes slow, and hazelcast is invoked in actuator, k8s liveness probe uses actuator, then causing other pod restarted by k8s.
hasancelik
commented
3 years ago @kaojunsong glad to see the issue is resolved. Thanks for the explanation, btw 👍
Our system hazelcast embedded model, and also use hazelcast-kubernetes, and we have 25 nodes in our cluster. Recently we found the issue that when a pod crashed, all the other pods that using hazelcast all crashed. I have no idea how to locate this issue, can anyone have any idea?