andersroos commented 8 months ago

Observed behavior

I discovered this when exploring key value and testing resillience and performace among other things.

I used a helm install in kubernetes.

While updating a key value bucket (in memory, with replication 3) with about 1000 updates/s I restarted one pod. It never came back up and it got stuck in a stream catchup state (waited over 20 minutes). Subsequent restarts did not resolve the issue. It only got resolved after the stream was deleted (or in subsequent tests when all nats pods where stopped).

See also this thread in slack: https://natsio.slack.com/archives/C06EN6HCWE4/p1710151450266159

Stream:

Information for Stream KV_gamesession-state created 2024-03-06 16:42:33

              Subjects: $KV.gamesession-state.>
              Replicas: 3
               Storage: Memory

Options:

             Retention: Limits
       Acknowledgments: true
        Discard Policy: Old
      Duplicate Window: 2m0s
            Direct Get: true
     Allows Msg Delete: false
          Allows Purge: true
        Allows Rollups: true

Limits:

      Maximum Messages: unlimited
   Maximum Per Subject: 1
         Maximum Bytes: unlimited
           Maximum Age: 30m0s
  Maximum Message Size: unlimited
     Maximum Consumers: unlimited

Cluster Information:

                  Name: nats
                Leader: nats-0
               Replica: nats-1, current, seen 97ms ago
               Replica: nats-2, current, seen 97ms ago

State:

              Messages: 10
                 Bytes: 1.7 KiB
        First Sequence: 1,245,135 @ 2024-03-11 11:10:15 UTC
         Last Sequence: 1,245,155 @ 2024-03-11 11:10:15 UTC
      Deleted Messages: 11
      Active Consumers: 0
    Number of Subjects: 10

Log snippet:

[57] 2024/03/11 10:08:28.171728 [WRN] Catchup for stream '$G > KV_gamesession-state' stalled
[57] 2024/03/11 10:08:36.806848 [WRN] Healthcheck failed: "JetStream stream '$G > KV_gamesession-state' is not current"
[57] 2024/03/11 10:08:46.806154 [WRN] Healthcheck failed: "JetStream stream '$G > KV_gamesession-state' is not current"
[57] 2024/03/11 10:08:56.805914 [WRN] Healthcheck failed: "JetStream stream '$G > KV_gamesession-state' is not current"
[57] 2024/03/11 10:08:58.172655 [WRN] Catchup for stream '$G > KV_gamesession-state' stalled
[57] 2024/03/11 10:08:58.177889 [WRN] Error applying entries to '$G > KV_gamesession-state': first sequence mismatch
[57] 2024/03/11 10:08:58.178309 [WRN] Resetting stream cluster state for '$G > KV_gamesession-state'
[57] 2024/03/11 10:08:58.301144 [WRN] Catchup for stream '$G > KV_gamesession-state' resetting first sequence: 1228966 on catchup request
[57] 2024/03/11 10:09:06.806746 [WRN] Healthcheck failed: "JetStream stream '$G > KV_gamesession-state' is not current"
[57] 2024/03/11 10:09:16.805821 [WRN] Healthcheck failed: "JetStream stream '$G > KV_gamesession-state' is not current"
[57] 2024/03/11 10:09:26.806109 [WRN] Healthcheck failed: "JetStream stream '$G > KV_gamesession-state' is not current"
[57] 2024/03/11 10:09:28.301893 [WRN] Catchup for stream '$G > KV_gamesession-state' stalled
[57] 2024/03/11 10:09:36.806382 [WRN] Healthcheck failed: "JetStream stream '$G > KV_gamesession-state' is not current"
[57] 2024/03/11 10:09:46.806260 [WRN] Healthcheck failed: "JetStream stream '$G > KV_gamesession-state' is not current"
[57] 2024/03/11 10:09:56.806782 [WRN] Healthcheck failed: "JetStream stream '$G > KV_gamesession-state' is not current"
[57] 2024/03/11 10:09:58.302241 [WRN] Catchup for stream '$G > KV_gamesession-state' stalled
[57] 2024/03/11 10:09:58.311134 [WRN] Error applying entries to '$G > KV_gamesession-state': first sequence mismatch
[57] 2024/03/11 10:09:58.311752 [WRN] Resetting stream cluster state for '$G > KV_gamesession-state'
[57] 2024/03/11 10:09:58.650302 [WRN] Catchup for stream '$G > KV_gamesession-state' resetting first sequence: 1243279 on catchup request
[57] 2024/03/11 10:10:06.806367 [WRN] Healthcheck failed: "JetStream stream '$G > KV_gamesession-state' is not current"
[57] 2024/03/11 10:10:16.806423 [WRN] Healthcheck failed: "JetStream stream '$G > KV_gamesession-state' is not current"
[57] 2024/03/11 10:10:26.805642 [WRN] Healthcheck failed: "JetStream stream '$G > KV_gamesession-state' is not current"
[57] 2024/03/11 10:10:28.652230 [WRN] Catchup for stream '$G > KV_gamesession-state' stalled
[57] 2024/03/11 10:10:36.806734 [WRN] Healthcheck failed: "JetStream stream '$G > KV_gamesession-state' is not current"
[57] 2024/03/11 10:10:46.805759 [WRN] Healthcheck failed: "JetStream stream '$G > KV_gamesession-state' is not current"
[57] 2024/03/11 10:10:56.806693 [WRN] Healthcheck failed: "JetStream stream '$G > KV_gamesession-state' is not current"
[57] 2024/03/11 10:10:58.652829 [WRN] Catchup for stream '$G > KV_gamesession-state' stalled
[57] 2024/03/11 10:10:58.656426 [WRN] Error applying entries to '$G > KV_gamesession-state': first sequence mismatch
[57] 2024/03/11 10:10:58.656974 [WRN] Resetting stream cluster state for '$G > KV_gamesession-state'
[57] 2024/03/11 10:10:58.988587 [WRN] Catchup for stream '$G > KV_gamesession-state' resetting first sequence: 1243279 on catchup request
[57] 2024/03/11 10:11:06.806418 [WRN] Healthcheck failed: "JetStream stream '$G > KV_gamesession-state' is not current"
[57] 2024/03/11 10:11:16.805928 [WRN] Healthcheck failed: "JetStream stream '$G > KV_gamesession-state' is not current"
[57] 2024/03/11 10:11:26.806630 [WRN] Healthcheck failed: "JetStream stream '$G > KV_gamesession-state' is not current"
[57] 2024/03/11 10:11:28.989443 [WRN] Catchup for stream '$G > KV_gamesession-state' stalled
[57] 2024/03/11 10:11:36.806795 [WRN] Healthcheck failed: "JetStream stream '$G > KV_gamesession-state' is not current"
[57] 2024/03/11 10:11:46.806430 [WRN] Healthcheck failed: "JetStream stream '$G > KV_gamesession-state' is not current"
[57] 2024/03/11 10:11:56.806165 [WRN] Healthcheck failed: "JetStream stream '$G > KV_gamesession-state' is not current"

Expected behavior

I expected the restarted pod to catch up and come back into the cluster.

Server and client version

Server version: 2.10.11 Client version (golang client): v1.32.0

Host environment

Both server and client was running in managed Kubernetes in Google Cloud, using Google Filestore as perstistence layer.

Steps to reproduce

It is a bit hard to reproduce, I only manged to reproduce it about 1/5 tries.

What I did was:

Run a pod with a nats client that does updates in a keyvalue as fast as possibile. This is done with 100 goroutines in paralell. The goroutines will update a set of 10 keys (so 10 goroutines for each key).
Seems to be running for a whle (I never reproduced when running it shorter than 5 minutes).
Restart one pod, observe it not coming up (fail ready check) and see log like snippet above.

I also tried to reproduce using a file store for the stream, but didn't manage to do it within 10 tries or so.

bondar-pavel commented 8 months ago

Observing the same issue on latest NATS version as well. Tested versions:

Server version: 2.10.11
Server version: 2.10.12 It was observed on previous versions too (approximately for a year).

We have ~20 streams each of them receiving multiple writes. NATS is deployed in Kubernetes via helm chart. During NATS node restart one or several of the streams are failing to recover:

[7] 2024/03/18 14:56:33.016521 [WRN] Catchup for stream 'test_stream > test-stream-notifications' resetting first sequence: 285724 on catchup request
[7] 2024/03/18 14:56:36.093024 [WRN] Healthcheck failed: "JetStream stream 'test_stream > test-stream-notifications' is not current"
[7] 2024/03/18 14:56:46.092946 [WRN] Healthcheck failed: "JetStream stream 'test_stream > test-stream-notifications' is not current"
[7] 2024/03/18 14:56:56.093213 [WRN] Healthcheck failed: "JetStream stream 'test_stream > test-stream-notifications' is not current"
[7] 2024/03/18 14:57:03.020724 [WRN] Catchup for stream 'test_stream > test-stream-notifications' stalled
[7] 2024/03/18 14:57:06.093278 [WRN] Healthcheck failed: "JetStream stream 'test_stream > test-stream-notifications' is not current"
[7] 2024/03/18 14:57:16.092459 [WRN] Healthcheck failed: "JetStream stream 'test_stream > test-stream-notifications' is not current"
[7] 2024/03/18 14:57:26.093089 [WRN] Healthcheck failed: "JetStream stream 'test_stream > test-stream-notifications' is not current"
[7] 2024/03/18 14:57:33.021379 [WRN] Catchup for stream 'test_stream > test-stream-notifications' stalled
[7] 2024/03/18 14:57:33.021563 [WRN] Error applying entries to 'test_stream > test-stream-notifications': first sequence mismatch
[7] 2024/03/18 14:57:33.024270 [WRN] Resetting stream cluster state for 'test_stream > test-stream-notifications'
[7] 2024/03/18 14:57:34.018831 [WRN] Catchup for stream 'test_stream > test-stream-notifications' resetting first sequence: 285724 on catchup request
[7] 2024/03/18 14:57:36.092558 [WRN] Healthcheck failed: "JetStream stream 'test_stream > test-stream-notifications' is not current"
[7] 2024/03/18 14:57:46.092281 [WRN] Healthcheck failed: "JetStream stream 'test_stream > test-stream-notifications' is not current"
[7] 2024/03/18 14:57:56.093019 [WRN] Healthcheck failed: "JetStream stream 'test_stream > test-stream-notifications' is not current"
[7] 2024/03/18 14:58:04.022581 [WRN] Catchup for stream 'test_stream > test-stream-notifications' stalled
[7] 2024/03/18 14:58:06.092985 [WRN] Healthcheck failed: "JetStream stream 'test_stream > test-stream-notifications' is not current"
[7] 2024/03/18 14:58:16.092579 [WRN] Healthcheck failed: "JetStream stream 'test_stream > test-stream-notifications' is not current"
[7] 2024/03/18 14:58:26.093080 [WRN] Healthcheck failed: "JetStream stream 'test_stream > test-stream-notifications' is not current"
[7] 2024/03/18 14:58:34.023119 [WRN] Catchup for stream 'test_stream > test-stream-notifications' stalled
[7] 2024/03/18 14:58:34.023352 [WRN] Error applying entries to 'test_stream > test-stream-notifications': first sequence mismatch
[7] 2024/03/18 14:58:34.025869 [WRN] Resetting stream cluster state for 'test_stream > test-stream-notifications'
[7] 2024/03/18 14:58:35.016221 [WRN] Catchup for stream 'test_stream > test-stream-notifications' resetting first sequence: 285724 on catchup request
[7] 2024/03/18 14:58:36.092940 [WRN] Healthcheck failed: "JetStream stream 'test_stream > test-stream-notifications' is not current"
[7] 2024/03/18 14:58:46.092269 [WRN] Healthcheck failed: "JetStream stream 'test_stream > test-stream-notifications' is not current"
[7] 2024/03/18 14:58:56.092615 [WRN] Healthcheck failed: "JetStream stream 'test_stream > test-stream-notifications' is not current"
[7] 2024/03/18 14:59:05.020531 [WRN] Catchup for stream 'test_stream > test-stream-notifications' stalled
[7] 2024/03/18 14:59:06.093114 [WRN] Healthcheck failed: "JetStream stream 'test_stream > test-stream-notifications' is not current"
[7] 2024/03/18 14:59:16.092664 [WRN] Healthcheck failed: "JetStream stream 'test_stream > test-stream-notifications' is not current"

Workaround we found so far to get node to healthy state without data loss is to temporarily change replication factor for affected stream (from 3 to 1), then restore original replication factor value.

nats stream edit test_stream --replicas=1

This way affected node does not need to host this stream, so node can finish initialisation.

Tarasovych commented 8 months ago

Same issue, nats-2.10.12. I went from 1 replica to 2 replicas for my stream, it helped.

jzhn commented 7 months ago

We have experienced the same issue in multiple environments for interest based Jetstream (not KV).

Environment information

Kubernetes based StatefulSet deployment with 3-replicas
3-replica, Interest based, file storage stream with single consumer
- the file storage is persistent, backed by EBS backed PersistentVolume.
Nats Helm chart: 1.1.10
Nats server: reproduciable on both 2.10.11 and 2.10.12.
The rolling deployment is graceful, as the helm chart ensures out of the box.

Symptom

We noticed that the symptom seems to be caused by the first_seq of stream(s) go out of sync between replicas during deployment. One or two of the pods would have first_seq stuck, while other(s) would have first_seq growing up as messages are consumed. After the cluster went in this inconsistent state, the next rolling deployment would risk trigging the issue of Error applying entries to X: first sequence mismatch as mentioned in this issue.

As demonstrated above, first_seq of a stream went out of sync, that kubernetes-nats-1 and kubernetes-nats-2 have their first_seq stuck, while kubernetes-nats-0 has first_seq moving normally after a rolling restart.

Reproduce

It's relatively easy to reproduce this issue in a 3-replica Kubernetes hosted NATs cluster with Jetstream enabled.

Create an Interest based Stream with 3 replicas and file storage
Create a consumer to above stream with subscribers that simply acks the message (to simulate a fast-moving consumer that always catches up with the publisher thus moving the first_seq of the interest-based stream).
In a shell session, execute nats pub <subject> --count 1000000000 "test message {{Count}} @ {{Time}}" to keep publishing messages to the stream.
In another shell session, execute kubectl rollout restart statefulset/kubernetes-nats to trigger graceful rolling restart of NATs cluster.

Actual behaviour

The nats_stream_first_seq metric of the stream went out of sync after the rolling deployment between 3 NATs nodes.

Expected behaviour

The nats_stream_first_seq metric of the stream should be consistent between all 3 NATs nodes.

Mitigation

As @bondar-pavel suggested, editing the stream replicas to 1 and then back to 3 seems to be the only mitigation without losing data or causing downtime.

Notes

Stream configuration

{
  "config": {
    "name": "<redacted>",
    "subjects": [
      "<redacted>"
    ],
    "retention": "interest",
    "max_consumers": -1,
    "max_msgs_per_subject": -1,
    "max_msgs": -1,
    "max_bytes": -1,
    "max_age": 0,
    "max_msg_size": -1,
    "storage": "file",
    "discard": "old",
    "num_replicas": 3,
    "duplicate_window": 10000000000,
    "sealed": false,
    "deny_delete": false,
    "deny_purge": false,
    "allow_rollup_hdrs": false,
    "allow_direct": true,
    "mirror_direct": false
  },
  "created": "2024-03-28T17:59:55.75752643Z",
  "state": {
    "messages": 1372,
    "bytes": 127596,
    "first_seq": 11761831,
    "first_ts": "2024-03-28T19:27:57.692021989Z",
    "last_seq": 11763599,
    "last_ts": "2024-03-28T19:27:58.395525207Z",
    "num_deleted": 397,
    "num_subjects": 1,
    "consumer_count": 1
  },
  "cluster": {
    "name": "kubernetes-nats",
    "leader": "kubernetes-nats-2",
    "replicas": [
      {
        "name": "kubernetes-nats-0",
        "current": true,
        "active": 196170
      },
      {
        "name": "kubernetes-nats-1",
        "current": true,
        "active": 220688
      }
    ]
  },
  "ts": "2024-03-28T19:27:58.395990925Z"
}

Consumer configuration

{
  "stream_name": "<redacted>",
  "name": "<redacted>",
  "config": {
    "ack_policy": "explicit",
    "ack_wait": 30000000000,
    "deliver_policy": "new",
    "durable_name": "<redacted>",
    "name": "<redacted>",
    "max_ack_pending": 20000,
    "max_deliver": 5,
    "max_waiting": 512,
    "replay_policy": "instant",
    "num_replicas": 0
  },
  "created": "2024-03-28T17:59:56.100249122Z",
  "delivered": {
    "consumer_seq": 38939132,
    "stream_seq": 12980344,
    "last_active": "2024-03-28T19:35:54.238507718Z"
  },
  "ack_floor": {
    "consumer_seq": 38933971,
    "stream_seq": 12978586,
    "last_active": "2024-03-28T19:35:54.197600505Z"
  },
  "num_ack_pending": 1545,
  "num_redelivered": 1121,
  "num_waiting": 4,
  "num_pending": 0,
  "cluster": {
    "name": "kubernetes-nats",
    "leader": "kubernetes-nats-2",
    "replicas": [
      {
        "name": "kubernetes-nats-0",
        "current": true,
        "active": 13341
      },
      {
        "name": "kubernetes-nats-1",
        "current": true,
        "active": 146374
      }
    ]
  },
  "ts": "2024-03-28T19:35:54.238705331Z"
}

derekcollison commented 7 months ago

We have some improvements coming in 2.10.14 around this which hopefully can help.

If you are feeling adventuresome feel free to grab a binary from the wq-orphan-msgs branch as follows and see if the issue still presents.

curl -sf https://binaries.nats.dev/github.com/nats-io/nats-server/v2@wq-orphan-msgs | sh

jzhn commented 7 months ago

I've been playing with @derekcollison 's dev build from wq-orphan-msgs branch in my Kubernetes setup, to report back:

With my test setup (described in https://github.com/nats-io/nats-server/issues/5205#issuecomment-2025964417) I'm no longer able to reproduce the first_seq out of sync issue with ~5 rolling restarts.

As shown in the graph, the nats_stream_first_seq metric of a fast-moving stream is pretty consistent between 3 nodes throughout the rolling restarts, as compared to the graph from previous comment where some nodes become flat.

The dev build seems to have compatibility issue with the 2.10.12 release build. During a rolling restart, not only the new pod with dev build fail to become healthy, the old pods with 2.10.12 build becomes unhealthy too. I can't find a way to make health check pass, not even with the trick to update streams to 1 replicas. Ended up wiping all persistent volumes to start everything clean.

Below are some error messages from the first dev build pod during rollout.

[129] 2024/04/01 19:43:16.028330 [WRN] RAFT [uG7ZhloB - C-R3F-c0T4IWCo] Error writing term and vote file for "C-R3F-c0T4IWCo": open /data/jetstream/$SYS/_js_/C-R3F-c0T4IWCo/tav.idx: no such file or directory
2024-04-01 12:43:16.028 
[129] 2024/04/01 19:43:16.028284 [ERR] RAFT [uG7ZhloB - C-R3F-c0T4IWCo] Resource not found: open /data/jetstream/$SYS/_js_/C-R3F-c0T4IWCo/tav.idx: no such file or directory
2024-04-01 12:43:07.663 
[129] 2024/04/01 19:43:07.663177 [WRN] RAFT [uG7ZhloB - _meta_] Falling behind in health check, commit 178 != applied 173

derekcollison commented 7 months ago

We will start cleaning up and cherry picking into main and then into 2.10.14. Thanks for checking though!

jnmoyne commented 6 months ago

Note that 2.10.14 has now been released (and now 2.10.15 should be very soon)

jzhn commented 6 months ago

(copying the reply from nats slack)

I just repeated my chaos tests with 2.10.14, basically having a fast moving stream with a publisher (nats pub ) and an interest based consumer, while rolling restarting my 3-replica statefulset of NATs. Overall it’s been a LOT better. previously, almost 100% of the time the first_seq number of the fast moving stream would go out of sync, particularly the nats-2 instance that were restarted the first would almost always have its first_seq number stuck flat after restart. But now out of 8 restarts, I haven’t seen a single case.

Note that the first rolling restart did reproduce the Healthcheck failed: "JetStream stream '$G > <stream-name>' is not current" issue on one of the node, I had to stop the producer and purge the stream in order to get the health check passing. I wasn’t able to reproduce the same issue later on. I hope this is just a transient issue caused by the inherited bad state from 2.10.12.

Overall I’m really happy with 2.10.14. If it goes well it would address a major operation pain point of Jetstream. I’ll deploy it to prod soon and let you know if anything comes up. Thanks again for the hard work!

bfoxstudio commented 6 months ago

After restart node: got error processing message: last sequence by subject mismatch: 112238 vs 115352

Note that 2.10.14 has now been released (and now 2.10.15 should be very soon)

we have same problem on 2.10.14 and on nightly-20240508 in kv stream:

 [INF] Starting nats-server\n
 [INF]   Version:  2.11.0-dev\n
 [INF]   Git:      [a895101]\n
 [DBG]   Go build: go1.21.9\n
 [INF]   Cluster:  js_egeo_kv_shard1_infile\n
 [INF]   Name:     js_egeo_kv_shard1_infile_node01\n
 [INF]   Node:     aZPpTWMr\n
 [INF]   ID:       NAVQASAKCWEM567JJSFHP4DTY2GAHDDQYOMOTPM4MOIYIJIV673YFNC7\n
 [INF] Using configuration file: /data/jetstream/configs/js_config\n
 [DBG] Created system account: \"$SYS\"\n
 [INF] Starting http monitor on 0.0.0.0:8222\n
 [INF] Starting JetStream\n
 [DBG] JetStream detected GOMEMLIMIT of 3.91 GB\n
 [DBG] JetStream creating dynamic configuration - 2.93 GB memory, 10.00 GB disk\n
 [INF]     _ ___ _____ ___ _____ ___ ___   _   __  __\n
 [INF]  _ | | __|_   _/ __|_   _| _ \\ __| /_\\ |  \\/  |\n
 [INF] | || | _|  | | \\__ \\ | | |   / _| / _ \\| |\\/| |\n
 [INF]  \\__/|___| |_| |___/ |_| |_|_\\___/_/ \\_\\_|  |_|\n
 [INF] \n
 [INF]          https://docs.nats.io/jetstream\n
 [INF] \n
 [INF] ---------------- JETSTREAM ----------------\n
 [INF]   Max Memory:      2.93 GB\n
 [INF]   Max Storage:     10.00 GB\n
 [INF]   Store Directory: \"/data/jetstream/jetstream\"\n
 [INF] -------------------------------------------\n
 [DBG]   Exports:\n
 [DBG]      $JS.API.\u003e\n
 [DBG] Enabled JetStream for account \"$G\"\n
 [DBG]   Max Memory:      -1 B\n
 [DBG]   Max Storage:     -1 B\n
 [DBG] Recovering JetStream state for account \"$G\"\n
 [INF]   Starting restore for stream '$G \u003e KV_wh-px-egeo-e-shard1-kv'\n
 [INF]   Restored 64,596 messages for stream '$G \u003e KV_wh-px-egeo-e-shard1-kv' in 55ms\n
 [INF]   Recovering 1 consumers for stream - '$G \u003e KV_wh-px-egeo-e-shard1-kv'\n
 [DBG] JetStream state for account \"$G\" recovered\n
 [INF] Starting JetStream cluster\n
 [DBG] JetStream cluster checking for stable cluster name and peers\n
 [INF] Creating JetStream metadata controller\n
 [INF] JetStream cluster recovering state\n
 [DBG] RAFT [aZPpTWMr - _meta_] Started\n
 [INF] Listening for client connections on 0.0.0.0:4222\n
 [DBG] Get non local IPs for \"0.0.0.0\"\n
 [DBG]   ip=150.0.3.235\n
 [DBG]   ip=172.18.0.7\n
 [INF] Server is ready\n
 [INF] Cluster name is js_egeo_kv_shard1_infile\n
 [INF] Listening for route connections on 0.0.0.0:6222\n
 [DBG] Starting metadata monitor\n
 [DBG] maxprocs: Updating GOMAXPROCS=5: determined from CPU quota\n
 [DBG] JetStream cluster creating raft group:\u0026{Name:S-R5F-Z6jshyaK Peers:[v2fh5Gis JrN1hGZO dGVGKtkV aZPpTWMr bXlHueQT] Storage:File Cluster:js_egeo_kv_shard1_infile Preferred: node:\u003cnil\u003e}\n
 [DBG] RAFT [aZPpTWMr - S-R5F-Z6jshyaK] Replaying state of 36 entries\n
 [DBG] RAFT [aZPpTWMr - S-R5F-Z6jshyaK] Started\n
 [DBG] RAFT [aZPpTWMr - S-R5F-Z6jshyaK] Update peers from leader to map[JrN1hGZO:0xc000b1e4c8 aZPpTWMr:0xc000b1e498 bXlHueQT:0xc000b1e4f8 dGVGKtkV:0xc000b1e4e0 v2fh5Gis:0xc000b1e4b0]\n
 [TRC] JETSTREAM - \u003c\u003c- [SUB $JSC.SI.$G.KV_wh-px-egeo-e-shard1-kv  63]\n
 [DBG] JetStream cluster creating raft group:\u0026{Name:C-R5F-GQUbjX2S Peers:[v2fh5Gis JrN1hGZO dGVGKtkV aZPpTWMr bXlHueQT] Storage:File Cluster:js_egeo_kv_shard1_infile Preferred: node:\u003cnil\u003e}\n
 [DBG] Trying to connect to route on js_egeo_kv_shard1_infile_node02:6222 (150.0.0.71:6222)\n
 [DBG] Trying to connect to route on js_egeo_kv_shard1_infile_node01:6222 (150.0.0.73:6222)\n
 [DBG] Trying to connect to route on js_egeo_kv_shard1_infile_node04:6222 (150.0.0.77:6222)\n
 [DBG] 150.0.0.73:6222 - rid:14 - Route connect msg sent\n
 [DBG] Trying to connect to route on js_egeo_kv_shard1_infile_node01:6222 (150.0.0.73:6222)\n
 [DBG] 150.0.0.73:6222 - rid:15 - Route connect msg sent\n
 [DBG] Trying to connect to route on js_egeo_kv_shard1_infile_node05:6222 (10.20.126.30:6222)\n
 [DBG] RAFT [aZPpTWMr - C-R5F-GQUbjX2S] Replaying state of 196 entries\n
 [INF] 150.0.0.73:6222 - rid:14 - Route connection created\n
 [DBG] Trying to connect to route on js_egeo_kv_shard1_infile_node03:6222 (150.0.0.75:6222)\n
 [DBG] Trying to connect to route on js_egeo_kv_shard1_infile_node05:6222 (10.20.126.30:6222)\n
 [DBG] Trying to connect to route on js_egeo_kv_shard1_infile_node02:6222 (150.0.0.71:6222)\n
 [DBG] Starting stream monitor for '$G \u003e KV_wh-px-egeo-e-shard1-kv' [S-R5F-Z6jshyaK]\n
 [ERR] Error trying to connect to route (attempt 1): dial tcp 10.20.126.30:6222: connect: connection refused\n
 [DBG] Apply stream entries for '$G \u003e KV_wh-px-egeo-e-shard1-kv' skipping message with sequence 115317 with last of 115353\n
 [DBG] Apply stream entries for '$G \u003e KV_wh-px-egeo-e-shard1-kv' skipping message with sequence 115318 with last of 115353\n
 [DBG] Apply stream entries for '$G \u003e KV_wh-px-egeo-e-shard1-kv' skipping message with sequence 115319 with last of 115353\n
 [DBG] Apply stream entries for '$G \u003e KV_wh-px-egeo-e-shard1-kv' skipping message with sequence 115320 with last of 115353\n
 [DBG] Apply stream entries for '$G \u003e KV_wh-px-egeo-e-shard1-kv' skipping message with sequence 115321 with last of 115353\n
 [DBG] Apply stream entries for '$G \u003e KV_wh-px-egeo-e-shard1-kv' skipping message with sequence 115322 with last of 115353\n
 [INF] 150.0.0.73:6222 - rid:15 - Route connection created\n
 [ERR] Error trying to connect to route (attempt 1): dial tcp 10.20.126.30:6222: connect: connection refused\n
 [DBG] Trying to connect to route on js_egeo_kv_shard1_infile_node04:6222 (150.0.0.77:6222)\n
 [DBG] Trying to connect to route on js_egeo_kv_shard1_infile_node03:6222 (150.0.0.75:6222)\n
 [DBG] Apply stream entries for '$G \u003e KV_wh-px-egeo-e-shard1-kv' skipping message with sequence 115323 with last of 115353\n
 [DBG] Apply stream entries for '$G \u003e KV_wh-px-egeo-e-shard1-kv' skipping message with sequence 115324 with last of 115353\n
 [DBG] Apply stream entries for '$G \u003e KV_wh-px-egeo-e-shard1-kv' skipping message with sequence 115325 with last of 115353\n
 [DBG] Apply stream entries for '$G \u003e KV_wh-px-egeo-e-shard1-kv' skipping message with sequence 115326 with last of 115353\n
 [DBG] Apply stream entries for '$G \u003e KV_wh-px-egeo-e-shard1-kv' skipping message with sequence 115327 with last of 115353\n
 [DBG] Apply stream entries for '$G \u003e KV_wh-px-egeo-e-shard1-kv' skipping message with sequence 115328 with last of 115353\n
 [DBG] Apply stream entries for '$G \u003e KV_wh-px-egeo-e-shard1-kv' skipping message with sequence 115329 with last of 115353\n
 [DBG] Apply stream entries for '$G \u003e KV_wh-px-egeo-e-shard1-kv' skipping message with sequence 115330 with last of 115353\n
 [DBG] Apply stream entries for '$G \u003e KV_wh-px-egeo-e-shard1-kv' skipping message with sequence 115331 with last of 115353\n
 [DBG] Apply stream entries for '$G \u003e KV_wh-px-egeo-e-shard1-kv' skipping message with sequence 115332 with last of 115353\n
 [DBG] Apply stream entries for '$G \u003e KV_wh-px-egeo-e-shard1-kv' skipping message with sequence 115333 with last of 115353\n
 [DBG] Apply stream entries for '$G \u003e KV_wh-px-egeo-e-shard1-kv' skipping message with sequence 115334 with last of 115353\n
 [DBG] Apply stream entries for '$G \u003e KV_wh-px-egeo-e-shard1-kv' skipping message with sequence 115335 with last of 115353\n
 [DBG] Apply stream entries for '$G \u003e KV_wh-px-egeo-e-shard1-kv' skipping message with sequence 115336 with last of 115353\n
 [DBG] Apply stream entries for '$G \u003e KV_wh-px-egeo-e-shard1-kv' skipping message with sequence 115337 with last of 115353\n
 [DBG] Apply stream entries for '$G \u003e KV_wh-px-egeo-e-shard1-kv' skipping message with sequence 115338 with last of 115353\n
 [DBG] Apply stream entries for '$G \u003e KV_wh-px-egeo-e-shard1-kv' skipping message with sequence 115339 with last of 115353\n
 [DBG] Apply stream entries for '$G \u003e KV_wh-px-egeo-e-shard1-kv' skipping message with sequence 115340 with last of 115353\n
 [DBG] Apply stream entries for '$G \u003e KV_wh-px-egeo-e-shard1-kv' skipping message with sequence 115341 with last of 115353\n
 [DBG] Apply stream entries for '$G \u003e KV_wh-px-egeo-e-shard1-kv' skipping message with sequence 115342 with last of 115353\n
 [DBG] Apply stream entries for '$G \u003e KV_wh-px-egeo-e-shard1-kv' skipping message with sequence 115343 with last of 115353\n
 [DBG] 150.0.0.77:6222 - rid:17 - Route connect msg sent\n
 [INF] 150.0.0.77:6222 - rid:17 - Route connection created\n
 [DBG] Apply stream entries for '$G \u003e KV_wh-px-egeo-e-shard1-kv' skipping message with sequence 115344 with last of 115353\n
 [DBG] Apply stream entries for '$G \u003e KV_wh-px-egeo-e-shard1-kv' skipping message with sequence 115345 with last of 115353\n
 [DBG] Apply stream entries for '$G \u003e KV_wh-px-egeo-e-shard1-kv' skipping message with sequence 115346 with last of 115353\n
 [DBG] 150.0.0.77:6222 - rid:18 - Route connect msg sent\n
 [INF] 150.0.0.77:6222 - rid:18 - Route connection created\n
 [DBG] Apply stream entries for '$G \u003e KV_wh-px-egeo-e-shard1-kv' skipping message with sequence 115347 with last of 115353\n
 [DBG] Apply stream entries for '$G \u003e KV_wh-px-egeo-e-shard1-kv' skipping message with sequence 115348 with last of 115353\n
 [DBG] Apply stream entries for '$G \u003e KV_wh-px-egeo-e-shard1-kv' skipping message with sequence 115349 with last of 115353\n
 [DBG] Apply stream entries for '$G \u003e KV_wh-px-egeo-e-shard1-kv' skipping message with sequence 115350 with last of 115353\n
 [DBG] Apply stream entries for '$G \u003e KV_wh-px-egeo-e-shard1-kv' skipping message with sequence 115351 with last of 115353\n
 [DBG] Apply stream entries for '$G \u003e KV_wh-px-egeo-e-shard1-kv' skipping message with sequence 115352 with last of 115353\n
 [DBG] Apply stream entries for '$G \u003e KV_wh-px-egeo-e-shard1-kv' skipping message with sequence 115353 with last of 115353\n
 [INF] 150.0.1.124:59030 - rid:16 - Route connection created\n
 [DBG] 150.0.0.75:6222 - rid:19 - Route connect msg sent\n
 [INF] 150.0.0.75:6222 - rid:19 - Route connection created\n
 [INF] 150.0.0.73:6222 - rid:15 - Router connection closed: Duplicate Route\n
 [TRC] 150.0.1.124:59030 - rid:16 - \u003c\u003c- [CONNECT {\"echo\":true,\"verbose\":false,\"pedantic\":false,\"user\":\"js_egeo_kv_cluster03_infile_user\",\"pass\":\"[REDACTED]\",\"tls_required\":false,\"headers\":true,\"name\":\"NAVQASAKCWEM567JJSFHP4DTY2GAHDDQYOMOTPM4MOIYIJIV673YFNC7\",\"cluster\":\"js_egeo_kv_shard1_infile\",\"lnoc\":true}]\n
 [DBG] 150.0.0.71:6222 - rid:20 - Route connect msg sent\n
 [INF] 150.0.0.71:6222 - rid:20 - Route connection created\n
 [INF] 150.0.1.124:59014 - rid:21 - Route connection created\n
 [TRC] 150.0.1.124:59014 - rid:21 - \u003c\u003c- [CONNECT {\"echo\":true,\"verbose\":false,\"pedantic\":false,\"user\":\"js_egeo_kv_cluster03_infile_user\",\"pass\":\"[REDACTED]\",\"tls_required\":false,\"headers\":true,\"name\":\"NAVQASAKCWEM567JJSFHP4DTY2GAHDDQYOMOTPM4MOIYIJIV673YFNC7\",\"cluster\":\"js_egeo_kv_shard1_infile\",\"lnoc\":true}]\n
 [DBG] Detected route to self, ignoring \"nats-route://js_egeo_kv_cluster03_infile_user:xxxxx@js_egeo_kv_shard1_infile_node01:6222\"\n
 [INF] 150.0.1.124:59030 - rid:16 - Router connection closed: Client Closed\n
 [DBG] 150.0.0.75:6222 - rid:22 - Route connect msg sent\n
 [INF] 150.0.0.75:6222 - rid:22 - Route connection created\n
 [DBG] 150.0.0.71:6222 - rid:23 - Route connect msg sent\n
 [INF] 150.0.0.71:6222 - rid:23 - Route connection created\n
 [INF] 150.0.0.73:6222 - rid:14 - Router connection closed: Duplicate Route\n
 [DBG] Detected route to self, ignoring \"nats-route://js_egeo_kv_cluster03_infile_user:xxxxx@js_egeo_kv_shard1_infile_node01:6222\"\n
 [INF] 150.0.1.124:59014 - rid:21 - Router connection closed: Client Closed\n
 [DBG] RAFT [aZPpTWMr - C-R5F-GQUbjX2S] Started\n
 [DBG] JetStream cluster, consumer '$G \u003e KV_wh-px-egeo-e-shard1-kv \u003e WhEgeoKvCronToEgeoCache1_consumer' was already running\n
 [DBG] 150.0.0.75:6222 - rid:19 - Sent local subscriptions to route\n
 [DBG] 150.0.0.75:6222 - rid:19 - Registering remote route \"NC5YXRJT4EBEHLOEFBDCDEBHKZI5ILHKT7OORNVDUJJUYBTIXM4SSKVS\" for account \"$SYS\"\n
 [DBG] 150.0.0.75:6222 - rid:22 - Registering remote route \"NC5YXRJT4EBEHLOEFBDCDEBHKZI5ILHKT7OORNVDUJJUYBTIXM4SSKVS\"\n
 [DBG] 150.0.0.77:6222 - rid:17 - Sent local subscriptions to route\n
 [DBG] 150.0.0.77:6222 - rid:17 - Registering remote route \"NBV22NKSB3UESVPFJCHQZ545AI26TOCUCLGUH4KL5RBO2GVRYJWOI5OA\" for account \"$SYS\"\n
 [DBG] 150.0.0.71:6222 - rid:20 - Registering remote route \"NBKYG4TJEC5AK277OUOJO6RYHDR7Q6FJWOUEWG7KIQ5RMOARWVTXNCKG\"\n
 [DBG] Apply stream entries for '$G \u003e KV_wh-px-egeo-e-shard1-kv' got error processing message: last sequence by subject mismatch: 112238 vs 115352\n
 [DBG] Starting consumer monitor for '$G \u003e KV_wh-px-egeo-e-shard1-kv \u003e WhEgeoKvCronToEgeoCache1_consumer' [C-R5F-GQUbjX2S]\n
 [DBG] 150.0.0.71:6222 - rid:23 - Sent local subscriptions to route\n
 [DBG] 150.0.0.71:6222 - rid:23 - Registering remote route \"NBKYG4TJEC5AK277OUOJO6RYHDR7Q6FJWOUEWG7KIQ5RMOARWVTXNCKG\" for account \"$SYS\"\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- [RS+ $SYS.REQ.ACCOUNT.*.CONNS]\n
 [DBG] 150.0.0.77:6222 - rid:18 - Registering remote route \"NBV22NKSB3UESVPFJCHQZ545AI26TOCUCLGUH4KL5RBO2GVRYJWOI5OA\"\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- [RS+ $SYS.REQ.ACCOUNT.NSUBS]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- [RS+ $NRG.AE.C-R5F-GQUbjX2S]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- [RS+ $JSC.CI.$G.KV_wh-px-egeo-e-shard1-kv.WhEgeoKvCronToEgeoCache1_consumer]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- [RS+ $NRG.R.XuhYhnIZ]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- [RS+ $SYS.REQ.SERVER.NBV22NKSB3UESVPFJCHQZ545AI26TOCUCLGUH4KL5RBO2GVRYJWOI5OA.STATSZ]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- [RS+ $SYS.ACCOUNT.*.LEAFNODE.CONNECT]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- [RS+ $SYS.REQ.SERVER.NBV22NKSB3UESVPFJCHQZ545AI26TOCUCLGUH4KL5RBO2GVRYJWOI5OA.LDM]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- [RS+ $SYS.SERVER.*.STATSZ]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- [RS+ $SYS.REQ.SERVER.NBV22NKSB3UESVPFJCHQZ545AI26TOCUCLGUH4KL5RBO2GVRYJWOI5OA.ACCOUNTZ]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- [RS+ $SYS.REQ.ACCOUNT.*.INFO]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- [RS+ $SYS.REQ.USER.*.INFO]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- [RS+ $NRG.R.gMijEdZe]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- [RS+ $SYS.REQ.SERVER.PING]\n
 [TRC] 150.0.0.75:6222 - rid:19 - \u003c\u003c- [RS+ $SYS.REQ.SERVER.PING.HEALTHZ]\n
 [TRC] 150.0.0.75:6222 - rid:19 - \u003c\u003c- [RS+ $SYS.ACCOUNT.*.LEAFNODE.CONNECT]\n
 [TRC] 150.0.0.75:6222 - rid:19 - \u003c\u003c- [RS+ $NRG.AE.C-R5F-GQUbjX2S]\n
 [TRC] 150.0.0.75:6222 - rid:19 - \u003c\u003c- [RS+ $SYS.REQ.SERVER.PING.SUBSZ]\n
 [TRC] 150.0.0.75:6222 - rid:19 - \u003c\u003c- [RS+ $SYS.ACCOUNT.*.CLAIMS.UPDATE]\n
 [TRC] 150.0.0.75:6222 - rid:19 - \u003c\u003c- [RS+ $JS.API.\u003e]\n
 [TRC] 150.0.0.75:6222 - rid:19 - \u003c\u003c- [RS+ $SYS.SERVER.ACCOUNT.*.CONNS]\n
 [TRC] 150.0.0.75:6222 - rid:19 - \u003c\u003c- [RS+ $SYS.SERVER.*.STATSZ]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- [RS+ $SYS.REQ.SERVER.PING.GATEWAYZ]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- [RS+ $SYS.REQ.SERVER.PING.ROUTEZ]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- [RS+ $NRG.R.ZOWVBgd7]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- [RS+ $SYS.REQ.SERVER.NBV22NKSB3UESVPFJCHQZ545AI26TOCUCLGUH4KL5RBO2GVRYJWOI5OA.CONNZ]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- [RS+ $SYS.REQ.ACCOUNT.*.CLAIMS.UPDATE]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- [RS+ _R_.4lk9bK.\u003e]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- [RS+ $SYS._INBOX_.NBV22NKSB3UESVPFJCHQZ545AI26TOCUCLGUH4KL5RBO2GVRYJWOI5OA]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- [RS+ $SYS.REQ.SERVER.PING.VARZ]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- [RS+ $SYS.REQ.SERVER.PING.SUBSZ]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- [RS+ $SYS.REQ.SERVER.PING.HEALTHZ]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- [RS+ $SYS.REQ.SERVER.NBV22NKSB3UESVPFJCHQZ545AI26TOCUCLGUH4KL5RBO2GVRYJWOI5OA.LEAFZ]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- [RS+ $SYS.REQ.ACCOUNT.*.LEAFZ]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- [RS+ $SYS.REQ.SERVER.NBV22NKSB3UESVPFJCHQZ545AI26TOCUCLGUH4KL5RBO2GVRYJWOI5OA.KICK]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- [RS+ $JSC.ARU.$G.*]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- [RS+ $SYS.REQ.SERVER.PING.STATSZ]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- [RS+ $NRG.P.C-R5F-GQUbjX2S]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- [RS+ $SYS.ACCOUNT.*.CLAIMS.UPDATE]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- [RS+ $SYS.REQ.SERVER.PING.IDZ]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- [RS+ $SYS.REQ.SERVER.PING.LEAFZ]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- [RS+ $SYS.SERVER.ACCOUNT.*.CONNS]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- [RS+ $JS.API.\u003e]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- [RS+ $SYS.REQ.ACCOUNT.*.CONNZ]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- [RS+ $SYS.LATENCY.M2.v2fh5Gis]\n
 [TRC] 150.0.0.71:6222 - rid:23 - \u003c\u003c- [RS+ $SYS._INBOX.dGVGKtkV.*]\n
 [TRC] 150.0.0.75:6222 - rid:19 - \u003c\u003c- [RS+ $SYS.SERVER.*.LAMEDUCK]\n
 [TRC] 150.0.0.71:6222 - rid:23 - \u003c\u003c- [RS+ $SYS.ACCOUNT.*.LEAFNODE.CONNECT]\n
 [TRC] 150.0.0.75:6222 - rid:19 - \u003c\u003c- [RS+ $SYS.REQ.SERVER.NC5YXRJT4EBEHLOEFBDCDEBHKZI5ILHKT7OORNVDUJJUYBTIXM4SSKVS.PROFILEZ]\n
 [TRC] 150.0.0.75:6222 - rid:19 - \u003c\u003c- [RS+ $SYS.REQ.SERVER.NC5YXRJT4EBEHLOEFBDCDEBHKZI5ILHKT7OORNVDUJJUYBTIXM4SSKVS.ROUTEZ]\n
 [TRC] 150.0.0.75:6222 - rid:19 - \u003c\u003c- [RS+ $SYS.REQ.SERVER.NC5YXRJT4EBEHLOEFBDCDEBHKZI5ILHKT7OORNVDUJJUYBTIXM4SSKVS.HEALTHZ]\n
 [TRC] 150.0.0.75:6222 - rid:19 - \u003c\u003c- [RS+ $SYS.REQ.SERVER.NC5YXRJT4EBEHLOEFBDCDEBHKZI5ILHKT7OORNVDUJJUYBTIXM4SSKVS.GATEWAYZ]\n
 [TRC] 150.0.0.75:6222 - rid:19 - \u003c\u003c- [RS+ $SYS.REQ.ACCOUNT.*.STATZ]\n
 [TRC] 150.0.0.75:6222 - rid:19 - \u003c\u003c- [RS+ $SYS.REQ.ACCOUNT.NSUBS]\n
 [TRC] 150.0.0.75:6222 - rid:19 - \u003c\u003c- [RS+ $NRG.R.WzCP3Sb5]\n
 [TRC] 150.0.0.75:6222 - rid:19 - \u003c\u003c- [RS+ $SYS.LATENCY.M2.JrN1hGZO]\n
 [TRC] 150.0.0.75:6222 - rid:19 - \u003c\u003c- [RS+ $NRG.V._meta_]\n
 [TRC] 150.0.0.75:6222 - rid:19 - \u003c\u003c- [RS+ $NRG.V.C-R5F-GQUbjX2S]\n
 [TRC] 150.0.0.75:6222 - rid:19 - \u003c\u003c- [RS+ $JSC.ARU.$G.*]\n
 [TRC] 150.0.0.75:6222 - rid:19 - \u003c\u003c- [RS+ $SYS.REQ.SERVER.NC5YXRJT4EBEHLOEFBDCDEBHKZI5ILHKT7OORNVDUJJUYBTIXM4SSKVS.IDZ]\n
 [TRC] 150.0.0.75:6222 - rid:19 - \u003c\u003c- [RS+ $SYS.REQ.SERVER.PING.IDZ]\n
 [TRC] 150.0.0.75:6222 - rid:19 - \u003c\u003c- [RS+ $SYS.REQ.ACCOUNT.*.CONNS]\n
 [TRC] 150.0.0.75:6222 - rid:19 - \u003c\u003c- [RS+ $SYS.REQ.SERVER.PING.ACCOUNTZ]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- [RS+ $NRG.R.vj7pYXm0]\n
 [TRC] 150.0.0.75:6222 - rid:19 - \u003c\u003c- [RS+ $SYS.REQ.SERVER.NC5YXRJT4EBEHLOEFBDCDEBHKZI5ILHKT7OORNVDUJJUYBTIXM4SSKVS.EXPVARZ]\n
 [TRC] 150.0.0.71:6222 - rid:23 - \u003c\u003c- [RS+ $SYS.JSC.STREAM.ASSIGNMENT.RESULT]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- [RS+ $NRG.AE._meta_]\n
 [TRC] 150.0.0.71:6222 - rid:23 - \u003c\u003c- [RS+ $SYS.SERVER.*.SHUTDOWN]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- [RS+ $SYS.REQ.SERVER.PING.EXPVARZ]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- [RS+ $SYS.REQ.SERVER.NBV22NKSB3UESVPFJCHQZ545AI26TOCUCLGUH4KL5RBO2GVRYJWOI5OA.VARZ]\n
 [TRC] 150.0.0.71:6222 - rid:23 - \u003c\u003c- [RS+ $SYS.REQ.SERVER.PING.VARZ]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- [RS+ $SYS.REQ.SERVER.PING.CONNZ]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- [RS+ $SYS.REQ.SERVER.NBV22NKSB3UESVPFJCHQZ545AI26TOCUCLGUH4KL5RBO2GVRYJWOI5OA.PROFILEZ]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- [RS+ $SYS.REQ.SERVER.NBV22NKSB3UESVPFJCHQZ545AI26TOCUCLGUH4KL5RBO2GVRYJWOI5OA.RELOAD]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- [RS+ $SYS.SERVER.*.LAMEDUCK]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- [RS+ $SYS.REQ.SERVER.NBV22NKSB3UESVPFJCHQZ545AI26TOCUCLGUH4KL5RBO2GVRYJWOI5OA.IDZ]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- [RS+ $SYS.REQ.SERVER.NBV22NKSB3UESVPFJCHQZ545AI26TOCUCLGUH4KL5RBO2GVRYJWOI5OA.HEALTHZ]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- [RS+ $SYS.REQ.SERVER.NBV22NKSB3UESVPFJCHQZ545AI26TOCUCLGUH4KL5RBO2GVRYJWOI5OA.JSZ]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- [RS+ $SYS.REQ.ACCOUNT.*.JSZ]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- [RS+ $SYS.REQ.ACCOUNT.*.SUBSZ]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- [RS+ $NRG.R.PHDNDLZa]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- [RS+ $SYS.SERVER.*.SHUTDOWN]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- [RS+ $SYS.REQ.ACCOUNT.*.STATZ]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- [RS+ $NRG.RP.C-R5F-GQUbjX2S]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- [RS+ $SYS.REQ.SERVER.NBV22NKSB3UESVPFJCHQZ545AI26TOCUCLGUH4KL5RBO2GVRYJWOI5OA.SUBSZ]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- [RS+ $NRG.AE.S-R5F-Z6jshyaK]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- [RS+ $JSC.SI.$G.KV_wh-px-egeo-e-shard1-kv]\n
 [TRC] 150.0.0.75:6222 - rid:19 - \u003c\u003c- [RS+ $SYS.REQ.ACCOUNT.*.LEAFZ]\n
 [TRC] 150.0.0.75:6222 - rid:19 - \u003c\u003c- [RS+ $SYS.REQ.SERVER.PING.ROUTEZ]\n
 [TRC] 150.0.0.75:6222 - rid:19 - \u003c\u003c- [RS+ $SYS.REQ.SERVER.NC5YXRJT4EBEHLOEFBDCDEBHKZI5ILHKT7OORNVDUJJUYBTIXM4SSKVS.STATSZ]\n
 [TRC] 150.0.0.75:6222 - rid:19 - \u003c\u003c- [RS+ $SYS.REQ.SERVER.NC5YXRJT4EBEHLOEFBDCDEBHKZI5ILHKT7OORNVDUJJUYBTIXM4SSKVS.LEAFZ]\n
 [TRC] 150.0.0.75:6222 - rid:19 - \u003c\u003c- [RS+ $NRG.R.4p5Rqq3j]\n
 [TRC] 150.0.0.75:6222 - rid:19 - \u003c\u003c- [RS+ $SYS.REQ.ACCOUNT.*.CLAIMS.UPDATE]\n
 [TRC] 150.0.0.75:6222 - rid:19 - \u003c\u003c- [RS+ $SYS.REQ.ACCOUNT.*.INFO]\n
 [TRC] 150.0.0.75:6222 - rid:19 - \u003c\u003c- [RS+ $SYS.REQ.ACCOUNT.*.CONNZ]\n
 [TRC] 150.0.0.75:6222 - rid:19 - \u003c\u003c- [RS+ $SYS.REQ.SERVER.NC5YXRJT4EBEHLOEFBDCDEBHKZI5ILHKT7OORNVDUJJUYBTIXM4SSKVS.JSZ]\n
 [TRC] 150.0.0.75:6222 - rid:19 - \u003c\u003c- [RS+ $SYS.REQ.SERVER.PING.CONNZ]\n
 [TRC] 150.0.0.75:6222 - rid:19 - \u003c\u003c- [RS+ $SYS.REQ.SERVER.NC5YXRJT4EBEHLOEFBDCDEBHKZI5ILHKT7OORNVDUJJUYBTIXM4SSKVS.RELOAD]\n
 [TRC] 150.0.0.75:6222 - rid:19 - \u003c\u003c- [RS+ $NRG.V.S-R5F-Z6jshyaK]\n
 [TRC] 150.0.0.75:6222 - rid:19 - \u003c\u003c- [RS+ $SYS._INBOX.JrN1hGZO.*]\n
 [TRC] 150.0.0.75:6222 - rid:19 - \u003c\u003c- [RS+ $SYS.REQ.SERVER.PING.GATEWAYZ]\n
 [TRC] 150.0.0.75:6222 - rid:19 - \u003c\u003c- [RS+ $SYS.REQ.SERVER.PING.LEAFZ]\n
 [TRC] 150.0.0.75:6222 - rid:19 - \u003c\u003c- [RS+ $SYS.REQ.SERVER.NC5YXRJT4EBEHLOEFBDCDEBHKZI5ILHKT7OORNVDUJJUYBTIXM4SSKVS.KICK]\n
 [TRC] 150.0.0.75:6222 - rid:19 - \u003c\u003c- [RS+ $SYS.REQ.SERVER.NC5YXRJT4EBEHLOEFBDCDEBHKZI5ILHKT7OORNVDUJJUYBTIXM4SSKVS.CONNZ]\n
 [TRC] 150.0.0.75:6222 - rid:19 - \u003c\u003c- [RS+ $SYS.REQ.SERVER.PING]\n
 [TRC] 150.0.0.75:6222 - rid:19 - \u003c\u003c- [RS+ $SYS.REQ.SERVER.PING.PROFILEZ]\n
 [TRC] 150.0.0.75:6222 - rid:19 - \u003c\u003c- [RS+ $SYS.REQ.SERVER.NC5YXRJT4EBEHLOEFBDCDEBHKZI5ILHKT7OORNVDUJJUYBTIXM4SSKVS.VARZ]\n
 [TRC] 150.0.0.75:6222 - rid:19 - \u003c\u003c- [RS+ $SYS.REQ.SERVER.PING.VARZ]\n
 [TRC] 150.0.0.75:6222 - rid:19 - \u003c\u003c- [RS+ $SYS.REQ.SERVER.PING.EXPVARZ]\n
 [TRC] 150.0.0.75:6222 - rid:19 - \u003c\u003c- [RS+ $SYS.REQ.SERVER.NC5YXRJT4EBEHLOEFBDCDEBHKZI5ILHKT7OORNVDUJJUYBTIXM4SSKVS.LDM]\n
 [TRC] 150.0.0.75:6222 - rid:19 - \u003c\u003c- [RS+ $NRG.AE.S-R5F-Z6jshyaK]\n
 [TRC] 150.0.0.75:6222 - rid:19 - \u003c\u003c- [RS+ $SYS.SERVER.*.SHUTDOWN]\n
 [TRC] 150.0.0.75:6222 - rid:19 - \u003c\u003c- [RS+ $SYS.REQ.ACCOUNT.*.JSZ]\n
 [TRC] 150.0.0.75:6222 - rid:19 - \u003c\u003c- [RS+ $NRG.R.eZnf8DVR]\n
 [TRC] 150.0.0.75:6222 - rid:19 - \u003c\u003c- [RS+ $SYS.REQ.SERVER.NC5YXRJT4EBEHLOEFBDCDEBHKZI5ILHKT7OORNVDUJJUYBTIXM4SSKVS.ACCOUNTZ]\n
 [TRC] 150.0.0.75:6222 - rid:19 - \u003c\u003c- [RS+ $SYS.REQ.SERVER.PING.JSZ]\n
 [TRC] 150.0.0.75:6222 - rid:19 - \u003c\u003c- [RS+ $SYS.REQ.ACCOUNT.*.SUBSZ]\n
 [TRC] 150.0.0.75:6222 - rid:19 - \u003c\u003c- [RS+ $SYS.REQ.SERVER.NC5YXRJT4EBEHLOEFBDCDEBHKZI5ILHKT7OORNVDUJJUYBTIXM4SSKVS.SUBSZ]\n
 [TRC] 150.0.0.75:6222 - rid:19 - \u003c\u003c- [RS+ $SYS.REQ.ACCOUNT.PING.STATZ]\n
 [TRC] 150.0.0.75:6222 - rid:19 - \u003c\u003c- [RS+ $NRG.R.EPDGUg9E]\n
 [TRC] 150.0.0.75:6222 - rid:19 - \u003c\u003c- [RS+ $JSC.SI.$G.KV_wh-px-egeo-e-shard1-kv]\n
 [TRC] 150.0.0.75:6222 - rid:19 - \u003c\u003c- [RS+ $NRG.R.Mao3BVVT]\n
 [TRC] 150.0.0.75:6222 - rid:19 - \u003c\u003c- [RS+ $SYS.REQ.USER.*.INFO]\n
 [TRC] 150.0.0.75:6222 - rid:19 - \u003c\u003c- [RS+ $SYS.REQ.SERVER.PING.STATSZ]\n
 [TRC] 150.0.0.75:6222 - rid:19 - \u003c\u003c- [RS+ $NRG.AE._meta_]\n
 [TRC] 150.0.0.75:6222 - rid:19 - \u003c\u003c- [RS+ $NRG.R.dGEDS1Jm]\n
 [TRC] 150.0.0.75:6222 - rid:19 - \u003c\u003c- [RS+ $SYS._INBOX_.NC5YXRJT4EBEHLOEFBDCDEBHKZI5ILHKT7OORNVDUJJUYBTIXM4SSKVS]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- [RS+ $SYS._INBOX.v2fh5Gis.*]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- [RS+ $SYS.REQ.SERVER.NBV22NKSB3UESVPFJCHQZ545AI26TOCUCLGUH4KL5RBO2GVRYJWOI5OA.ROUTEZ]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- [RS+ $SYS.REQ.SERVER.PING.PROFILEZ]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- [RS+ $SYS.REQ.SERVER.PING.ACCOUNTZ]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- [RS+ $SYS.REQ.ACCOUNT.PING.STATZ]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- [RS+ $NRG.V._meta_]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- [RS+ $NRG.V.C-R5F-GQUbjX2S]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- [RS+ $NRG.R.JORnpMu0]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- [RS+ $SYS.REQ.SERVER.NBV22NKSB3UESVPFJCHQZ545AI26TOCUCLGUH4KL5RBO2GVRYJWOI5OA.EXPVARZ]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- [RS+ $SYS.REQ.SERVER.PING.JSZ]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- [RS+ $NRG.V.S-R5F-Z6jshyaK]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- [RS+ $SYS.REQ.SERVER.NBV22NKSB3UESVPFJCHQZ545AI26TOCUCLGUH4KL5RBO2GVRYJWOI5OA.GATEWAYZ]\n
 [TRC] 150.0.0.71:6222 - rid:23 - \u003c\u003c- [RS+ $NRG.AE.S-R5F-Z6jshyaK]\n
 [TRC] 150.0.0.71:6222 - rid:23 - \u003c\u003c- [RS+ $NRG.V.C-R5F-GQUbjX2S]\n
 [TRC] 150.0.0.71:6222 - rid:23 - \u003c\u003c- [RS+ $SYS.SERVER.ACCOUNT.*.CONNS]\n
 [TRC] 150.0.0.71:6222 - rid:23 - \u003c\u003c- [RS+ $SYS.REQ.SERVER.PING.LEAFZ]\n
 [TRC] 150.0.0.71:6222 - rid:23 - \u003c\u003c- [RS+ $SYS.REQ.SERVER.PING.PROFILEZ]\n
 [TRC] 150.0.0.71:6222 - rid:23 - \u003c\u003c- [RS+ $SYS.REQ.SERVER.NBKYG4TJEC5AK277OUOJO6RYHDR7Q6FJWOUEWG7KIQ5RMOARWVTXNCKG.JSZ]\n
 [TRC] 150.0.0.71:6222 - rid:23 - \u003c\u003c- [RS+ $SYS.REQ.USER.*.INFO]\n
 [TRC] 150.0.0.71:6222 - rid:23 - \u003c\u003c- [RS+ $NRG.RP._meta_]\n
 [TRC] 150.0.0.71:6222 - rid:23 - \u003c\u003c- [RS+ $JS.API.ACCOUNT.PURGE.*]\n
 [TRC] 150.0.0.71:6222 - rid:23 - \u003c\u003c- [RS+ $SYS.REQ.ACCOUNT.*.INFO]\n
 [TRC] 150.0.0.71:6222 - rid:23 - \u003c\u003c- [RS+ $NRG.R.Ky0k8ifF]\n
 [TRC] 150.0.0.71:6222 - rid:23 - \u003c\u003c- [RS+ $NRG.R.NHOdYjYZ]\n
 [TRC] 150.0.0.71:6222 - rid:23 - \u003c\u003c- [RS+ $NRG.AE.C-R5F-GQUbjX2S]\n
 [TRC] 150.0.0.71:6222 - rid:23 - \u003c\u003c- [RS+ $SYS.REQ.SERVER.NBKYG4TJEC5AK277OUOJO6RYHDR7Q6FJWOUEWG7KIQ5RMOARWVTXNCKG.GATEWAYZ]\n
 [TRC] 150.0.0.71:6222 - rid:23 - \u003c\u003c- [RS+ $SYS.REQ.SERVER.NBKYG4TJEC5AK277OUOJO6RYHDR7Q6FJWOUEWG7KIQ5RMOARWVTXNCKG.EXPVARZ]\n
 [TRC] 150.0.0.71:6222 - rid:23 - \u003c\u003c- [RS+ $NRG.R.fYtaPy9U]\n
 [TRC] 150.0.0.71:6222 - rid:23 - \u003c\u003c- [RS+ $SYS.REQ.SERVER.NBKYG4TJEC5AK277OUOJO6RYHDR7Q6FJWOUEWG7KIQ5RMOARWVTXNCKG.HEALTHZ]\n
 [TRC] 150.0.0.71:6222 - rid:23 - \u003c\u003c- [RS+ $SYS.REQ.SERVER.NBKYG4TJEC5AK277OUOJO6RYHDR7Q6FJWOUEWG7KIQ5RMOARWVTXNCKG.VARZ]\n
 [TRC] 150.0.0.71:6222 - rid:23 - \u003c\u003c- [RS+ $SYS.JSC.CONSUMER.ASSIGNMENT.RESULT]\n
 [TRC] 150.0.0.71:6222 - rid:23 - \u003c\u003c- [RS+ $JS.API.SERVER.REMOVE]\n
 [TRC] 150.0.0.71:6222 - rid:23 - \u003c\u003c- [RS+ $JS.API.ACCOUNT.STREAM.CANCEL_MOVE.*.*]\n
 [TRC] 150.0.0.71:6222 - rid:23 - \u003c\u003c- [RS+ $SYS.REQ.SERVER.NBKYG4TJEC5AK277OUOJO6RYHDR7Q6FJWOUEWG7KIQ5RMOARWVTXNCKG.PROFILEZ]\n
 [TRC] 150.0.0.71:6222 - rid:23 - \u003c\u003c- [RS+ $SYS.REQ.SERVER.NBKYG4TJEC5AK277OUOJO6RYHDR7Q6FJWOUEWG7KIQ5RMOARWVTXNCKG.KICK]\n
 [TRC] 150.0.0.71:6222 - rid:23 - \u003c\u003c- [RS+ $JS.API.ACCOUNT.STREAM.MOVE.*.*]\n
 [TRC] 150.0.0.71:6222 - rid:23 - \u003c\u003c- [RS+ $JSC.SI.$G.KV_wh-px-egeo-e-shard1-kv]\n
 [TRC] 150.0.0.71:6222 - rid:23 - \u003c\u003c- [RS+ $SYS.REQ.SERVER.NBKYG4TJEC5AK277OUOJO6RYHDR7Q6FJWOUEWG7KIQ5RMOARWVTXNCKG.SUBSZ]\n
 [TRC] 150.0.0.71:6222 - rid:23 - \u003c\u003c- [RS+ $JSC.ARU.$G.*]\n
 [TRC] 150.0.0.71:6222 - rid:23 - \u003c\u003c- [RS+ $NRG.R.wHdJHKmd]\n
 [TRC] 150.0.0.71:6222 - rid:23 - \u003c\u003c- [RS+ $SYS.REQ.SERVER.PING]\n
 [TRC] 150.0.0.71:6222 - rid:23 - \u003c\u003c- [RS+ $SYS.REQ.SERVER.NBKYG4TJEC5AK277OUOJO6RYHDR7Q6FJWOUEWG7KIQ5RMOARWVTXNCKG.ACCOUNTZ]\n
 [TRC] 150.0.0.71:6222 - rid:23 - \u003c\u003c- [RS+ $SYS.LATENCY.M2.dGVGKtkV]\n
 [TRC] 150.0.0.71:6222 - rid:23 - \u003c\u003c- [RS+ $JS.API.\u003e]\n
 [TRC] 150.0.0.71:6222 - rid:23 - \u003c\u003c- [RS+ $JS.API.META.LEADER.STEPDOWN]\n
 [TRC] 150.0.0.71:6222 - rid:23 - \u003c\u003c- [RS+ _R_.yYz1ii.\u003e]\n
 [TRC] 150.0.0.71:6222 - rid:23 - \u003c\u003c- [RS+ $SYS.REQ.SERVER.NBKYG4TJEC5AK277OUOJO6RYHDR7Q6FJWOUEWG7KIQ5RMOARWVTXNCKG.LEAFZ]\n
 [TRC] 150.0.0.71:6222 - rid:23 - \u003c\u003c- [RS+ $SYS.REQ.SERVER.PING.STATSZ]\n
 [TRC] 150.0.0.71:6222 - rid:23 - \u003c\u003c- [RS+ $SYS.REQ.ACCOUNT.*.JSZ]\n
 [TRC] 150.0.0.71:6222 - rid:23 - \u003c\u003c- [RS+ $SYS.REQ.SERVER.NBKYG4TJEC5AK277OUOJO6RYHDR7Q6FJWOUEWG7KIQ5RMOARWVTXNCKG.LDM]\n
 [TRC] 150.0.0.71:6222 - rid:23 - \u003c\u003c- [RS+ $SYS.REQ.ACCOUNT.NSUBS]\n
 [TRC] 150.0.0.71:6222 - rid:23 - \u003c\u003c- [RS+ $SYS.SERVER.*.STATSZ]\n
 [TRC] 150.0.0.71:6222 - rid:23 - \u003c\u003c- [RS+ $SYS.ACCOUNT.*.CLAIMS.UPDATE]\n
 [TRC] 150.0.0.71:6222 - rid:23 - \u003c\u003c- [RS+ $SYS.REQ.SERVER.PING.IDZ]\n
 [TRC] 150.0.0.71:6222 - rid:23 - \u003c\u003c- [RS+ $SYS.REQ.SERVER.PING.SUBSZ]\n
 [TRC] 150.0.0.71:6222 - rid:23 - \u003c\u003c- [RS+ $SYS.REQ.ACCOUNT.*.CONNZ]\n
 [TRC] 150.0.0.71:6222 - rid:23 - \u003c\u003c- [RS+ $SYS.REQ.ACCOUNT.*.LEAFZ]\n
 [TRC] 150.0.0.71:6222 - rid:23 - \u003c\u003c- [RS+ $NRG.AE._meta_]\n
 [TRC] 150.0.0.71:6222 - rid:23 - \u003c\u003c- [RS+ $SYS.REQ.ACCOUNT.*.CLAIMS.UPDATE]\n
 [TRC] 150.0.0.71:6222 - rid:23 - \u003c\u003c- [RS+ $SYS.REQ.SERVER.NBKYG4TJEC5AK277OUOJO6RYHDR7Q6FJWOUEWG7KIQ5RMOARWVTXNCKG.CONNZ]\n
 [TRC] 150.0.0.71:6222 - rid:23 - \u003c\u003c- [RS+ $SYS.REQ.SERVER.PING.CONNZ]\n
 [TRC] 150.0.0.71:6222 - rid:23 - \u003c\u003c- [RS+ $SYS.REQ.SERVER.PING.ACCOUNTZ]\n
 [TRC] 150.0.0.71:6222 - rid:23 - \u003c\u003c- [RS+ $NRG.V.S-R5F-Z6jshyaK]\n
 [TRC] 150.0.0.71:6222 - rid:23 - \u003c\u003c- [RS+ $SYS._INBOX_.NBKYG4TJEC5AK277OUOJO6RYHDR7Q6FJWOUEWG7KIQ5RMOARWVTXNCKG]\n
 [TRC] 150.0.0.71:6222 - rid:23 - \u003c\u003c- [RS+ $SYS.SERVER.*.LAMEDUCK]\n
 [TRC] 150.0.0.71:6222 - rid:23 - \u003c\u003c- [RS+ $SYS.REQ.SERVER.PING.HEALTHZ]\n
 [TRC] 150.0.0.71:6222 - rid:23 - \u003c\u003c- [RS+ $SYS.REQ.SERVER.NBKYG4TJEC5AK277OUOJO6RYHDR7Q6FJWOUEWG7KIQ5RMOARWVTXNCKG.ROUTEZ]\n
 [TRC] 150.0.0.71:6222 - rid:23 - \u003c\u003c- [RS+ $SYS.REQ.SERVER.PING.JSZ]\n
 [TRC] 150.0.0.71:6222 - rid:23 - \u003c\u003c- [RS+ $SYS.REQ.ACCOUNT.*.CONNS]\n
 [TRC] 150.0.0.71:6222 - rid:23 - \u003c\u003c- [RS+ $SYS.REQ.ACCOUNT.PING.STATZ]\n
 [TRC] 150.0.0.71:6222 - rid:23 - \u003c\u003c- [RS+ $SYS.REQ.SERVER.NBKYG4TJEC5AK277OUOJO6RYHDR7Q6FJWOUEWG7KIQ5RMOARWVTXNCKG.RELOAD]\n
 [TRC] 150.0.0.71:6222 - rid:23 - \u003c\u003c- [RS+ $NRG.V._meta_]\n
 [TRC] 150.0.0.71:6222 - rid:23 - \u003c\u003c- [RS+ $NRG.R.dQGRSUBc]\n
 [TRC] 150.0.0.71:6222 - rid:23 - \u003c\u003c- [RS+ $NRG.R.6RjqZtll]\n
 [TRC] 150.0.0.71:6222 - rid:23 - \u003c\u003c- [RS+ $NRG.P.S-R5F-Z6jshyaK]\n
 [TRC] 150.0.0.71:6222 - rid:23 - \u003c\u003c- [RS+ $JSC.SYNC.a5Mq4gCA]\n
 [TRC] 150.0.0.71:6222 - rid:23 - \u003c\u003c- [RS+ $SYS.REQ.SERVER.PING.EXPVARZ]\n
 [TRC] 150.0.0.71:6222 - rid:23 - \u003c\u003c- [RS+ $SYS.REQ.ACCOUNT.*.STATZ]\n
 [TRC] 150.0.0.71:6222 - rid:23 - \u003c\u003c- [RS+ $SYS.REQ.ACCOUNT.*.SUBSZ]\n
 [TRC] 150.0.0.71:6222 - rid:23 - \u003c\u003c- [RS+ $SYS.REQ.SERVER.NBKYG4TJEC5AK277OUOJO6RYHDR7Q6FJWOUEWG7KIQ5RMOARWVTXNCKG.STATSZ]\n
 [TRC] 150.0.0.71:6222 - rid:23 - \u003c\u003c- [RS+ $SYS.REQ.SERVER.PING.GATEWAYZ]\n
 [TRC] 150.0.0.71:6222 - rid:23 - \u003c\u003c- [RS+ $SYS.REQ.SERVER.NBKYG4TJEC5AK277OUOJO6RYHDR7Q6FJWOUEWG7KIQ5RMOARWVTXNCKG.IDZ]\n
 [TRC] 150.0.0.71:6222 - rid:23 - \u003c\u003c- [RS+ $SYS.REQ.SERVER.PING.ROUTEZ]\n
 [TRC] 150.0.0.71:6222 - rid:23 - \u003c\u003c- [RS+ $NRG.P._meta_]\n
 [TRC] 150.0.0.71:6222 - rid:23 - \u003c\u003c- [RS+ $NRG.RP.S-R5F-Z6jshyaK]\n
 [DBG] Trying to connect to route on js_egeo_kv_shard1_infile_node04:6222 (150.0.0.77:6222)\n
 [DBG] 150.0.0.77:6222 - rid:24 - Route connect msg sent\n
 [INF] 150.0.0.77:6222 - rid:24 - Route connection created\n
 [DBG] 150.0.0.77:6222 - rid:24 - Registering remote route \"NBV22NKSB3UESVPFJCHQZ545AI26TOCUCLGUH4KL5RBO2GVRYJWOI5OA\"\n
 [DBG] Trying to connect to route on js_egeo_kv_shard1_infile_node03:6222 (150.0.0.75:6222)\n
 [DBG] 150.0.0.75:6222 - rid:25 - Route connect msg sent\n
 [INF] 150.0.0.75:6222 - rid:25 - Route connection created\n
 [TRC] 150.0.0.71:6222 - rid:23 - \u003c\u003c- [RMSG $NRG.AE._meta_ $NRG.R.fYtaPy9U 42]\n
 [TRC] 150.0.0.71:6222 - rid:23 - \u003c\u003c- MSG_PAYLOAD: [\"dGVGKtkV\\x03\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x12\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x03\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x12\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\"]\n
 [DBG] 150.0.0.75:6222 - rid:25 - Registering remote route \"NC5YXRJT4EBEHLOEFBDCDEBHKZI5ILHKT7OORNVDUJJUYBTIXM4SSKVS\"\n
 [DBG] Trying to connect to route on js_egeo_kv_shard1_infile_node02:6222 (150.0.0.71:6222)\n
 [DBG] 150.0.0.71:6222 - rid:26 - Route connect msg sent\n
 [INF] 150.0.0.71:6222 - rid:26 - Route connection created\n
 [DBG] 150.0.0.71:6222 - rid:26 - Registering remote route \"NBKYG4TJEC5AK277OUOJO6RYHDR7Q6FJWOUEWG7KIQ5RMOARWVTXNCKG\"\n
 [DBG] Trying to connect to route on js_egeo_kv_shard1_infile_node04:6222 (150.0.0.77:6222)\n
 [DBG] 150.0.0.77:6222 - rid:27 - Route connect msg sent\n
 [INF] 150.0.0.77:6222 - rid:27 - Route connection created\n
 [DBG] 150.0.0.77:6222 - rid:27 - Registering remote route \"NBV22NKSB3UESVPFJCHQZ545AI26TOCUCLGUH4KL5RBO2GVRYJWOI5OA\"\n
 [TRC] 150.0.0.77:6222 - rid:27 - \u003c\u003c- [RS+ $G _INBOX.m870eY6IGv5HVMp82Hbrx4.*]\n
 [TRC] 150.0.0.77:6222 - rid:27 - \u003c\u003c- [RS+ $G _INBOX.U5FeSqv252c2Q7uIKgmlGS.*]\n
 [TRC] 150.0.0.77:6222 - rid:27 - \u003c\u003c- [RS+ $G _INBOX.DYC7nbd1veTuK84quXOOo3.*]\n
 [TRC] 150.0.0.77:6222 - rid:27 - \u003c\u003c- [RS+ $G _INBOX.QBIlt4TE6Zu1Hn68yqt5RV.*]\n
 [TRC] 150.0.0.77:6222 - rid:27 - \u003c\u003c- [RS+ $G _INBOX.n1Xx83V6fOl3QqBCZ2gdoa.*]\n
 [TRC] 150.0.0.77:6222 - rid:27 - \u003c\u003c- [RS+ $G _INBOX.qLwbVdSsvUeFJwx6L7lFnV.*]\n
 [TRC] 150.0.0.77:6222 - rid:27 - \u003c\u003c- [RS+ $G $JS.API.CONSUMER.MSG.NEXT.KV_wh-px-egeo-e-shard1-kv.WhEgeoKvCronToEgeoCache1_consumer]\n
 [TRC] 150.0.0.77:6222 - rid:27 - \u003c\u003c- [RS+ $G _INBOX.TQmIjxKBV0xHQ2PqwO51Wk.*]\n
 [TRC] 150.0.0.77:6222 - rid:27 - \u003c\u003c- [RS+ $G _INBOX.uld06GFkewgrfX9UgSxE1e.*]\n
 [TRC] 150.0.0.77:6222 - rid:27 - \u003c\u003c- [RS+ $G _INBOX.ozBM87x4KfMriMyFIrfuP3.*]\n
 [TRC] 150.0.0.77:6222 - rid:27 - \u003c\u003c- [RS+ $G _INBOX.kRuYiAoIF5Ho784MshmHkf.*]\n
 [TRC] 150.0.0.77:6222 - rid:27 - \u003c\u003c- [RS+ $G _INBOX.DjE0780eWGv3PuoYG5CCvW.*]\n
 [TRC] 150.0.0.77:6222 - rid:27 - \u003c\u003c- [RS+ $G _INBOX.s2ALNx052m8pt2XaST8Ra4.*]\n
 [TRC] 150.0.0.77:6222 - rid:27 - \u003c\u003c- [RS+ $G _INBOX.YusNb5thZsxTeR4aMYT3iO.*]\n
 [TRC] 150.0.0.77:6222 - rid:27 - \u003c\u003c- [RS+ $G _INBOX.G1nGjp47aCP2VoAbB3sAOb.*]\n
 [TRC] 150.0.0.77:6222 - rid:27 - \u003c\u003c- [RS+ $G _INBOX.cDfBtls4IGGEN5qvrXadrT.*]\n
 [TRC] 150.0.0.77:6222 - rid:27 - \u003c\u003c- [RS+ $G $JS.ACK.KV_wh-px-egeo-e-shard1-kv.WhEgeoKvCronToEgeoCache1_consumer.*.*.*.*.*]\n
 [TRC] 150.0.0.77:6222 - rid:27 - \u003c\u003c- [RS+ $G _INBOX.1K44MM2bNQ7b5VW1cqMvE2.*]\n
 [DBG] Trying to connect to route on js_egeo_kv_shard1_infile_node03:6222 (150.0.0.75:6222)\n
 [DBG] 150.0.0.75:6222 - rid:28 - Route connect msg sent\n
 [INF] 150.0.0.75:6222 - rid:28 - Route connection created\n
 [WRN] Waiting for routing to be established...\n
 [DBG] RAFT [aZPpTWMr - _meta_] AppendEntry updating leader to \"dGVGKtkV\"\n
 [DBG] Recovered JetStream cluster metadata\n
 [DBG] JetStream cluster checking for orphans\n
 [INF] JetStream cluster new metadata leader: js_egeo_kv_shard1_infile_node02/js_egeo_kv_shard1_infile\n
 [TRC] 150.0.0.71:6222 - rid:23 - -\u003e\u003e [RMSG $NRG.R.fYtaPy9U 25]\n
 [DBG] 150.0.0.75:6222 - rid:28 - Registering remote route \"NC5YXRJT4EBEHLOEFBDCDEBHKZI5ILHKT7OORNVDUJJUYBTIXM4SSKVS\"\n
 [DBG] RAFT [aZPpTWMr - S-R5F-Z6jshyaK] Installing snapshot of 309781 bytes\n
 [DBG] Trying to connect to route on js_egeo_kv_shard1_infile_node04:6222 (150.0.0.77:6222)\n
 [DBG] 150.0.0.77:6222 - rid:29 - Route connect msg sent\n
 [INF] 150.0.0.77:6222 - rid:29 - Route connection created\n
 [DBG] 150.0.0.77:6222 - rid:29 - Registering remote route \"NBV22NKSB3UESVPFJCHQZ545AI26TOCUCLGUH4KL5RBO2GVRYJWOI5OA\"\n
 [DBG] Trying to connect to route on js_egeo_kv_shard1_infile_node02:6222 (150.0.0.71:6222)\n
 [DBG] Trying to connect to route on js_egeo_kv_shard1_infile_node04:6222 (150.0.0.77:6222)\n
 [DBG] 150.0.0.77:6222 - rid:30 - Route connect msg sent\n
 [INF] 150.0.0.77:6222 - rid:30 - Route connection created\n
 [DBG] 150.0.0.71:6222 - rid:31 - Route connect msg sent\n
 [INF] 150.0.0.71:6222 - rid:31 - Route connection created\n
 [DBG] 150.0.0.71:6222 - rid:31 - Registering remote route \"NBKYG4TJEC5AK277OUOJO6RYHDR7Q6FJWOUEWG7KIQ5RMOARWVTXNCKG\"\n
 [DBG] 150.0.0.77:6222 - rid:30 - Registering remote route \"NBV22NKSB3UESVPFJCHQZ545AI26TOCUCLGUH4KL5RBO2GVRYJWOI5OA\"\n
 [TRC] 150.0.0.71:6222 - rid:31 - \u003c\u003c- [RS+ $G _INBOX.AXcJU2al4Z1YBsWYQIE4q9.*]\n
 [TRC] 150.0.0.71:6222 - rid:31 - \u003c\u003c- [RS+ $G _INBOX.Dm12hJqm7hBMllwOB484yr.*]\n
 [TRC] 150.0.0.71:6222 - rid:31 - \u003c\u003c- [RS+ $G _INBOX.mCeb43377bgbqssMNpFody.*]\n
 [TRC] 150.0.0.71:6222 - rid:31 - \u003c\u003c- [RS+ $G $KV.wh-px-egeo-e-shard1-kv.\u003e]\n
 [TRC] 150.0.0.71:6222 - rid:31 - \u003c\u003c- [RS+ $G _INBOX.RdleT3nWctSn7NoF0ckHVJ.*]\n
 [TRC] 150.0.0.71:6222 - rid:31 - \u003c\u003c- [RS+ $G _INBOX.57rsy1Xmyp5ePooaJuTryi.*]\n
 [TRC] 150.0.0.71:6222 - rid:31 - \u003c\u003c- [RS+ $G _INBOX.j1WAb65g0U3LAMaGP2tPdT.*]\n
 [TRC] 150.0.0.71:6222 - rid:31 - \u003c\u003c- [RS+ $G _INBOX.S1cEJe0P4uW22XVt9dKfQJ.*]\n
 [TRC] 150.0.0.71:6222 - rid:31 - \u003c\u003c- [RS+ $G _INBOX.R7jDIUUEhqhd3IAtd2NJmf.*]\n
 [TRC] 150.0.0.71:6222 - rid:31 - \u003c\u003c- [RS+ $G _INBOX.mIv6gw7005U3Z8VRVOv3vO.*]\n
 [TRC] 150.0.0.71:6222 - rid:31 - \u003c\u003c- [RS+ $G _INBOX.ZO0b7bwiKbeYJ00IUlifth.*]\n
 [TRC] 150.0.0.71:6222 - rid:31 - \u003c\u003c- [RS+ $G _INBOX.uYuWUGHqEoSgmNrdaBoWKm.*]\n
 [TRC] 150.0.0.71:6222 - rid:31 - \u003c\u003c- [RS+ $G _INBOX.D2bZymvymlErPMT3nI4WXs.*]\n
 [TRC] 150.0.0.71:6222 - rid:31 - \u003c\u003c- [RS+ $G _INBOX.D2bZymvymlErPMT3nI4WYy.*]\n
 [TRC] 150.0.0.71:6222 - rid:31 - \u003c\u003c- [RS+ $G _INBOX.58dGi9ubFmmsIPgqusoXXf.*]\n
 [TRC] 150.0.0.71:6222 - rid:31 - \u003c\u003c- [RS+ $G _INBOX.9F9sL0hHA0cxeYk4W6hfu5.*]\n
 [TRC] 150.0.0.71:6222 - rid:31 - \u003c\u003c- [RS+ $G _INBOX.rte5fJ2PfuJKOvwZxyxetf.*]\n
 [TRC] 150.0.0.71:6222 - rid:31 - \u003c\u003c- [RS+ $G _INBOX.5630Y5hUs09mHWyNTZxXno.*]\n
 [TRC] 150.0.0.71:6222 - rid:31 - \u003c\u003c- [RS+ $G _INBOX.Wk4Lz03SMsuQh2A6hMTvYx.*]\n
 [DBG] Trying to connect to route on js_egeo_kv_shard1_infile_node03:6222 (150.0.0.75:6222)\n
 [DBG] 150.0.0.75:6222 - rid:32 - Route connect msg sent\n
 [INF] 150.0.0.75:6222 - rid:32 - Route connection created\n
 [DBG] 150.0.0.75:6222 - rid:32 - Registering remote route \"NC5YXRJT4EBEHLOEFBDCDEBHKZI5ILHKT7OORNVDUJJUYBTIXM4SSKVS\"\n
 [DBG] Trying to connect to route on js_egeo_kv_shard1_infile_node02:6222 (150.0.0.71:6222)\n
 [DBG] 150.0.0.71:6222 - rid:33 - Route connect msg sent\n
 [INF] 150.0.0.71:6222 - rid:33 - Route connection created\n
 [DBG] 150.0.0.71:6222 - rid:33 - Registering remote route \"NBKYG4TJEC5AK277OUOJO6RYHDR7Q6FJWOUEWG7KIQ5RMOARWVTXNCKG\"\n
 [DBG] Trying to connect to route on js_egeo_kv_shard1_infile_node04:6222 (150.0.0.77:6222)\n
 [DBG] 150.0.0.77:6222 - rid:34 - Route connect msg sent\n
 [INF] 150.0.0.77:6222 - rid:34 - Route connection created\n
 [DBG] 150.0.0.77:6222 - rid:34 - Registering remote route \"NBV22NKSB3UESVPFJCHQZ545AI26TOCUCLGUH4KL5RBO2GVRYJWOI5OA\"\n
 [TRC] 150.0.0.77:6222 - rid:17 - -\u003e\u003e [RMSG $SYS.SERVER.NAVQASAKCWEM567JJSFHP4DTY2GAHDDQYOMOTPM4MOIYIJIV673YFNC7.STATSZ 2993]\n
 [TRC] 150.0.0.75:6222 - rid:19 - -\u003e\u003e [RMSG $SYS.SERVER.NAVQASAKCWEM567JJSFHP4DTY2GAHDDQYOMOTPM4MOIYIJIV673YFNC7.STATSZ 2993]\n
 [TRC] 150.0.0.71:6222 - rid:23 - -\u003e\u003e [RMSG $SYS.SERVER.NAVQASAKCWEM567JJSFHP4DTY2GAHDDQYOMOTPM4MOIYIJIV673YFNC7.STATSZ 2993]\n
 [TRC] 150.0.0.75:6222 - rid:19 - \u003c\u003c- [RMSG $SYS.SERVER.NC5YXRJT4EBEHLOEFBDCDEBHKZI5ILHKT7OORNVDUJJUYBTIXM4SSKVS.STATSZ 4014]\n
 [TRC] 150.0.0.75:6222 - rid:19 - \u003c\u003c- MSG_PAYLOAD: [\"{\\\"server\\\":{\\\"name\\\":\\\"js_egeo_kv_shard1_infile_node03\\\",\\\"host\\\":\\\"0.0.0.0\\\",\\\"id\\\":\\\"NC5YXRJT4EBEHLOEFBDCDEBHKZI5ILHKT7OORNVDUJJUYBTIXM4SSKVS\\\",\\\"cluster\\\":\\\"js_egeo_kv_shard1_infile\\\",\\\"ver\\\":\\\"2.11.0-dev\\\",\\\"jetstream\\\":true,\\\"flags\\\":3,\\\"seq\\\":25,\\\"time\\\":\\\"2024-05-08T15:22:46.433026167Z\\\"},\\\"statsz\\\":{\\\"start\\\":\\\"2024-05-08T15:22:43.547667697Z\\\",\\\"mem\\\":171642880,\\\"cores\\\":5,\\\"cpu\\\":4,\\\"connections\\\":0,\\\"total_connections\\\":0,\\\"active_accounts\\\":2,\\\"subscriptions\\\":321,\\\"sent\\\":{\\\"msgs\\\":69,\\\"bytes\\\":27683},\\\"received\\\":{\\\"msgs\\\":69,\\\"bytes\\\":18979},\\\"slow_consumers\\\":0,\\\"routes\\\":[{\\\"rid\\\":58,\\\"name\\\":\\\"js_egeo_kv_shard1_infile_node01\\\",\\\"sent\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"received\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"pending\\\":0},{\\\"rid\\\":60,\\\"name\\\":\\\"js_egeo_kv_shard1_infile_node01\\\",\\\"sent\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"received\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"pending\\\":0},{\\\"rid\\\":61,\\\"name\\\":\\\"js_egeo_kv_shard1_infile_node01\\\",\\\"sent\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"received\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"pending\\\":0},{\\\"rid\\\":62,\\\"name\\\":\\\"js_egeo_kv_shard1_infile_node01\\\",\\\"sent\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"received\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"pending\\\":0},{\\\"rid\\\":19,\\\"name\\\":\\\"js_egeo_kv_shard1_infile_node04\\\",\\\"sent\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"received\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"pending\\\":0},{\\\"rid\\\":23,\\\"name\\\":\\\"js_egeo_kv_shard1_infile_node04\\\",\\\"sent\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"received\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"pending\\\":0},{\\\"rid\\\":25,\\\"name\\\":\\\"js_egeo_kv_shard1_infile_node04\\\",\\\"sent\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"received\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"pending\\\":0},{\\\"rid\\\":27,\\\"name\\\":\\\"js_egeo_kv_shard1_infile_node04\\\",\\\"sent\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"received\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"pending\\\":0},{\\\"rid\\\":28,\\\"name\\\":\\\"js_egeo_kv_shard1_infile_node04\\\",\\\"sent\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"received\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"pending\\\":0},{\\\"rid\\\":29,\\\"name\\\":\\\"js_egeo_kv_shard1_infile_node04\\\",\\\"sent\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"received\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"pending\\\":0},{\\\"rid\\\":31,\\\"name\\\":\\\"js_egeo_kv_shard1_infile_node04\\\",\\\"sent\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"received\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"pending\\\":0},{\\\"rid\\\":32,\\\"name\\\":\\\"js_egeo_kv_shard1_infile_node04\\\",\\\"sent\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"received\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"pending\\\":0},{\\\"rid\\\":33,\\\"name\\\":\\\"js_egeo_kv_shard1_infile_node04\\\",\\\"sent\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"received\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"pending\\\":0},{\\\"rid\\\":21,\\\"name\\\":\\\"js_egeo_kv_shard1_infile_node02\\\",\\\"sent\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"received\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"pending\\\":0},{\\\"rid\\\":22,\\\"name\\\":\\\"js_egeo_kv_shard1_infile_node02\\\",\\\"sent\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"received\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"pending\\\":0},{\\\"rid\\\":24,\\\"name\\\":\\\"js_egeo_kv_shard1_infile_node02\\\",\\\"sent\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"received\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"pending\\\":0},{\\\"rid\\\":26,\\\"name\\\":\\\"js_egeo_kv_shard1_infile_node02\\\",\\\"sent\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"received\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"pending\\\":0},{\\\"rid\\\":30,\\\"name\\\":\\\"js_egeo_kv_shard1_infile_node02\\\",\\\"sent\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"received\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"pending\\\":0},{\\\"rid\\\":34,\\\"name\\\":\\\"js_egeo_kv_shard1_infile_node02\\\",\\\"sent\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"received\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"pending\\\":0},{\\\"rid\\\":35,\\\"name\\\":\\\"js_egeo_kv_shard1_infile_node02\\\",\\\"sent\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"received\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"pending\\\":0},{\\\"rid\\\":36,\\\"name\\\":\\\"js_egeo_kv_shard1_infile_node02\\\",\\\"sent\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"received\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"pending\\\":0},{\\\"rid\\\":38,\\\"name\\\":\\\"js_egeo_kv_shard1_infile_node02\\\",\\\"sent\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"received\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"pending\\\":0},{\\\"rid\\\":59,\\\"name\\\":\\\"js_egeo_kv_shard1_infile_node01\\\",\\\"sent\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"received\\\":{\\\"msgs\\\":1,\\\"bytes\\\":2993},\\\"pending\\\":0},{\\\"rid\\\":20,\\\"name\\\":\\\"js_egeo_kv_shard1_infile_node04\\\",\\\"sent\\\":{\\\"msgs\\\":43,\\\"bytes\\\":14054},\\\"received\\\":{\\\"msgs\\\":42,\\\"bytes\\\":7086},\\\"pending\\\":0},{\\\"rid\\\":18,\\\"name\\\":\\\"js_egeo_kv_shard1_infile_node02\\\",\\\"sent\\\":{\\\"msgs\\\":26,\\\"bytes\\\":13629},\\\"received\\\":{\\\"msgs\\\":26,\\\"bytes\\\":8900},\\\"pending\\\":0}],\\\"active_servers\\\":4,\\\"jetstream\\\":{\\\"config\\\":{\\\"max_memory\\\":3145728000,\\\"max_storage\\\":10737418240,\\\"sync_interval\\\":120000000000},\\\"stats\\\":{\\\"memory\\\":0,\\\"storage\\\":20263801,\\\"reserved_memory\\\":0,\\\"reserved_storage\\\":0,\\\"accounts\\\":1,\\\"ha_assets\\\":3,\\\"api\\\":{\\\"total\\\":0,\\\"errors\\\":0}},\\\"meta\\\":{\\\"name\\\":\\\"_meta_\\\",\\\"leader\\\":\\\"js_egeo_kv_shard1_infile_node02\\\",\\\"peer\\\":\\\"dGVGKtkV\\\",\\\"cluster_size\\\":5}}}}\"]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- [RMSG $SYS.SERVER.NBV22NKSB3UESVPFJCHQZ545AI26TOCUCLGUH4KL5RBO2GVRYJWOI5OA.STATSZ 4336]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- MSG_PAYLOAD: [\"{\\\"server\\\":{\\\"name\\\":\\\"js_egeo_kv_shard1_infile_node04\\\",\\\"host\\\":\\\"0.0.0.0\\\",\\\"id\\\":\\\"NBV22NKSB3UESVPFJCHQZ545AI26TOCUCLGUH4KL5RBO2GVRYJWOI5OA\\\",\\\"cluster\\\":\\\"js_egeo_kv_shard1_infile\\\",\\\"ver\\\":\\\"2.11.0-dev\\\",\\\"jetstream\\\":true,\\\"flags\\\":3,\\\"seq\\\":150,\\\"time\\\":\\\"2024-05-08T15:22:46.433436722Z\\\"},\\\"statsz\\\":{\\\"start\\\":\\\"2024-05-08T14:26:10.128220292Z\\\",\\\"mem\\\":525848576,\\\"cores\\\":5,\\\"cpu\\\":4,\\\"connections\\\":16,\\\"total_connections\\\":16,\\\"active_accounts\\\":2,\\\"subscriptions\\\":321,\\\"sent\\\":{\\\"msgs\\\":642888,\\\"bytes\\\":127187803},\\\"received\\\":{\\\"msgs\\\":598219,\\\"bytes\\\":140598025},\\\"slow_consumers\\\":0,\\\"routes\\\":[{\\\"rid\\\":126,\\\"name\\\":\\\"js_egeo_kv_shard1_infile_node03\\\",\\\"sent\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"received\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"pending\\\":0},{\\\"rid\\\":128,\\\"name\\\":\\\"js_egeo_kv_shard1_infile_node03\\\",\\\"sent\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"received\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"pending\\\":0},{\\\"rid\\\":129,\\\"name\\\":\\\"js_egeo_kv_shard1_infile_node03\\\",\\\"sent\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"received\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"pending\\\":0},{\\\"rid\\\":130,\\\"name\\\":\\\"js_egeo_kv_shard1_infile_node03\\\",\\\"sent\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"received\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"pending\\\":0},{\\\"rid\\\":131,\\\"name\\\":\\\"js_egeo_kv_shard1_infile_node03\\\",\\\"sent\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"received\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"pending\\\":0},{\\\"rid\\\":132,\\\"name\\\":\\\"js_egeo_kv_shard1_infile_node03\\\",\\\"sent\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"received\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"pending\\\":0},{\\\"rid\\\":133,\\\"name\\\":\\\"js_egeo_kv_shard1_infile_node03\\\",\\\"sent\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"received\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"pending\\\":0},{\\\"rid\\\":134,\\\"name\\\":\\\"js_egeo_kv_shard1_infile_node03\\\",\\\"sent\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"received\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"pending\\\":0},{\\\"rid\\\":135,\\\"name\\\":\\\"js_egeo_kv_shard1_infile_node03\\\",\\\"sent\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"received\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"pending\\\":0},{\\\"rid\\\":147,\\\"name\\\":\\\"js_egeo_kv_shard1_infile_node01\\\",\\\"sent\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"received\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"pending\\\":0},{\\\"rid\\\":148,\\\"name\\\":\\\"js_egeo_kv_shard1_infile_node01\\\",\\\"sent\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"received\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"pending\\\":0},{\\\"rid\\\":149,\\\"name\\\":\\\"js_egeo_kv_shard1_infile_node01\\\",\\\"sent\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"received\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"pending\\\":0},{\\\"rid\\\":150,\\\"name\\\":\\\"js_egeo_kv_shard1_infile_node01\\\",\\\"sent\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"received\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"pending\\\":0},{\\\"rid\\\":151,\\\"name\\\":\\\"js_egeo_kv_shard1_infile_node01\\\",\\\"sent\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"received\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"pending\\\":0},{\\\"rid\\\":152,\\\"name\\\":\\\"js_egeo_kv_shard1_infile_node01\\\",\\\"sent\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"received\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"pending\\\":0},{\\\"rid\\\":15,\\\"name\\\":\\\"js_egeo_kv_shard1_infile_node02\\\",\\\"sent\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"received\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"pending\\\":0},{\\\"rid\\\":17,\\\"name\\\":\\\"js_egeo_kv_shard1_infile_node02\\\",\\\"sent\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"received\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"pending\\\":0},{\\\"rid\\\":21,\\\"name\\\":\\\"js_egeo_kv_shard1_infile_node02\\\",\\\"sent\\\":{\\\"msgs\\\":49905,\\\"bytes\\\":12489290},\\\"received\\\":{\\\"msgs\\\":49905,\\\"bytes\\\":2677602},\\\"pending\\\":0},{\\\"rid\\\":24,\\\"name\\\":\\\"js_egeo_kv_shard1_infile_node02\\\",\\\"sent\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"received\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"pending\\\":0},{\\\"rid\\\":31,\\\"name\\\":\\\"js_egeo_kv_shard1_infile_node02\\\",\\\"sent\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"received\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"pending\\\":0},{\\\"rid\\\":34,\\\"name\\\":\\\"js_egeo_kv_shard1_infile_node02\\\",\\\"sent\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"received\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"pending\\\":0},{\\\"rid\\\":41,\\\"name\\\":\\\"js_egeo_kv_shard1_infile_node02\\\",\\\"sent\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"received\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"pending\\\":0},{\\\"rid\\\":42,\\\"name\\\":\\\"js_egeo_kv_shard1_infile_node02\\\",\\\"sent\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"received\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"pending\\\":0},{\\\"rid\\\":45,\\\"name\\\":\\\"js_egeo_kv_shard1_infile_node02\\\",\\\"sent\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"received\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"pending\\\":0},{\\\"rid\\\":146,\\\"name\\\":\\\"js_egeo_kv_shard1_infile_node01\\\",\\\"sent\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"received\\\":{\\\"msgs\\\":1,\\\"bytes\\\":2993},\\\"pending\\\":0},{\\\"rid\\\":16,\\\"name\\\":\\\"js_egeo_kv_shard1_infile_node02\\\",\\\"sent\\\":{\\\"msgs\\\":203202,\\\"bytes\\\":25859484},\\\"received\\\":{\\\"msgs\\\":204390,\\\"bytes\\\":66214565},\\\"pending\\\":0},{\\\"rid\\\":127,\\\"name\\\":\\\"js_egeo_kv_shard1_infile_node03\\\",\\\"sent\\\":{\\\"msgs\\\":42,\\\"bytes\\\":7086},\\\"received\\\":{\\\"msgs\\\":43,\\\"bytes\\\":14054},\\\"pending\\\":0}],\\\"active_servers\\\":4,\\\"jetstream\\\":{\\\"config\\\":{\\\"max_memory\\\":3145728000,\\\"max_storage\\\":10737418240,\\\"sync_interval\\\":120000000000},\\\"stats\\\":{\\\"memory\\\":0,\\\"storage\\\":20263801,\\\"reserved_memory\\\":0,\\\"reserved_storage\\\":0,\\\"accounts\\\":1,\\\"ha_assets\\\":3,\\\"api\\\":{\\\"total\\\":317,\\\"errors\\\":0}},\\\"meta\\\":{\\\"name\\\":\\\"_meta_\\\",\\\"leader\\\":\\\"js_egeo_kv_shard1_infile_node02\\\",\\\"peer\\\":\\\"dGVGKtkV\\\",\\\"cluster_size\\\":5}}}}\"]\n
 [TRC] 150.0.0.71:6222 - rid:23 - \u003c\u003c- [RMSG $SYS.SERVER.NBKYG4TJEC5AK277OUOJO6RYHDR7Q6FJWOUEWG7KIQ5RMOARWVTXNCKG.STATSZ 4520]\n
 [TRC] 150.0.0.77:6222 - rid:17 - -\u003e\u003e [RMSG $SYS.SERVER.NAVQASAKCWEM567JJSFHP4DTY2GAHDDQYOMOTPM4MOIYIJIV673YFNC7.STATSZ 3011]\n
 [TRC] 150.0.0.75:6222 - rid:19 - -\u003e\u003e [RMSG $SYS.SERVER.NAVQASAKCWEM567JJSFHP4DTY2GAHDDQYOMOTPM4MOIYIJIV673YFNC7.STATSZ 3011]\n
 [TRC] 150.0.0.71:6222 - rid:23 - -\u003e\u003e [RMSG $SYS.SERVER.NAVQASAKCWEM567JJSFHP4DTY2GAHDDQYOMOTPM4MOIYIJIV673YFNC7.STATSZ 3011]\n
 [TRC] 150.0.0.77:6222 - rid:17 - -\u003e\u003e [RMSG $SYS.SERVER.NAVQASAKCWEM567JJSFHP4DTY2GAHDDQYOMOTPM4MOIYIJIV673YFNC7.STATSZ 3014]\n
 [TRC] 150.0.0.75:6222 - rid:19 - -\u003e\u003e [RMSG $SYS.SERVER.NAVQASAKCWEM567JJSFHP4DTY2GAHDDQYOMOTPM4MOIYIJIV673YFNC7.STATSZ 3014]\n
 [TRC] 150.0.0.71:6222 - rid:23 - \u003c\u003c- MSG_PAYLOAD: [\"{\\\"server\\\":{\\\"name\\\":\\\"js_egeo_kv_shard1_infile_node02\\\",\\\"host\\\":\\\"0.0.0.0\\\",\\\"id\\\":\\\"NBKYG4TJEC5AK277OUOJO6RYHDR7Q6FJWOUEWG7KIQ5RMOARWVTXNCKG\\\",\\\"cluster\\\":\\\"js_egeo_kv_shard1_infile\\\",\\\"ver\\\":\\\"2.11.0-dev\\\",\\\"jetstream\\\":true,\\\"flags\\\":3,\\\"seq\\\":146,\\\"time\\\":\\\"2024-05-08T15:22:46.433974755Z\\\"},\\\"statsz\\\":{\\\"start\\\":\\\"2024-05-08T14:25:48.930375731Z\\\",\\\"mem\\\":540131328,\\\"cores\\\":5,\\\"cpu\\\":3,\\\"connections\\\":17,\\\"total_connections\\\":20,\\\"active_accounts\\\":2,\\\"subscriptions\\\":321,\\\"sent\\\":{\\\"msgs\\\":987645,\\\"bytes\\\":253396495},\\\"received\\\":{\\\"msgs\\\":1005649,\\\"bytes\\\":125048855},\\\"slow_consumers\\\":0,\\\"routes\\\":[{\\\"rid\\\":129,\\\"name\\\":\\\"js_egeo_kv_shard1_infile_node03\\\",\\\"sent\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"received\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"pending\\\":0},{\\\"rid\\\":130,\\\"name\\\":\\\"js_egeo_kv_shard1_infile_node03\\\",\\\"sent\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"received\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"pending\\\":0},{\\\"rid\\\":131,\\\"name\\\":\\\"js_egeo_kv_shard1_infile_node03\\\",\\\"sent\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"received\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"pending\\\":0},{\\\"rid\\\":132,\\\"name\\\":\\\"js_egeo_kv_shard1_infile_node03\\\",\\\"sent\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"received\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"pending\\\":0},{\\\"rid\\\":133,\\\"name\\\":\\\"js_egeo_kv_shard1_infile_node03\\\",\\\"sent\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"received\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"pending\\\":0},{\\\"rid\\\":134,\\\"name\\\":\\\"js_egeo_kv_shard1_infile_node03\\\",\\\"sent\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"received\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"pending\\\":0},{\\\"rid\\\":135,\\\"name\\\":\\\"js_egeo_kv_shard1_infile_node03\\\",\\\"sent\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"received\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"pending\\\":0},{\\\"rid\\\":136,\\\"name\\\":\\\"js_egeo_kv_shard1_infile_node03\\\",\\\"sent\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"received\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"pending\\\":0},{\\\"rid\\\":137,\\\"name\\\":\\\"js_egeo_kv_shard1_infile_node03\\\",\\\"sent\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"received\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"pending\\\":0},{\\\"rid\\\":148,\\\"name\\\":\\\"js_egeo_kv_shard1_infile_node01\\\",\\\"sent\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"received\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"pending\\\":0},{\\\"rid\\\":150,\\\"name\\\":\\\"js_egeo_kv_shard1_infile_node01\\\",\\\"sent\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"received\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"pending\\\":0},{\\\"rid\\\":151,\\\"name\\\":\\\"js_egeo_kv_shard1_infile_node01\\\",\\\"sent\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"received\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"pending\\\":0},{\\\"rid\\\":152,\\\"name\\\":\\\"js_egeo_kv_shard1_infile_node01\\\",\\\"sent\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"received\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"pending\\\":0},{\\\"rid\\\":39,\\\"name\\\":\\\"js_egeo_kv_shard1_infile_node04\\\",\\\"sent\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"received\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"pending\\\":0},{\\\"rid\\\":40,\\\"name\\\":\\\"js_egeo_kv_shard1_infile_node04\\\",\\\"sent\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"received\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"pending\\\":0},{\\\"rid\\\":41,\\\"name\\\":\\\"js_egeo_kv_shard1_infile_node04\\\",\\\"sent\\\":{\\\"msgs\\\":49905,\\\"bytes\\\":2677602},\\\"received\\\":{\\\"msgs\\\":49905,\\\"bytes\\\":12489290},\\\"pending\\\":0},{\\\"rid\\\":42,\\\"name\\\":\\\"js_egeo_kv_shard1_infile_node04\\\",\\\"sent\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"received\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"pending\\\":0},{\\\"rid\\\":43,\\\"name\\\":\\\"js_egeo_kv_shard1_infile_node04\\\",\\\"sent\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"received\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"pending\\\":0},{\\\"rid\\\":44,\\\"name\\\":\\\"js_egeo_kv_shard1_infile_node04\\\",\\\"sent\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"received\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"pending\\\":0},{\\\"rid\\\":45,\\\"name\\\":\\\"js_egeo_kv_shard1_infile_node04\\\",\\\"sent\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"received\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"pending\\\":0},{\\\"rid\\\":46,\\\"name\\\":\\\"js_egeo_kv_shard1_infile_node04\\\",\\\"sent\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"received\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"pending\\\":0},{\\\"rid\\\":47,\\\"name\\\":\\\"js_egeo_kv_shard1_infile_node04\\\",\\\"sent\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"received\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"pending\\\":0},{\\\"rid\\\":149,\\\"name\\\":\\\"js_egeo_kv_shard1_infile_node01\\\",\\\"sent\\\":{\\\"msgs\\\":1,\\\"bytes\\\":42},\\\"received\\\":{\\\"msgs\\\":2,\\\"bytes\\\":3018},\\\"pending\\\":0},{\\\"rid\\\":38,\\\"name\\\":\\\"js_egeo_kv_shard1_infile_node04\\\",\\\"sent\\\":{\\\"msgs\\\":204390,\\\"bytes\\\":66214565},\\\"received\\\":{\\\"msgs\\\":203202,\\\"bytes\\\":25859484},\\\"pending\\\":0},{\\\"rid\\\":128,\\\"name\\\":\\\"js_egeo_kv_shard1_infile_node03\\\",\\\"sent\\\":{\\\"msgs\\\":26,\\\"bytes\\\":8900},\\\"received\\\":{\\\"msgs\\\":26,\\\"bytes\\\":13629},\\\"pending\\\":0}],\\\"active_servers\\\":4,\\\"jetstream\\\":{\\\"config\\\":{\\\"max_memory\\\":3145728000,\\\"max_storage\\\":10737418240,\\\"sync_interval\\\":120000000000},\\\"stats\\\":{\\\"memory\\\":0,\\\"storage\\\":20263801,\\\"reserved_memory\\\":0,\\\"reserved_storage\\\":0,\\\"accounts\\\":1,\\\"ha_assets\\\":3,\\\"api\\\":{\\\"total\\\":81608,\\\"errors\\\":81547}},\\\"meta\\\":{\\\"name\\\":\\\"js_egeo_kv_shard1_infile\\\",\\\"leader\\\":\\\"js_egeo_kv_shard1_infile_node02\\\",\\\"peer\\\":\\\"dGVGKtkV\\\",\\\"replicas\\\":[{\\\"name\\\":\\\"js_egeo_kv_shard1_infile_node01\\\",\\\"current\\\":true,\\\"active\\\":93967950,\\\"peer\\\":\\\"aZPpTWMr\\\"},{\\\"name\\\":\\\"js_egeo_kv_shard1_infile_node03\\\",\\\"current\\\":true,\\\"active\\\":148614547,\\\"peer\\\":\\\"JrN1hGZO\\\"},{\\\"name\\\":\\\"js_egeo_kv_shard1_infile_node04\\\",\\\"current\\\":true,\\\"active\\\":148571152,\\\"peer\\\":\\\"v2fh5Gis\\\"},{\\\"name\\\":\\\"js_egeo_kv_shard1_infile_node05\\\",\\\"current\\\":true,\\\"offline\\\":true,\\\"active\\\":4148344804,\\\"peer\\\":\\\"bXlHueQT\\\"}],\\\"cluster_size\\\":5}}}}\"]\n
 [TRC] 150.0.0.71:6222 - rid:23 - -\u003e\u003e [RMSG $SYS.SERVER.NAVQASAKCWEM567JJSFHP4DTY2GAHDDQYOMOTPM4MOIYIJIV673YFNC7.STATSZ 3014]\n
 [TRC] 150.0.0.77:6222 - rid:17 - -\u003e\u003e [RMSG $SYS.SERVER.NAVQASAKCWEM567JJSFHP4DTY2GAHDDQYOMOTPM4MOIYIJIV673YFNC7.STATSZ 3022]\n
 [TRC] 150.0.0.75:6222 - rid:19 - -\u003e\u003e [RMSG $SYS.SERVER.NAVQASAKCWEM567JJSFHP4DTY2GAHDDQYOMOTPM4MOIYIJIV673YFNC7.STATSZ 3022]\n
 [TRC] 150.0.0.71:6222 - rid:23 - -\u003e\u003e [RMSG $SYS.SERVER.NAVQASAKCWEM567JJSFHP4DTY2GAHDDQYOMOTPM4MOIYIJIV673YFNC7.STATSZ 3022]\n
 [DBG] Trying to connect to route on js_egeo_kv_shard1_infile_node03:6222 (150.0.0.75:6222)\n
 [DBG] 150.0.0.75:6222 - rid:35 - Route connect msg sent\n
 [INF] 150.0.0.75:6222 - rid:35 - Route connection created\n
 [DBG] 150.0.0.75:6222 - rid:35 - Registering remote route \"NC5YXRJT4EBEHLOEFBDCDEBHKZI5ILHKT7OORNVDUJJUYBTIXM4SSKVS\"\n
 [DBG] Trying to connect to route on js_egeo_kv_shard1_infile_node02:6222 (150.0.0.71:6222)\n
 [DBG] 150.0.0.71:6222 - rid:36 - Route connect msg sent\n
 [INF] 150.0.0.71:6222 - rid:36 - Route connection created\n
 [DBG] 150.0.0.71:6222 - rid:36 - Registering remote route \"NBKYG4TJEC5AK277OUOJO6RYHDR7Q6FJWOUEWG7KIQ5RMOARWVTXNCKG\"\n
 [DBG] Trying to connect to route on js_egeo_kv_shard1_infile_node02:6222 (150.0.0.71:6222)\n
 [DBG] Trying to connect to route on js_egeo_kv_shard1_infile_node03:6222 (150.0.0.75:6222)\n
 [DBG] 150.0.0.75:6222 - rid:37 - Route connect msg sent\n
 [INF] 150.0.0.75:6222 - rid:37 - Route connection created\n
 [DBG] 150.0.0.71:6222 - rid:38 - Route connect msg sent\n
 [INF] 150.0.0.71:6222 - rid:38 - Route connection created\n
 [DBG] 150.0.0.75:6222 - rid:37 - Registering remote route \"NC5YXRJT4EBEHLOEFBDCDEBHKZI5ILHKT7OORNVDUJJUYBTIXM4SSKVS\"\n
 [DBG] 150.0.0.71:6222 - rid:38 - Registering remote route \"NBKYG4TJEC5AK277OUOJO6RYHDR7Q6FJWOUEWG7KIQ5RMOARWVTXNCKG\"\n
 [DBG] Trying to connect to route on js_egeo_kv_shard1_infile_node04:6222 (150.0.0.77:6222)\n
 [DBG] 150.0.0.77:6222 - rid:39 - Route connect msg sent\n
 [INF] 150.0.0.77:6222 - rid:39 - Route connection created\n
 [DBG] 150.0.0.77:6222 - rid:39 - Registering remote route \"NBV22NKSB3UESVPFJCHQZ545AI26TOCUCLGUH4KL5RBO2GVRYJWOI5OA\"\n
 [DBG] Trying to connect to route on js_egeo_kv_shard1_infile_node02:6222 (150.0.0.71:6222)\n
 [DBG] 150.0.0.71:6222 - rid:40 - Route connect msg sent\n
 [INF] 150.0.0.71:6222 - rid:40 - Route connection created\n
 [DBG] 150.0.0.71:6222 - rid:40 - Registering remote route \"NBKYG4TJEC5AK277OUOJO6RYHDR7Q6FJWOUEWG7KIQ5RMOARWVTXNCKG\"\n
 [DBG] Trying to connect to route on js_egeo_kv_shard1_infile_node03:6222 (150.0.0.75:6222)\n
 [DBG] 150.0.0.75:6222 - rid:41 - Route connect msg sent\n
 [INF] 150.0.0.75:6222 - rid:41 - Route connection created\n
 [DBG] 150.0.0.75:6222 - rid:41 - Registering remote route \"NC5YXRJT4EBEHLOEFBDCDEBHKZI5ILHKT7OORNVDUJJUYBTIXM4SSKVS\"\n
 [TRC] 150.0.0.75:6222 - rid:19 - \u003c\u003c- [RMSG $JSC.ARU.$G.JrN1hGZO 44]\n
 [TRC] 150.0.0.75:6222 - rid:19 - \u003c\u003c- MSG_PAYLOAD: [\"\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00y35\\x01\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\"]\n
 [DBG] Trying to connect to route on js_egeo_kv_shard1_infile_node02:6222 (150.0.0.71:6222)\n
 [DBG] 150.0.0.71:6222 - rid:42 - Route connect msg sent\n
 [INF] 150.0.0.71:6222 - rid:42 - Route connection created\n
 [DBG] Trying to connect to route on js_egeo_kv_shard1_infile_node04:6222 (150.0.0.77:6222)\n
 [DBG] 150.0.0.77:6222 - rid:43 - Route connect msg sent\n
 [INF] 150.0.0.77:6222 - rid:43 - Route connection created\n
 [DBG] 150.0.0.71:6222 - rid:42 - Registering remote route \"NBKYG4TJEC5AK277OUOJO6RYHDR7Q6FJWOUEWG7KIQ5RMOARWVTXNCKG\"\n
 [DBG] 150.0.0.77:6222 - rid:43 - Registering remote route \"NBV22NKSB3UESVPFJCHQZ545AI26TOCUCLGUH4KL5RBO2GVRYJWOI5OA\"\n
 [DBG] Trying to connect to route on js_egeo_kv_shard1_infile_node03:6222 (150.0.0.75:6222)\n
 [DBG] 150.0.0.75:6222 - rid:44 - Route connect msg sent\n
 [INF] 150.0.0.75:6222 - rid:44 - Route connection created\n
 [DBG] 150.0.0.75:6222 - rid:44 - Registering remote route \"NC5YXRJT4EBEHLOEFBDCDEBHKZI5ILHKT7OORNVDUJJUYBTIXM4SSKVS\"\n
 [DBG] Trying to connect to route on js_egeo_kv_shard1_infile_node03:6222 (150.0.0.75:6222)\n
 [DBG] 150.0.0.75:6222 - rid:45 - Route connect msg sent\n
 [INF] 150.0.0.75:6222 - rid:45 - Route connection created\n
 [DBG] 150.0.0.75:6222 - rid:45 - Registering remote route \"NC5YXRJT4EBEHLOEFBDCDEBHKZI5ILHKT7OORNVDUJJUYBTIXM4SSKVS\"\n
 [DBG] Trying to connect to route on js_egeo_kv_shard1_infile_node04:6222 (150.0.0.77:6222)\n
 [DBG] 150.0.0.77:6222 - rid:46 - Route connect msg sent\n
 [INF] 150.0.0.77:6222 - rid:46 - Route connection created\n
 [INF] 150.0.0.12:58640 - rid:47 - Route connection created\n
 [INF] 150.0.0.12:58636 - rid:48 - Route connection created\n
 [TRC] 150.0.0.12:58640 - rid:47 - \u003c\u003c- [CONNECT {\"echo\":true,\"verbose\":false,\"pedantic\":false,\"user\":\"js_egeo_kv_cluster03_infile_user\",\"pass\":\"[REDACTED]\",\"tls_required\":false,\"headers\":true,\"name\":\"NC5YXRJT4EBEHLOEFBDCDEBHKZI5ILHKT7OORNVDUJJUYBTIXM4SSKVS\",\"cluster\":\"js_egeo_kv_shard1_infile\",\"lnoc\":true}]\n
 [TRC] 150.0.0.12:58636 - rid:48 - \u003c\u003c- [CONNECT {\"echo\":true,\"verbose\":false,\"pedantic\":false,\"user\":\"js_egeo_kv_cluster03_infile_user\",\"pass\":\"[REDACTED]\",\"tls_required\":false,\"headers\":true,\"name\":\"NC5YXRJT4EBEHLOEFBDCDEBHKZI5ILHKT7OORNVDUJJUYBTIXM4SSKVS\",\"cluster\":\"js_egeo_kv_shard1_infile\",\"lnoc\":true}]\n
 [DBG] 150.0.0.77:6222 - rid:46 - Registering remote route \"NBV22NKSB3UESVPFJCHQZ545AI26TOCUCLGUH4KL5RBO2GVRYJWOI5OA\"\n
 [DBG] 150.0.0.12:58640 - rid:47 - Detected duplicate remote route \"NC5YXRJT4EBEHLOEFBDCDEBHKZI5ILHKT7OORNVDUJJUYBTIXM4SSKVS\"\n
 [INF] 150.0.0.12:58640 - rid:47 - Router connection closed: Duplicate Route\n
 [INF] 150.0.0.12:58636 - rid:48 - Router connection closed: Client Closed\n
 [DBG] Trying to connect to route on js_egeo_kv_shard1_infile_node02:6222 (150.0.0.71:6222)\n
 [DBG] 150.0.0.71:6222 - rid:49 - Route connect msg sent\n
 [INF] 150.0.0.71:6222 - rid:49 - Route connection created\n
 [DBG] 150.0.0.71:6222 - rid:49 - Registering remote route \"NBKYG4TJEC5AK277OUOJO6RYHDR7Q6FJWOUEWG7KIQ5RMOARWVTXNCKG\"\n
 [INF] 150.0.0.22:36108 - rid:50 - Route connection created\n
 [TRC] 150.0.0.22:36108 - rid:50 - \u003c\u003c- [CONNECT {\"echo\":true,\"verbose\":false,\"pedantic\":false,\"user\":\"js_egeo_kv_cluster03_infile_user\",\"pass\":\"[REDACTED]\",\"tls_required\":false,\"headers\":true,\"name\":\"NBV22NKSB3UESVPFJCHQZ545AI26TOCUCLGUH4KL5RBO2GVRYJWOI5OA\",\"cluster\":\"js_egeo_kv_shard1_infile\",\"lnoc\":true}]\n
 [INF] 150.0.0.22:36108 - rid:50 - Router connection closed: Client Closed\n
 [INF] 150.0.0.22:36118 - rid:51 - Route connection created\n
 [TRC] 150.0.0.22:36118 - rid:51 - \u003c\u003c- [CONNECT {\"echo\":true,\"verbose\":false,\"pedantic\":false,\"user\":\"js_egeo_kv_cluster03_infile_user\",\"pass\":\"[REDACTED]\",\"tls_required\":false,\"headers\":true,\"name\":\"NBV22NKSB3UESVPFJCHQZ545AI26TOCUCLGUH4KL5RBO2GVRYJWOI5OA\",\"cluster\":\"js_egeo_kv_shard1_infile\",\"lnoc\":true}]\n
 [INF] 150.0.0.22:36128 - rid:52 - Route connection created\n
 [TRC] 150.0.0.22:36128 - rid:52 - \u003c\u003c- [CONNECT {\"echo\":true,\"verbose\":false,\"pedantic\":false,\"user\":\"js_egeo_kv_cluster03_infile_user\",\"pass\":\"[REDACTED]\",\"tls_required\":false,\"headers\":true,\"name\":\"NBV22NKSB3UESVPFJCHQZ545AI26TOCUCLGUH4KL5RBO2GVRYJWOI5OA\",\"cluster\":\"js_egeo_kv_shard1_infile\",\"lnoc\":true}]\n
 [INF] 150.0.0.22:36118 - rid:51 - Router connection closed: Client Closed\n
 [INF] 150.0.0.22:36128 - rid:52 - Router connection closed: Client Closed\n
 [INF] 150.0.0.22:36152 - rid:53 - Route connection created\n
 [INF] 150.0.0.22:36136 - rid:54 - Route connection created\n
 [TRC] 150.0.0.22:36152 - rid:53 - \u003c\u003c- [CONNECT {\"echo\":true,\"verbose\":false,\"pedantic\":false,\"user\":\"js_egeo_kv_cluster03_infile_user\",\"pass\":\"[REDACTED]\",\"tls_required\":false,\"headers\":true,\"name\":\"NBV22NKSB3UESVPFJCHQZ545AI26TOCUCLGUH4KL5RBO2GVRYJWOI5OA\",\"cluster\":\"js_egeo_kv_shard1_infile\",\"lnoc\":true}]\n
 [TRC] 150.0.0.22:36136 - rid:54 - \u003c\u003c- [CONNECT {\"echo\":true,\"verbose\":false,\"pedantic\":false,\"user\":\"js_egeo_kv_cluster03_infile_user\",\"pass\":\"[REDACTED]\",\"tls_required\":false,\"headers\":true,\"name\":\"NBV22NKSB3UESVPFJCHQZ545AI26TOCUCLGUH4KL5RBO2GVRYJWOI5OA\",\"cluster\":\"js_egeo_kv_shard1_infile\",\"lnoc\":true}]\n
 [INF] 150.0.0.22:36152 - rid:53 - Router connection closed: Client Closed\n
 [INF] 150.0.0.22:36136 - rid:54 - Router connection closed: Client Closed\n
 [INF] 150.0.0.9:52386 - rid:55 - Route connection created\n
 [INF] 150.0.0.9:52372 - rid:56 - Route connection created\n
 [TRC] 150.0.0.9:52386 - rid:55 - \u003c\u003c- [CONNECT {\"echo\":true,\"verbose\":false,\"pedantic\":false,\"user\":\"js_egeo_kv_cluster03_infile_user\",\"pass\":\"[REDACTED]\",\"tls_required\":false,\"headers\":true,\"name\":\"NBKYG4TJEC5AK277OUOJO6RYHDR7Q6FJWOUEWG7KIQ5RMOARWVTXNCKG\",\"cluster\":\"js_egeo_kv_shard1_infile\",\"lnoc\":true}]\n
 [TRC] 150.0.0.9:52372 - rid:56 - \u003c\u003c- [CONNECT {\"echo\":true,\"verbose\":false,\"pedantic\":false,\"user\":\"js_egeo_kv_cluster03_infile_user\",\"pass\":\"[REDACTED]\",\"tls_required\":false,\"headers\":true,\"name\":\"NBKYG4TJEC5AK277OUOJO6RYHDR7Q6FJWOUEWG7KIQ5RMOARWVTXNCKG\",\"cluster\":\"js_egeo_kv_shard1_infile\",\"lnoc\":true}]\n
 [INF] 150.0.0.22:36162 - rid:57 - Route connection created\n
 [TRC] 150.0.0.22:36162 - rid:57 - \u003c\u003c- [CONNECT {\"echo\":true,\"verbose\":false,\"pedantic\":false,\"user\":\"js_egeo_kv_cluster03_infile_user\",\"pass\":\"[REDACTED]\",\"tls_required\":false,\"headers\":true,\"name\":\"NBV22NKSB3UESVPFJCHQZ545AI26TOCUCLGUH4KL5RBO2GVRYJWOI5OA\",\"cluster\":\"js_egeo_kv_shard1_infile\",\"lnoc\":true}]\n
 [INF] 150.0.0.9:52388 - rid:58 - Route connection created\n
 [TRC] 150.0.0.9:52388 - rid:58 - \u003c\u003c- [CONNECT {\"echo\":true,\"verbose\":false,\"pedantic\":false,\"user\":\"js_egeo_kv_cluster03_infile_user\",\"pass\":\"[REDACTED]\",\"tls_required\":false,\"headers\":true,\"name\":\"NBKYG4TJEC5AK277OUOJO6RYHDR7Q6FJWOUEWG7KIQ5RMOARWVTXNCKG\",\"cluster\":\"js_egeo_kv_shard1_infile\",\"lnoc\":true}]\n
 [INF] 150.0.0.9:52372 - rid:56 - Router connection closed: Client Closed\n
 [INF] 150.0.0.9:52386 - rid:55 - Router connection closed: Client Closed\n
 [INF] 150.0.0.22:36162 - rid:57 - Router connection closed: Client Closed\n
 [INF] 150.0.0.9:52388 - rid:58 - Router connection closed: Client Closed\n
 [INF] 150.0.0.22:36166 - rid:59 - Route connection created\n
 [TRC] 150.0.0.22:36166 - rid:59 - \u003c\u003c- [CONNECT {\"echo\":true,\"verbose\":false,\"pedantic\":false,\"user\":\"js_egeo_kv_cluster03_infile_user\",\"pass\":\"[REDACTED]\",\"tls_required\":false,\"headers\":true,\"name\":\"NBV22NKSB3UESVPFJCHQZ545AI26TOCUCLGUH4KL5RBO2GVRYJWOI5OA\",\"cluster\":\"js_egeo_kv_shard1_infile\",\"lnoc\":true}]\n
 [DBG] 150.0.0.22:36166 - rid:59 - Detected duplicate remote route \"NBV22NKSB3UESVPFJCHQZ545AI26TOCUCLGUH4KL5RBO2GVRYJWOI5OA\"\n
 [INF] 150.0.0.22:36166 - rid:59 - Router connection closed: Duplicate Route\n
 [INF] 150.0.0.9:52400 - rid:60 - Route connection created\n
 [TRC] 150.0.0.9:52400 - rid:60 - \u003c\u003c- [CONNECT {\"echo\":true,\"verbose\":false,\"pedantic\":false,\"user\":\"js_egeo_kv_cluster03_infile_user\",\"pass\":\"[REDACTED]\",\"tls_required\":false,\"headers\":true,\"name\":\"NBKYG4TJEC5AK277OUOJO6RYHDR7Q6FJWOUEWG7KIQ5RMOARWVTXNCKG\",\"cluster\":\"js_egeo_kv_shard1_infile\",\"lnoc\":true}]\n
 [INF] 150.0.0.22:36180 - rid:61 - Route connection created\n
 [TRC] 150.0.0.22:36180 - rid:61 - \u003c\u003c- [CONNECT {\"echo\":true,\"verbose\":false,\"pedantic\":false,\"user\":\"js_egeo_kv_cluster03_infile_user\",\"pass\":\"[REDACTED]\",\"tls_required\":false,\"headers\":true,\"name\":\"NBV22NKSB3UESVPFJCHQZ545AI26TOCUCLGUH4KL5RBO2GVRYJWOI5OA\",\"cluster\":\"js_egeo_kv_shard1_infile\",\"lnoc\":true}]\n
 [INF] 150.0.0.9:52416 - rid:62 - Route connection created\n
 [TRC] 150.0.0.9:52416 - rid:62 - \u003c\u003c- [CONNECT {\"echo\":true,\"verbose\":false,\"pedantic\":false,\"user\":\"js_egeo_kv_cluster03_infile_user\",\"pass\":\"[REDACTED]\",\"tls_required\":false,\"headers\":true,\"name\":\"NBKYG4TJEC5AK277OUOJO6RYHDR7Q6FJWOUEWG7KIQ5RMOARWVTXNCKG\",\"cluster\":\"js_egeo_kv_shard1_infile\",\"lnoc\":true}]\n
 [INF] 150.0.0.22:36180 - rid:61 - Router connection closed: Client Closed\n
 [INF] 150.0.0.9:52400 - rid:60 - Router connection closed: Client Closed\n
 [INF] 150.0.0.9:52432 - rid:63 - Route connection created\n
 [INF] 150.0.0.9:52436 - rid:64 - Route connection created\n
 [TRC] 150.0.0.9:52432 - rid:63 - \u003c\u003c- [CONNECT {\"echo\":true,\"verbose\":false,\"pedantic\":false,\"user\":\"js_egeo_kv_cluster03_infile_user\",\"pass\":\"[REDACTED]\",\"tls_required\":false,\"headers\":true,\"name\":\"NBKYG4TJEC5AK277OUOJO6RYHDR7Q6FJWOUEWG7KIQ5RMOARWVTXNCKG\",\"cluster\":\"js_egeo_kv_shard1_infile\",\"lnoc\":true}]\n
 [TRC] 150.0.0.9:52436 - rid:64 - \u003c\u003c- [CONNECT {\"echo\":true,\"verbose\":false,\"pedantic\":false,\"user\":\"js_egeo_kv_cluster03_infile_user\",\"pass\":\"[REDACTED]\",\"tls_required\":false,\"headers\":true,\"name\":\"NBKYG4TJEC5AK277OUOJO6RYHDR7Q6FJWOUEWG7KIQ5RMOARWVTXNCKG\",\"cluster\":\"js_egeo_kv_shard1_infile\",\"lnoc\":true}]\n
 [INF] 150.0.0.9:52416 - rid:62 - Router connection closed: Client Closed\n
 [INF] 150.0.0.9:52436 - rid:64 - Router connection closed: Client Closed\n
 [DBG] 150.0.0.9:52432 - rid:63 - Detected duplicate remote route \"NBKYG4TJEC5AK277OUOJO6RYHDR7Q6FJWOUEWG7KIQ5RMOARWVTXNCKG\"\n
 [INF] 150.0.0.9:52432 - rid:63 - Router connection closed: Duplicate Route\n
 [INF] 150.0.0.9:52440 - rid:65 - Route connection created\n
 [TRC] 150.0.0.9:52440 - rid:65 - \u003c\u003c- [CONNECT {\"echo\":true,\"verbose\":false,\"pedantic\":false,\"user\":\"js_egeo_kv_cluster03_infile_user\",\"pass\":\"[REDACTED]\",\"tls_required\":false,\"headers\":true,\"name\":\"NBKYG4TJEC5AK277OUOJO6RYHDR7Q6FJWOUEWG7KIQ5RMOARWVTXNCKG\",\"cluster\":\"js_egeo_kv_shard1_infile\",\"lnoc\":true}]\n
 [INF] 150.0.0.22:36186 - rid:66 - Route connection created\n
 [TRC] 150.0.0.22:36186 - rid:66 - \u003c\u003c- [CONNECT {\"echo\":true,\"verbose\":false,\"pedantic\":false,\"user\":\"js_egeo_kv_cluster03_infile_user\",\"pass\":\"[REDACTED]\",\"tls_required\":false,\"headers\":true,\"name\":\"NBV22NKSB3UESVPFJCHQZ545AI26TOCUCLGUH4KL5RBO2GVRYJWOI5OA\",\"cluster\":\"js_egeo_kv_shard1_infile\",\"lnoc\":true}]\n
 [INF] 150.0.0.9:52456 - rid:67 - Route connection created\n
 [TRC] 150.0.0.9:52456 - rid:67 - \u003c\u003c- [CONNECT {\"echo\":true,\"verbose\":false,\"pedantic\":false,\"user\":\"js_egeo_kv_cluster03_infile_user\",\"pass\":\"[REDACTED]\",\"tls_required\":false,\"headers\":true,\"name\":\"NBKYG4TJEC5AK277OUOJO6RYHDR7Q6FJWOUEWG7KIQ5RMOARWVTXNCKG\",\"cluster\":\"js_egeo_kv_shard1_infile\",\"lnoc\":true}]\n
 [INF] 150.0.0.9:52440 - rid:65 - Router connection closed: Client Closed\n
 [INF] 150.0.0.22:36186 - rid:66 - Router connection closed: Client Closed\n
 [INF] 150.0.0.9:52456 - rid:67 - Router connection closed: Client Closed\n
 [INF] 150.0.0.22:36190 - rid:68 - Route connection created\n
 [TRC] 150.0.0.22:36190 - rid:68 - \u003c\u003c- [CONNECT {\"echo\":true,\"verbose\":false,\"pedantic\":false,\"user\":\"js_egeo_kv_cluster03_infile_user\",\"pass\":\"[REDACTED]\",\"tls_required\":false,\"headers\":true,\"name\":\"NBV22NKSB3UESVPFJCHQZ545AI26TOCUCLGUH4KL5RBO2GVRYJWOI5OA\",\"cluster\":\"js_egeo_kv_shard1_infile\",\"lnoc\":true}]\n
 [INF] 150.0.0.22:36190 - rid:68 - Router connection closed: Client Closed\n
 [INF] 150.0.0.9:52470 - rid:69 - Route connection created\n
 [TRC] 150.0.0.9:52470 - rid:69 - \u003c\u003c- [CONNECT {\"echo\":true,\"verbose\":false,\"pedantic\":false,\"user\":\"js_egeo_kv_cluster03_infile_user\",\"pass\":\"[REDACTED]\",\"tls_required\":false,\"headers\":true,\"name\":\"NBKYG4TJEC5AK277OUOJO6RYHDR7Q6FJWOUEWG7KIQ5RMOARWVTXNCKG\",\"cluster\":\"js_egeo_kv_shard1_infile\",\"lnoc\":true}]\n
 [INF] 150.0.0.9:52470 - rid:69 - Router connection closed: Client Closed\n
 [TRC] 150.0.0.77:6222 - rid:17 - -\u003e\u003e [RMSG $SYS.SERVER.NAVQASAKCWEM567JJSFHP4DTY2GAHDDQYOMOTPM4MOIYIJIV673YFNC7.STATSZ 4632]\n
 [TRC] 150.0.0.75:6222 - rid:19 - -\u003e\u003e [RMSG $SYS.SERVER.NAVQASAKCWEM567JJSFHP4DTY2GAHDDQYOMOTPM4MOIYIJIV673YFNC7.STATSZ 4632]\n
 [TRC] 150.0.0.71:6222 - rid:23 - -\u003e\u003e [RMSG $SYS.SERVER.NAVQASAKCWEM567JJSFHP4DTY2GAHDDQYOMOTPM4MOIYIJIV673YFNC7.STATSZ 4632]\n
 [TRC] 150.0.0.71:6222 - rid:23 - \u003c\u003c- [RMSG $JSC.ARU.$G.dGVGKtkV 44]\n
 [TRC] 150.0.0.71:6222 - rid:23 - \u003c\u003c- MSG_PAYLOAD: [\"\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00y35\\x01\\x00\\x00\\x00\\x00\\xc8\u003e\\x01\\x00\\x00\\x00\\x00\\x00\\x8b\u003e\\x01\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\"]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- [RMSG $NRG.AE.C-R5F-GQUbjX2S $NRG.R.JORnpMu0 42]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- MSG_PAYLOAD: [\"v2fh5Gis\\x02\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x9a\\xe6\\x00\\x00\\x00\\x00\\x00\\x00\\x02\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x9a\\xe6\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\"]\n
 [DBG] RAFT [aZPpTWMr - C-R5F-GQUbjX2S] AppendEntry updating leader to \"v2fh5Gis\"\n
 [DBG] RAFT [aZPpTWMr - C-R5F-GQUbjX2S] AppendEntry did not match 2 59034 with 2 59011\n
 [TRC] 150.0.0.75:6222 - rid:19 - -\u003e\u003e [RS+ $NRG.CR.m3SIgNH6]\n
 [TRC] 150.0.0.77:6222 - rid:17 - -\u003e\u003e [RS+ $NRG.CR.m3SIgNH6]\n
 [TRC] 150.0.0.71:6222 - rid:23 - -\u003e\u003e [RS+ $NRG.CR.m3SIgNH6]\n
 [TRC] 150.0.0.77:6222 - rid:17 - -\u003e\u003e [RMSG $NRG.R.JORnpMu0 $NRG.CR.m3SIgNH6 25]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- [RMSG $NRG.CR.m3SIgNH6 $NRG.R.JORnpMu0 122]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- MSG_PAYLOAD: [\"v2fh5Gis\\x02\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x83\\xe6\\x00\\x00\\x00\\x00\\x00\\x00\\x02\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x83\\xe6\\x00\\x00\\x00\\x00\\x00\\x00\\x02\\x00\\x12\\x00\\x00\\x00\\x00\\a\\x88\\xe1\\x06\\x9a\\x85\\a\\x01\\xc2֛\\xb5\\x88\\xae\\xc6\\xcd/6\\x00\\x00\\x00\\x00\\r_INBOX.D2bZymvymlErPMT3nI4WYy.D2bZymvymlErPMT3nI7IIS\"]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- [RMSG $NRG.CR.m3SIgNH6 $NRG.R.JORnpMu0 64]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- MSG_PAYLOAD: [\"v2fh5Gis\\x02\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x83\\xe6\\x00\\x00\\x00\\x00\\x00\\x00\\x02\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x84\\xe6\\x00\\x00\\x00\\x00\\x00\\x00\\x01\\x00\\x12\\x00\\x00\\x00\\x00\\a\\x89\\xe1\\x06\\x9b\\x85\\a\\x01\\xb8\\x95\\xf5\\xb6\\x88\\xae\\xc6\\xcd/\"]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- [RMSG $NRG.CR.m3SIgNH6 $NRG.R.JORnpMu0 64]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- MSG_PAYLOAD: [\"v2fh5Gis\\x02\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x85\\xe6\\x00\\x00\\x00\\x00\\x00\\x00\\x02\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x85\\xe6\\x00\\x00\\x00\\x00\\x00\\x00\\x01\\x00\\x12\\x00\\x00\\x00\\x00\\a\\x8a\\xe1\\x06\\x9c\\x85\\a\\x01ȴ\\xa6\\xbb\\x88\\xae\\xc6\\xcd/\"]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- [RMSG $NRG.CR.m3SIgNH6 $NRG.R.JORnpMu0 54]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- MSG_PAYLOAD: [\"v2fh5Gis\\x02\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x85\\xe6\\x00\\x00\\x00\\x00\\x00\\x00\\x02\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x86\\xe6\\x00\\x00\\x00\\x00\\x00\\x00\\x01\\x00\\b\\x00\\x00\\x00\\x00\\b\\x88\\xe1\\x06\\x9a\\x85\\a\"]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- [RMSG $NRG.CR.m3SIgNH6 $NRG.R.JORnpMu0 100]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- MSG_PAYLOAD: [\"v2fh5Gis\\x02\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x87\\xe6\\x00\\x00\\x00\\x00\\x00\\x00\\x02\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x87\\xe6\\x00\\x00\\x00\\x00\\x00\\x00\\x01\\x006\\x00\\x00\\x00\\x00\\f_INBOX.D2bZymvymlErPMT3nI4WYy.D2bZymvymlErPMT3nI7IJY\"]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- [RMSG $NRG.CR.m3SIgNH6 $NRG.R.JORnpMu0 100]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- MSG_PAYLOAD: [\"v2fh5Gis\\x02\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x88\\xe6\\x00\\x00\\x00\\x00\\x00\\x00\\x02\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x88\\xe6\\x00\\x00\\x00\\x00\\x00\\x00\\x01\\x006\\x00\\x00\\x00\\x00\\r_INBOX.D2bZymvymlErPMT3nI4WYy.D2bZymvymlErPMT3nI7IIS\"]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- [RMSG $NRG.CR.m3SIgNH6 $NRG.R.JORnpMu0 66]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- MSG_PAYLOAD: [\"v2fh5Gis\\x02\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x89\\xe6\\x00\\x00\\x00\\x00\\x00\\x00\\x02\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x89\\xe6\\x00\\x00\\x00\\x00\\x00\\x00\\x02\\x00\\b\\x00\\x00\\x00\\x00\\b\\x89\\xe1\\x06\\x9b\\x85\\a\\b\\x00\\x00\\x00\\x00\\b\\x8a\\xe1\\x06\\x9c\\x85\\a\"]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- [RMSG $NRG.CR.m3SIgNH6 $NRG.R.JORnpMu0 100]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- MSG_PAYLOAD: [\"v2fh5Gis\\x02\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x8a\\xe6\\x00\\x00\\x00\\x00\\x00\\x00\\x02\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x8a\\xe6\\x00\\x00\\x00\\x00\\x00\\x00\\x01\\x006\\x00\\x00\\x00\\x00\\f_INBOX.D2bZymvymlErPMT3nI4WYy.D2bZymvymlErPMT3nI7IKe\"]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- [RMSG $NRG.CR.m3SIgNH6 $NRG.R.JORnpMu0 122]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- MSG_PAYLOAD: [\"v2fh5Gis\\x02\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x8b\\xe6\\x00\\x00\\x00\\x00\\x00\\x00\\x02\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x8b\\xe6\\x00\\x00\\x00\\x00\\x00\\x00\\x02\\x00\\x12\\x00\\x00\\x00\\x00\\a\\x8b\\xe1\\x06\\x9d\\x85\\a\\x01\\xd0٩\\xbb\\x98\\xae\\xc6\\xcd/6\\x00\\x00\\x00\\x00\\r_INBOX.D2bZymvymlErPMT3nI4WYy.D2bZymvymlErPMT3nI7IJY\"]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- [RMSG $NRG.CR.m3SIgNH6 $NRG.R.JORnpMu0 122]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- MSG_PAYLOAD: [\"v2fh5Gis\\x02\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x8b\\xe6\\x00\\x00\\x00\\x00\\x00\\x00\\x02\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x8c\\xe6\\x00\\x00\\x00\\x00\\x00\\x00\\x02\\x00\\x12\\x00\\x00\\x00\\x00\\a\\x8c\\xe1\\x06\\x9e\\x85\\a\\x01\\xbc\\xab鼘\\xae\\xc6\\xcd/6\\x00\\x00\\x00\\x00\\r_INBOX.D2bZymvymlErPMT3nI4WYy.D2bZymvymlErPMT3nI7IKe\"]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- [RMSG $NRG.CR.m3SIgNH6 $NRG.R.JORnpMu0 54]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- MSG_PAYLOAD: [\"v2fh5Gis\\x02\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x8d\\xe6\\x00\\x00\\x00\\x00\\x00\\x00\\x02\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x8d\\xe6\\x00\\x00\\x00\\x00\\x00\\x00\\x01\\x00\\b\\x00\\x00\\x00\\x00\\b\\x8b\\xe1\\x06\\x9d\\x85\\a\"]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- [RMSG $NRG.CR.m3SIgNH6 $NRG.R.JORnpMu0 100]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- MSG_PAYLOAD: [\"v2fh5Gis\\x02\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x8e\\xe6\\x00\\x00\\x00\\x00\\x00\\x00\\x02\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x8e\\xe6\\x00\\x00\\x00\\x00\\x00\\x00\\x01\\x006\\x00\\x00\\x00\\x00\\f_INBOX.D2bZymvymlErPMT3nI4WYy.D2bZymvymlErPMT3nI7ILk\"]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- [RMSG $NRG.CR.m3SIgNH6 $NRG.R.JORnpMu0 122]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- MSG_PAYLOAD: [\"v2fh5Gis\\x02\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x8e\\xe6\\x00\\x00\\x00\\x00\\x00\\x00\\x02\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x8f\\xe6\\x00\\x00\\x00\\x00\\x00\\x00\\x02\\x00\\x12\\x00\\x00\\x00\\x00\\a\\x8d\\xe1\\x06\\x9f\\x85\\a\\x01\\x86\\xa2ÿ\\x98\\xae\\xc6\\xcd/6\\x00\\x00\\x00\\x00\\r_INBOX.D2bZymvymlErPMT3nI4WYy.D2bZymvymlErPMT3nI7ILk\"]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- [RMSG $NRG.CR.m3SIgNH6 $NRG.R.JORnpMu0 54]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- MSG_PAYLOAD: [\"v2fh5Gis\\x02\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x90\\xe6\\x00\\x00\\x00\\x00\\x00\\x00\\x02\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x90\\xe6\\x00\\x00\\x00\\x00\\x00\\x00\\x01\\x00\\b\\x00\\x00\\x00\\x00\\b\\x8c\\xe1\\x06\\x9e\\x85\\a\"]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- [RMSG $NRG.CR.m3SIgNH6 $NRG.R.JORnpMu0 54]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- MSG_PAYLOAD: [\"v2fh5Gis\\x02\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x90\\xe6\\x00\\x00\\x00\\x00\\x00\\x00\\x02\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x91\\xe6\\x00\\x00\\x00\\x00\\x00\\x00\\x01\\x00\\b\\x00\\x00\\x00\\x00\\b\\x8d\\xe1\\x06\\x9f\\x85\\a\"]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- [RMSG $NRG.CR.m3SIgNH6 $NRG.R.JORnpMu0 100]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- MSG_PAYLOAD: [\"v2fh5Gis\\x02\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x92\\xe6\\x00\\x00\\x00\\x00\\x00\\x00\\x02\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x92\\xe6\\x00\\x00\\x00\\x00\\x00\\x00\\x01\\x006\\x00\\x00\\x00\\x00\\f_INBOX.D2bZymvymlErPMT3nI4WYy.D2bZymvymlErPMT3nI7IMq\"]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- [RMSG $NRG.CR.m3SIgNH6 $NRG.R.JORnpMu0 100]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- MSG_PAYLOAD: [\"v2fh5Gis\\x02\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x93\\xe6\\x00\\x00\\x00\\x00\\x00\\x00\\x02\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x93\\xe6\\x00\\x00\\x00\\x00\\x00\\x00\\x01\\x006\\x00\\x00\\x00\\x00\\r_INBOX.D2bZymvymlErPMT3nI4WYy.D2bZymvymlErPMT3nI7IMq\"]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- [RMSG $NRG.CR.m3SIgNH6 $NRG.R.JORnpMu0 100]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- MSG_PAYLOAD: [\"v2fh5Gis\\x02\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x93\\xe6\\x00\\x00\\x00\\x00\\x00\\x00\\x02\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x94\\xe6\\x00\\x00\\x00\\x00\\x00\\x00\\x01\\x006\\x00\\x00\\x00\\x00\\f_INBOX.D2bZymvymlErPMT3nI4WYy.D2bZymvymlErPMT3nI7IMq\"]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- [RMSG $NRG.CR.m3SIgNH6 $NRG.R.JORnpMu0 100]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- MSG_PAYLOAD: [\"v2fh5Gis\\x02\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x93\\xe6\\x00\\x00\\x00\\x00\\x00\\x00\\x02\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x95\\xe6\\x00\\x00\\x00\\x00\\x00\\x00\\x01\\x006\\x00\\x00\\x00\\x00\\r_INBOX.D2bZymvymlErPMT3nI4WYy.D2bZymvymlErPMT3nI7IMq\"]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- [RMSG $NRG.CR.m3SIgNH6 $NRG.R.JORnpMu0 100]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- MSG_PAYLOAD: [\"v2fh5Gis\\x02\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x93\\xe6\\x00\\x00\\x00\\x00\\x00\\x00\\x02\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x96\\xe6\\x00\\x00\\x00\\x00\\x00\\x00\\x01\\x006\\x00\\x00\\x00\\x00\\f_INBOX.D2bZymvymlErPMT3nI4WYy.D2bZymvymlErPMT3nI7INw\"]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- [RMSG $NRG.CR.m3SIgNH6 $NRG.R.JORnpMu0 144]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- MSG_PAYLOAD: [\"v2fh5Gis\\x02\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x97\\xe6\\x00\\x00\\x00\\x00\\x00\\x00\\x02\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x97\\xe6\\x00\\x00\\x00\\x00\\x00\\x00\\x03\\x00\\x12\\x00\\x00\\x00\\x00\\a\\x8e\\xe1\\x06\\xa0\\x85\\a\\x01\\x9c\\xfd\\x8e\\x90Ӯ\\xc6\\xcd/\\x12\\x00\\x00\\x00\\x00\\a\\x8f\\xe1\\x06\\xa1\\x85\\a\\x01\\xec塐Ӯ\\xc6\\xcd/6\\x00\\x00\\x00\\x00\\r_INBOX.D2bZymvymlErPMT3nI4WYy.D2bZymvymlErPMT3nI7INw\"]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- [RMSG $NRG.CR.m3SIgNH6 $NRG.R.JORnpMu0 66]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- MSG_PAYLOAD: [\"v2fh5Gis\\x02\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x98\\xe6\\x00\\x00\\x00\\x00\\x00\\x00\\x02\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x98\\xe6\\x00\\x00\\x00\\x00\\x00\\x00\\x02\\x00\\b\\x00\\x00\\x00\\x00\\b\\x8e\\xe1\\x06\\xa0\\x85\\a\\b\\x00\\x00\\x00\\x00\\b\\x8f\\xe1\\x06\\xa1\\x85\\a\"]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- [RMSG $NRG.CR.m3SIgNH6 $NRG.R.JORnpMu0 100]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- MSG_PAYLOAD: [\"v2fh5Gis\\x02\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x99\\xe6\\x00\\x00\\x00\\x00\\x00\\x00\\x02\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x99\\xe6\\x00\\x00\\x00\\x00\\x00\\x00\\x01\\x006\\x00\\x00\\x00\\x00\\f_INBOX.D2bZymvymlErPMT3nI4WYy.D2bZymvymlErPMT3nI7IP2\"]\n
 [TRC] 150.0.0.77:6222 - rid:17 - -\u003e\u003e [RMSG $NRG.R.JORnpMu0 25]\n
 [TRC] 150.0.0.77:6222 - rid:17 - -\u003e\u003e [RMSG $NRG.R.JORnpMu0 25]\n
 [TRC] 150.0.0.77:6222 - rid:17 - -\u003e\u003e [RMSG $NRG.R.JORnpMu0 25]\n
 [TRC] 150.0.0.77:6222 - rid:17 - -\u003e\u003e [RMSG $NRG.R.JORnpMu0 25]\n
 [TRC] 150.0.0.77:6222 - rid:17 - -\u003e\u003e [RMSG $NRG.R.JORnpMu0 25]\n
 [TRC] 150.0.0.77:6222 - rid:17 - -\u003e\u003e [RMSG $NRG.R.JORnpMu0 25]\n
 [TRC] 150.0.0.77:6222 - rid:17 - -\u003e\u003e [RMSG $NRG.R.JORnpMu0 25]\n
 [TRC] 150.0.0.77:6222 - rid:17 - -\u003e\u003e [RMSG $NRG.R.JORnpMu0 25]\n
 [TRC] 150.0.0.77:6222 - rid:17 - -\u003e\u003e [RMSG $NRG.R.JORnpMu0 25]\n
 [TRC] 150.0.0.77:6222 - rid:17 - -\u003e\u003e [RMSG $NRG.R.JORnpMu0 25]\n
 [TRC] 150.0.0.77:6222 - rid:17 - -\u003e\u003e [RMSG $NRG.R.JORnpMu0 25]\n
 [TRC] 150.0.0.77:6222 - rid:17 - -\u003e\u003e [RMSG $NRG.R.JORnpMu0 25]\n
 [TRC] 150.0.0.77:6222 - rid:17 - -\u003e\u003e [RMSG $NRG.R.JORnpMu0 25]\n
 [TRC] 150.0.0.77:6222 - rid:17 - -\u003e\u003e [RMSG $NRG.R.JORnpMu0 25]\n
 [TRC] 150.0.0.77:6222 - rid:17 - -\u003e\u003e [RMSG $NRG.R.JORnpMu0 25]\n
 [TRC] 150.0.0.77:6222 - rid:17 - -\u003e\u003e [RMSG $NRG.R.JORnpMu0 25]\n
 [TRC] 150.0.0.77:6222 - rid:17 - -\u003e\u003e [RMSG $NRG.R.JORnpMu0 25]\n
 [TRC] 150.0.0.77:6222 - rid:17 - -\u003e\u003e [RMSG $NRG.R.JORnpMu0 25]\n
 [TRC] 150.0.0.77:6222 - rid:17 - -\u003e\u003e [RMSG $NRG.R.JORnpMu0 25]\n
 [TRC] 150.0.0.77:6222 - rid:17 - -\u003e\u003e [RMSG $NRG.R.JORnpMu0 25]\n
 [TRC] 150.0.0.77:6222 - rid:17 - -\u003e\u003e [RMSG $NRG.R.JORnpMu0 25]\n
 [TRC] 150.0.0.77:6222 - rid:17 - -\u003e\u003e [RMSG $NRG.R.JORnpMu0 25]\n
 [TRC] 150.0.0.77:6222 - rid:17 - -\u003e\u003e [RMSG $NRG.R.JORnpMu0 25]\n
 [DBG] Checking JetStream cluster state\n
 [DBG] Trying to connect to route on js_egeo_kv_shard1_infile_node05:6222 (10.20.126.30:6222)\n
 [DBG] Error trying to connect to route (attempt 2): dial tcp 10.20.126.30:6222: connect: connection refused\n
 [DBG] Trying to connect to route on js_egeo_kv_shard1_infile_node05:6222 (10.20.126.30:6222)\n
 [DBG] Error trying to connect to route (attempt 2): dial tcp 10.20.126.30:6222: connect: connection refused\n
 [DBG] 150.0.0.77:6222 - rid:17 - Router Ping Timer\n
 [TRC] 150.0.0.77:6222 - rid:17 - -\u003e\u003e [PING]\n
 [TRC] 150.0.0.77:6222 - rid:17 - \u003c\u003c- [PONG]\n
 [DBG] 150.0.0.71:6222 - rid:20 - Router Ping Timer\n
 [TRC] 150.0.0.71:6222 - rid:20 - -\u003e\u003e [PING]\n
 [TRC] 150.0.0.71:6222 - rid:20 - \u003c\u003c- [PONG]\n
 [DBG] 150.0.0.75:6222 - rid:19 - Router Ping Timer\n
 [TRC] 150.0.0.75:6222 - rid:19 - -\u003e\u003e [PING]\n
 [TRC] 150.0.0.75:6222 - rid:22 - \u003c\u003c- [PING]\n
 [TRC] 150.0.0.75:6222 - rid:22 - -\u003e\u003e [PONG]\n
 [TRC] 150.0.0.75:6222 - rid:19 - \u003c\u003c- [PONG]\n
 [TRC] 150.0.0.71:6222 - rid:20 - \u003c\u003c- [PING]\n
 [TRC] 150.0.0.71:6222 - rid:20 - -\u003e\u003e [PONG]\n
 [TRC] 150.0.0.75:6222 - rid:19 - \u003c\u003c- [PING]\n
 [TRC] 150.0.0.75:6222 - rid:19 - -\u003e\u003e [PONG]\n
 [TRC] 150.0.0.71:6222 - rid:23 - \u003c\u003c- [RMSG $NRG.AE._meta_ $NRG.R.fYtaPy9U 42]\n
 [TRC] 150.0.0.71:6222 - rid:23 - \u003c\u003c- MSG_PAYLOAD: [\"dGVGKtkV\\x03\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x12\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x03\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x12\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\"]\n
 [TRC] 150.0.0.71:6222 - rid:23 - -\u003e\u003e [RMSG $NRG.R.fYtaPy9U 25]\n
 [DBG] 150.0.0.71:6222 - rid:23 - Router Ping Timer\n
 [TRC] 150.0.0.71:6222 - rid:23 - -\u003e\u003e [PING]\n
 [TRC] 150.0.0.71:6222 - rid:23 - \u003c\u003c- [PONG]\n
 [DBG] 150.0.0.75:6222 - rid:25 - Router Ping Timer\n
 [TRC] 150.0.0.75:6222 - rid:25 - -\u003e\u003e [PING]\n
 [TRC] 150.0.0.75:6222 - rid:25 - \u003c\u003c- [PONG]\n
 [TRC] 150.0.0.75:6222 - rid:19 - \u003c\u003c- [RMSG $SYS.SERVER.NC5YXRJT4EBEHLOEFBDCDEBHKZI5ILHKT7OORNVDUJJUYBTIXM4SSKVS.STATSZ 4640]\n
 [TRC] 150.0.0.75:6222 - rid:19 - \u003c\u003c- MSG_PAYLOAD: [\"{\\\"server\\\":{\\\"name\\\":\\\"js_egeo_kv_shard1_infile_node03\\\",\\\"host\\\":\\\"0.0.0.0\\\",\\\"id\\\":\\\"NC5YXRJT4EBEHLOEFBDCDEBHKZI5ILHKT7OORNVDUJJUYBTIXM4SSKVS\\\",\\\"cluster\\\":\\\"js_egeo_kv_shard1_infile\\\",\\\"ver\\\":\\\"2.11.0-dev\\\",\\\"jetstream\\\":true,\\\"flags\\\":3,\\\"seq\\\":26,\\\"time\\\":\\\"2024-05-08T15:22:47.301192623Z\\\"},\\\"statsz\\\":{\\\"start\\\":\\\"2024-05-08T15:22:43.547667697Z\\\",\\\"mem\\\":180326400,\\\"cores\\\":5,\\\"cpu\\\":5,\\\"connections\\\":0,\\\"total_connections\\\":0,\\\"active_accounts\\\":2,\\\"subscriptions\\\":322,\\\"sent\\\":{\\\"msgs\\\":77,\\\"bytes\\\":39907},\\\"received\\\":{\\\"msgs\\\":78,\\\"bytes\\\":41642},\\\"slow_consumers\\\":0,\\\"routes\\\":[{\\\"rid\\\":19,\\\"name\\\":\\\"js_egeo_kv_shard1_infile_node04\\\",\\\"sent\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"received\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"pending\\\":0},{\\\"rid\\\":23,\\\"name\\\":\\\"js_egeo_kv_shard1_infile_node04\\\",\\\"sent\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"received\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"pending\\\":0},{\\\"rid\\\":25,\\\"name\\\":\\\"js_egeo_kv_shard1_infile_node04\\\",\\\"sent\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"received\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"pending\\\":0},{\\\"rid\\\":27,\\\"name\\\":\\\"js_egeo_kv_shard1_infile_node04\\\",\\\"sent\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"received\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"pending\\\":0},{\\\"rid\\\":28,\\\"name\\\":\\\"js_egeo_kv_shard1_infile_node04\\\",\\\"sent\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"received\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"pending\\\":0},{\\\"rid\\\":29,\\\"name\\\":\\\"js_egeo_kv_shard1_infile_node04\\\",\\\"sent\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"received\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"pending\\\":0},{\\\"rid\\\":31,\\\"name\\\":\\\"js_egeo_kv_shard1_infile_node04\\\",\\\"sent\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"received\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"pending\\\":0},{\\\"rid\\\":32,\\\"name\\\":\\\"js_egeo_kv_shard1_infile_node04\\\",\\\"sent\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"received\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"pending\\\":0},{\\\"rid\\\":33,\\\"name\\\":\\\"js_egeo_kv_shard1_infile_node04\\\",\\\"sent\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"received\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"pending\\\":0},{\\\"rid\\\":21,\\\"name\\\":\\\"js_egeo_kv_shard1_infile_node02\\\",\\\"sent\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"received\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"pending\\\":0},{\\\"rid\\\":22,\\\"name\\\":\\\"js_egeo_kv_shard1_infile_node02\\\",\\\"sent\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"received\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"pending\\\":0},{\\\"rid\\\":24,\\\"name\\\":\\\"js_egeo_kv_shard1_infile_node02\\\",\\\"sent\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"received\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"pending\\\":0},{\\\"rid\\\":26,\\\"name\\\":\\\"js_egeo_kv_shard1_infile_node02\\\",\\\"sent\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"received\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"pending\\\":0},{\\\"rid\\\":30,\\\"name\\\":\\\"js_egeo_kv_shard1_infile_node02\\\",\\\"sent\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"received\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"pending\\\":0},{\\\"rid\\\":34,\\\"name\\\":\\\"js_egeo_kv_shard1_infile_node02\\\",\\\"sent\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"received\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"pending\\\":0},{\\\"rid\\\":35,\\\"name\\\":\\\"js_egeo_kv_shard1_infile_node02\\\",\\\"sent\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"received\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"pending\\\":0},{\\\"rid\\\":36,\\\"name\\\":\\\"js_egeo_kv_shard1_infile_node02\\\",\\\"sent\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"received\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"pending\\\":0},{\\\"rid\\\":38,\\\"name\\\":\\\"js_egeo_kv_shard1_infile_node02\\\",\\\"sent\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"received\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"pending\\\":0},{\\\"rid\\\":58,\\\"name\\\":\\\"js_egeo_kv_shard1_infile_node01\\\",\\\"sent\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"received\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"pending\\\":0},{\\\"rid\\\":60,\\\"name\\\":\\\"js_egeo_kv_shard1_infile_node01\\\",\\\"sent\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"received\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"pending\\\":0},{\\\"rid\\\":61,\\\"name\\\":\\\"js_egeo_kv_shard1_infile_node01\\\",\\\"sent\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"received\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"pending\\\":0},{\\\"rid\\\":62,\\\"name\\\":\\\"js_egeo_kv_shard1_infile_node01\\\",\\\"sent\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"received\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"pending\\\":0},{\\\"rid\\\":63,\\\"name\\\":\\\"js_egeo_kv_shard1_infile_node01\\\",\\\"sent\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"received\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"pending\\\":0},{\\\"rid\\\":64,\\\"name\\\":\\\"js_egeo_kv_shard1_infile_node01\\\",\\\"sent\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"received\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"pending\\\":0},{\\\"rid\\\":65,\\\"name\\\":\\\"js_egeo_kv_shard1_infile_node01\\\",\\\"sent\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"received\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"pending\\\":0},{\\\"rid\\\":66,\\\"name\\\":\\\"js_egeo_kv_shard1_infile_node01\\\",\\\"sent\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"received\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"pending\\\":0},{\\\"rid\\\":67,\\\"name\\\":\\\"js_egeo_kv_shard1_infile_node01\\\",\\\"sent\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"received\\\":{\\\"msgs\\\":0,\\\"bytes\\\":0},\\\"pending\\\":0},{\\\"rid\\\":20,\\\"name\\\":\\\"js_egeo_kv_shard1_infile_node04\\\",\\\"sent\\\":{\\\"msgs\\\":46,\\\"bytes\\\":18137},\\\"received\\\":{\\\"msgs\\\":44,\\\"bytes\\\":11464},\\\"pending\\\":0},{\\\"rid\\\":18,\\\"name\\\":\\\"js_egeo_kv_shard1_infile_node02\\\",\\\"sent\\\":{\\\"msgs\\\":29,\\\"bytes\\\":17712},\\\"received\\\":{\\\"msgs\\\":29,\\\"bytes\\\":13506},\\\"pending\\\":0},{\\\"rid\\\":59,\\\"name\\\":\\\"js_egeo_kv_shard1_infile_node01\\\",\\\"sent\\\":{\\\"msgs\\\":2,\\\"bytes\\\":4058},\\\"received\\\":{\\\"msgs\\\":5,\\\"bytes\\\":16672},\\\"pending\\\":0}],\\\"active_servers\\\":4,\\\"jetstream\\\":{\\\"config\\\":{\\\"max_memory\\\":3145728000,\\\"max_storage\\\":10737418240,\\\"sync_interval\\\":120000000000},\\\"stats\\\":{\\\"memory\\\":0,\\\"storage\\\":20263801,\\\"reserved_memory\\\":0,\\\"reserved_storage\\\":0,\\\"accounts\\\":1,\\\"ha_assets\\\":3,\\\"api\\\":{\\\"total\\\":0,\\\"errors\\\":0}},\\\"meta\\\":{\\\"name\\\":\\\"_meta_\\\",\\\"leader\\\":\\\"js_egeo_kv_shard1_infile_node02\\\",\\\"peer\\\":\\\"dGVGKtkV\\\",\\\"cluster_size\\\":5}}}}\"]\n
 [TRC] 150.0.0.77:6222 - rid:24 - \u003c\u003c- [PING]\n
 [TRC] 150.0.0.77:6222 - rid:24 - -\u003e\u003e [PONG]\n
 [TRC] 150.0.0.75:6222 - rid:25 - \u003c\u003c- [PING]\n
 [TRC] 150.0.0.75:6222 - rid:25 - -\u003e\u003e [PONG]\n
 [DBG] 150.0.0.77:6222 - rid:18 - Router Ping Timer\n

bfoxstudio commented 6 months ago

After restart 1 by 1 node in cluster on 2.10.14 we lost 60% kv storage. Reproduce on 1 of 10 clusters after restart.

[INF]   Starting restore for stream '$G \u003e KV_wh-px-egeo-e-shard1-kv'\n
 [INF]   Restored 64,596 messages for stream '$G \u003e KV_wh-px-egeo-e-shard1-kv' in 55ms\n
 [INF]   Recovering 1 consumers for stream - '$G \u003e KV_wh-px-egeo-e-shard1-kv'\n
 ---
 [INF]   Starting restore for stream '$G \u003e KV_wh-px-egeo-e-shard1-kv'\n
 [INF]   Restored 17,641 messages for stream '$G \u003e KV_wh-px-egeo-e-shard1-kv' in 18ms\n
 [INF]   Recovering 1 consumers for stream - '$G \u003e KV_wh-px-egeo-e-shard1-kv'\n

derekcollison commented 6 months ago

What does stream info for the underlying KV show?

bfoxstudio commented 6 months ago

01 node

~ # nats stream info KV_wh-px-egeo-e-shard1-kv
Information for Stream KV_wh-px-egeo-e-shard1-kv created 2024-05-08 14:38:06

              Subjects: $KV.wh-px-egeo-e-shard1-kv.>
              Replicas: 5
               Storage: File

Options:

             Retention: Limits
       Acknowledgments: true
        Discard Policy: New
      Duplicate Window: 2m0s
     Allows Msg Delete: false
          Allows Purge: true
        Allows Rollups: true

Limits:

      Maximum Messages: unlimited
   Maximum Per Subject: 1
         Maximum Bytes: unlimited
           Maximum Age: unlimited
  Maximum Message Size: unlimited
     Maximum Consumers: unlimited

Cluster Information:

                  Name: js_egeo_kv_shard1_infile
                Leader: js_egeo_kv_shard1_infile_node02
               Replica: js_egeo_kv_shard1_infile_node01, current, seen 686ms ago
               Replica: js_egeo_kv_shard1_infile_node03, current, seen 685ms ago
               Replica: js_egeo_kv_shard1_infile_node04, current, seen 685ms ago
               Replica: js_egeo_kv_shard1_infile_node05, current, seen 685ms ago

State:

              Messages: 64,660
                 Bytes: 19 MiB
        First Sequence: 806 @ 2024-05-08 14:54:04 UTC
         Last Sequence: 157,636 @ 2024-05-09 20:32:22 UTC
      Deleted Messages: 92,171
      Active Consumers: 1
    Number of Subjects: 64,660

02 node

  ~ # nats stream info KV_wh-px-egeo-e-shard1-kv
Information for Stream KV_wh-px-egeo-e-shard1-kv created 2024-05-08 14:38:06

              Subjects: $KV.wh-px-egeo-e-shard1-kv.>
              Replicas: 5
               Storage: File

Options:

             Retention: Limits
       Acknowledgments: true
        Discard Policy: New
      Duplicate Window: 2m0s
     Allows Msg Delete: false
          Allows Purge: true
        Allows Rollups: true

Limits:

      Maximum Messages: unlimited
   Maximum Per Subject: 1
         Maximum Bytes: unlimited
           Maximum Age: unlimited
  Maximum Message Size: unlimited
     Maximum Consumers: unlimited

Cluster Information:

                  Name: js_egeo_kv_shard1_infile
                Leader: js_egeo_kv_shard1_infile_node02
               Replica: js_egeo_kv_shard1_infile_node01, current, seen 269ms ago
               Replica: js_egeo_kv_shard1_infile_node03, current, seen 268ms ago
               Replica: js_egeo_kv_shard1_infile_node04, current, seen 268ms ago
               Replica: js_egeo_kv_shard1_infile_node05, current, seen 268ms ago

State:

              Messages: 64,660
                 Bytes: 19 MiB
        First Sequence: 806 @ 2024-05-08 14:54:04 UTC
         Last Sequence: 157,648 @ 2024-05-09 20:33:42 UTC
      Deleted Messages: 92,183
      Active Consumers: 1
    Number of Subjects: 64,660

bfoxstudio commented 6 months ago

What does stream info for the underlying KV show?

You can look on last 10k lines log t0lbztwfu30jthnczav7o3nne_logs.txt

01 STOP DELETE

look on "num_deleted": 59009,

{
  "server_id": "NBKR7V4N2IAWWSJUZVTQTSO52LIZBYNPWX7A7VFKVVK67MNSJR563W3T",
  "now": "2024-05-09T20:54:20.868114301Z",
  "config": {
    "max_memory": 3145728000,
    "max_storage": 10737418240,
    "store_dir": "/data/jetstream/jetstream",
    "sync_interval": 120000000000
  },
  "memory": 0,
  "storage": 20233449,
  "reserved_memory": 0,
  "reserved_storage": 0,
  "accounts": 1,
  "ha_assets": 3,
  "api": {
    "total": 1,
    "errors": 0
  },
  "streams": 1,
  "consumers": 1,
  "messages": 64659,
  "bytes": 20233449,
  "meta_cluster": {
    "name": "js_egeo_kv_shard1_infile",
    "leader": "js_egeo_kv_shard1_infile_node02",
    "peer": "dGVGKtkV",
    "cluster_size": 5
  },
  "account_details": [
    {
      "name": "$G",
      "id": "$G",
      "memory": 0,
      "storage": 101490613,
      "reserved_memory": 18446744073709551615,
      "reserved_storage": 18446744073709551615,
      "accounts": 0,
      "ha_assets": 0,
      "api": {
        "total": 145014,
        "errors": 81630
      },
      "stream_detail": [
        {
          "name": "KV_wh-px-egeo-e-shard1-kv",
          "created": "2024-05-08T14:38:06.541612784Z",
          "cluster": {
            "name": "js_egeo_kv_shard1_infile",
            "raft_group": "S-R5F-Z6jshyaK",
            "leader": "js_egeo_kv_shard1_infile_node02",
            "replicas": [
              {
                "name": "js_egeo_kv_shard1_infile_node02",
                "current": true,
                "active": 946149362,
                "lag": 111562,
                "peer": "dGVGKtkV"
              },
              {
                "name": "js_egeo_kv_shard1_infile_node03",
                "current": false,
                "active": 0,
                "lag": 111562,
                "peer": "JrN1hGZO"
              },
              {
                "name": "js_egeo_kv_shard1_infile_node04",
                "current": false,
                "active": 0,
                "lag": 111562,
                "peer": "v2fh5Gis"
              },
              {
                "name": "js_egeo_kv_shard1_infile_node05",
                "current": false,
                "active": 0,
                "lag": 111562,
                "peer": "bXlHueQT"
              }
            ]
          },
          "state": {
            "messages": 64659,
            "bytes": 20233449,
            "first_seq": 806,
            "first_ts": "2024-05-08T14:54:04.226161003Z",
            "last_seq": 124472,
            "last_ts": "2024-05-09T20:53:11.947240373Z",
            "num_subjects": 64659,
            "num_deleted": 59008,
            "consumer_count": 1
          }
        }
      ]
    }
  ]
}

02

{
  "server_id": "NBKYG4TJEC5AK277OUOJO6RYHDR7Q6FJWOUEWG7KIQ5RMOARWVTXNCKG",
  "now": "2024-05-09T20:54:58.381639604Z",
  "config": {
    "max_memory": 3145728000,
    "max_storage": 10737418240,
    "store_dir": "/data/jetstream/jetstream",
    "sync_interval": 120000000000
  },
  "memory": 0,
  "storage": 20314380,
  "reserved_memory": 0,
  "reserved_storage": 0,
  "accounts": 1,
  "ha_assets": 3,
  "api": {
    "total": 81751,
    "errors": 81630
  },
  "streams": 1,
  "consumers": 1,
  "messages": 64660,
  "bytes": 20314380,
  "meta_cluster": {
    "name": "js_egeo_kv_shard1_infile",
    "leader": "js_egeo_kv_shard1_infile_node02",
    "peer": "dGVGKtkV",
    "replicas": [
      {
        "name": "js_egeo_kv_shard1_infile_node01",
        "current": true,
        "active": 96882235,
        "peer": "aZPpTWMr"
      },
      {
        "name": "js_egeo_kv_shard1_infile_node03",
        "current": true,
        "active": 96370735,
        "peer": "JrN1hGZO"
      },
      {
        "name": "js_egeo_kv_shard1_infile_node04",
        "current": true,
        "active": 96051661,
        "peer": "v2fh5Gis"
      },
      {
        "name": "js_egeo_kv_shard1_infile_node05",
        "current": true,
        "active": 96104740,
        "peer": "bXlHueQT"
      }
    ],
    "cluster_size": 5
  },
  "account_details": [
    {
      "name": "$G",
      "id": "$G",
      "memory": 0,
      "storage": 101490969,
      "reserved_memory": 18446744073709551615,
      "reserved_storage": 18446744073709551615,
      "accounts": 0,
      "ha_assets": 0,
      "api": {
        "total": 145032,
        "errors": 81630
      },
      "stream_detail": [
        {
          "name": "KV_wh-px-egeo-e-shard1-kv",
          "created": "2024-05-08T14:38:06.541612784Z",
          "cluster": {
            "name": "js_egeo_kv_shard1_infile",
            "raft_group": "S-R5F-Z6jshyaK",
            "leader": "js_egeo_kv_shard1_infile_node02",
            "replicas": [
              {
                "name": "js_egeo_kv_shard1_infile_node01",
                "current": true,
                "active": 458920330,
                "peer": "aZPpTWMr"
              },
              {
                "name": "js_egeo_kv_shard1_infile_node03",
                "current": true,
                "active": 458401464,
                "peer": "JrN1hGZO"
              },
              {
                "name": "js_egeo_kv_shard1_infile_node04",
                "current": true,
                "active": 458134233,
                "peer": "v2fh5Gis"
              },
              {
                "name": "js_egeo_kv_shard1_infile_node05",
                "current": true,
                "active": 458092888,
                "peer": "bXlHueQT"
              }
            ]
          },
          "state": {
            "messages": 64660,
            "bytes": 20314380,
            "first_seq": 806,
            "first_ts": "2024-05-08T14:54:04.226161003Z",
            "last_seq": 157924,
            "last_ts": "2024-05-09T20:54:53.169845826Z",
            "num_subjects": 64660,
            "num_deleted": 92459,
            "consumer_count": 1
          }
        }
      ]
    }
  ]
}

03

{
  "server_id": "NAFE5PQME7QQHQQ7DRNGZH52LRLDWE4SXC36QLVMBKBWXKDLWLOTZY3Q",
  "now": "2024-05-09T20:55:41.189096784Z",
  "config": {
    "max_memory": 3145728000,
    "max_storage": 10737418240,
    "store_dir": "/data/jetstream/jetstream",
    "sync_interval": 120000000000
  },
  "memory": 0,
  "storage": 20314290,
  "reserved_memory": 0,
  "reserved_storage": 0,
  "accounts": 1,
  "ha_assets": 3,
  "api": {
    "total": 0,
    "errors": 0
  },
  "streams": 1,
  "consumers": 1,
  "messages": 64660,
  "bytes": 20314290,
  "meta_cluster": {
    "name": "js_egeo_kv_shard1_infile",
    "leader": "js_egeo_kv_shard1_infile_node02",
    "peer": "dGVGKtkV",
    "cluster_size": 5
  },
  "account_details": [
    {
      "name": "$G",
      "id": "$G",
      "memory": 0,
      "storage": 101490607,
      "reserved_memory": 18446744073709551615,
      "reserved_storage": 18446744073709551615,
      "accounts": 0,
      "ha_assets": 0,
      "api": {
        "total": 145053,
        "errors": 81630
      },
      "stream_detail": [
        {
          "name": "KV_wh-px-egeo-e-shard1-kv",
          "created": "2024-05-08T14:38:06.541612784Z",
          "cluster": {
            "name": "js_egeo_kv_shard1_infile",
            "raft_group": "S-R5F-Z6jshyaK",
            "leader": "js_egeo_kv_shard1_infile_node02",
            "replicas": [
              {
                "name": "js_egeo_kv_shard1_infile_node01",
                "current": false,
                "active": 0,
                "lag": 111584,
                "peer": "aZPpTWMr"
              },
              {
                "name": "js_egeo_kv_shard1_infile_node02",
                "current": true,
                "active": 266734506,
                "lag": 111584,
                "peer": "dGVGKtkV"
              },
              {
                "name": "js_egeo_kv_shard1_infile_node04",
                "current": false,
                "active": 0,
                "lag": 111584,
                "peer": "v2fh5Gis"
              },
              {
                "name": "js_egeo_kv_shard1_infile_node05",
                "current": false,
                "active": 0,
                "lag": 111584,
                "peer": "bXlHueQT"
              }
            ]
          },
          "state": {
            "messages": 64660,
            "bytes": 20314290,
            "first_seq": 806,
            "first_ts": "2024-05-08T14:54:04.226161003Z",
            "last_seq": 157936,
            "last_ts": "2024-05-09T20:55:33.025912043Z",
            "num_subjects": 64660,
            "num_deleted": 92471,
            "consumer_count": 1
          }
        }
      ]
    }
  ]
}

config

          "config": {
            "name": "KV_wh-px-egeo-e-shard1-kv",
            "subjects": [
              "$KV.wh-px-egeo-e-shard1-kv.\u003e"
            ],
            "retention": "limits",
            "max_consumers": -1,
            "max_msgs": -1,
            "max_bytes": -1,
            "max_age": 0,
            "max_msgs_per_subject": 1,
            "max_msg_size": -1,
            "discard": "new",
            "storage": "file",
            "num_replicas": 5,
            "duplicate_window": 120000000000,
            "compression": "none",
            "allow_direct": false,
            "mirror_direct": false,
            "sealed": false,
            "deny_delete": true,
            "deny_purge": false,
            "allow_rollup_hdrs": true,
            "consumer_limits": {}
          },

derekcollison commented 6 months ago

We fixed an issue with discard new that is in main and will be in 2.10.15, however your KV may have already had inconsistencies so when a new leader was elected it used its state.

The way to sync a known good state is make sure that replica is leader, and scale down to 1 then back up. Once upgraded to 2.10.15 the issue should not occur anymore.

bfoxstudio commented 6 months ago

we create new clear cluster

                     Version: 2.11.0-dev
                  Git Commit: 31b155b
                  Go Version: go1.21.9

We fixed an issue with discard new that is in main and will be in 2.10.15, however your KV may have already had inconsistencies so when a new leader was elected it used its state.

The way to sync a known good state is make sure that replica is leader, and scale down to 1 then back up. Once upgraded to 2.10.15 the issue should not occur anymore.

reproduced... if change replica to 1 and return to 5 last sequence synced.

bfoxstudio commented 6 months ago

Try to terminate the server unsafe. i think problem near Sync Interval in file storage. if we kill meta leader we have problem on consumers - no new messages on subscribe and lag....

We try in sync mode. 5 msg lost after stop server.

log when server node stop:

derekcollison commented 6 months ago

Could you retest from top of main branch or nightly builds?

bfoxstudio commented 5 months ago

Could you retest from top of main branch or nightly builds?

on latest nightly

sequence problem not fixed.
consumer problem (stop read new messages).

derekcollison commented 5 months ago

What sync interval is the system configured with?

Woukld you be open to a video call?

grrrvahrrr commented 5 months ago

What sync interval is the system configured with?

Woukld you be open to a video call?

Hi! Sorry for the delayed answer. If it is possible, let's organize a call next week. We continue testing using the latest version and trying to find the roots of this behaviour.

derekcollison commented 5 months ago

Looping in @neilalexander and @Jarema and @wallyqs who can coordinate.

Jarema commented 5 months ago

Hey @grrrvahrrr! Please ping me on NATS slack (Tomasz Pietrek) or via email (tomasz@synadia.com) so we can schedule something.

Marquis42 commented 5 months ago

I experienced this same issue last night after a node pool upgrade, I'd be quite Interested in a resolution here. Was anything gleaned from the aforementioned video call (assuming it's happened)?

derekcollison commented 5 months ago

@Marquis42 what server version were you using to do the node pool upgrade?

Any logs you could share?

We did fix a stuck catchup situation hence server version is helpful.

Marquis42 commented 5 months ago

We're running 2.10.14 at the moment.

On Thu, Jun 6, 2024 at 7:32 PM Derek Collison @.***> wrote:

@Marquis42 https://github.com/Marquis42 what server version were you using to do the node pool uphrade?

— Reply to this email directly, view it on GitHub https://github.com/nats-io/nats-server/issues/5205#issuecomment-2153562945, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAG6PH66ZLOHKCNSLWVAGVDZGDWSPAVCNFSM6AAAAABER4ZOHGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNJTGU3DEOJUGU . You are receiving this because you were mentioned.Message ID: @.***>

derekcollison commented 5 months ago

I believe it was fixed in 2.10.16.

We also are staging 2.10.17 as RC1 now if you want to test..

derekcollison commented 5 months ago

https://github.com/nats-io/nats-server/pull/5454

Marquis42 commented 5 months ago

That's great news, thank you! I'm not sure if I can reproduce as it's only happened the one time, but I'll be sure to upgrade at the earliest possible time.

On Thu, Jun 6, 2024 at 7:42 PM Derek Collison @.***> wrote:

5454 https://github.com/nats-io/nats-server/pull/5454

— Reply to this email directly, view it on GitHub https://github.com/nats-io/nats-server/issues/5205#issuecomment-2153575191, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAG6PH64JB5WU7U6E7SZWMLZGDXU5AVCNFSM6AAAAABER4ZOHGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNJTGU3TKMJZGE . You are receiving this because you were mentioned.Message ID: @.***>

bfoxstudio commented 3 months ago

not fixed in 2.10.17 and 2.10.18

5454

derekcollison commented 3 months ago

@bfoxstudio Do you have clear instructions we could use to recreate what you are seeing on our side?

grrrvahrrr commented 3 months ago

@bfoxstudio Do you have clear instructions we could use to recreate what you are seeing on our side?

What we are doing is simple - tested for both docker and k3s:

we are sending a stream of data into KV
check who is the metaleader at the moment and leave this be
choose any other node but the metaleader for the test
we drain the chosen node in docker or deactivate the pod in k3s
wait for the difference in sequences between the offline node and online ones to be around 500
we turn the offline node back on and the difference in sequences remain as seen on the last graph

derekcollison commented 3 months ago

How big is each message? How many different keys / subjects? Does the KV have history? Could you share stream info for this KV? What do you mean by drain the node? What does stream info report for the KV when the offline node is brought back online? (After say 2 mins)

grrrvahrrr commented 3 months ago

How big is each message? 0,33 kb

How many different keys / subjects? 22.5k

Does the KV have history? no

What do you mean by drain the node? we switch node availability to Drain in Portainer to simulate server shutdown. here is more info https://docs.docker.com/engine/swarm/swarm-tutorial/drain-node/

stream info and kv info for the stream with desync

Information for Stream KV_wh-px-egeo-e-shard1-kv created 2024-06-28 15:08:29

              Subjects: $KV.wh-px-egeo-e-shard1-kv.>
              Replicas: 5
               Storage: File

Options:

             Retention: Limits
       Acknowledgments: true
        Discard Policy: New
      Duplicate Window: 2m0s
     Allows Msg Delete: false
          Allows Purge: true
        Allows Rollups: true

Limits:

      Maximum Messages: unlimited
   Maximum Per Subject: 1
         Maximum Bytes: unlimited
           Maximum Age: unlimited
  Maximum Message Size: unlimited
     Maximum Consumers: unlimited

Cluster Information:

                  Name: js_egeo_kv_shard1_infile
                Leader: js_egeo_kv_shard1_infile_node05
               Replica: js_egeo_kv_shard1_infile_node01, current, seen 1.54s ago
               Replica: js_egeo_kv_shard1_infile_node02, current, seen 1.54s ago
               Replica: js_egeo_kv_shard1_infile_node03, current, seen 1.54s ago
               Replica: js_egeo_kv_shard1_infile_node04, current, seen 1.54s ago

State:

              Messages: 22,521
                 Bytes: 7.6 MiB
        First Sequence: 565 @ 2024-06-28 15:09:20 UTC
         Last Sequence: 3,326,244 @ 2024-08-12 11:42:22 UTC
      Deleted Messages: 3,303,159
      Active Consumers: 0
    Number of Subjects: 22,521

    Information for Key-Value Store Bucket wh-px-egeo-e-shard1-kv created 2024-06-28T15:08:29Z

Configuration:

          Bucket Name: wh-px-egeo-e-shard1-kv
         History Kept: 1
        Values Stored: 22,521
   Backing Store Kind: JetStream
          Bucket Size: 7.6 MiB
  Maximum Bucket Size: unlimited
   Maximum Value Size: unlimited
          Maximum Age: unlimited
     JetStream Stream: KV_wh-px-egeo-e-shard1-kv
              Storage: File

Cluster Information:

                 Name: js_egeo_kv_shard1_infile
               Leader: js_egeo_kv_shard1_infile_node05
              Replica: js_egeo_kv_shard1_infile_node01, current, seen 713ms ago
              Replica: js_egeo_kv_shard1_infile_node02, current, seen 712ms ago
              Replica: js_egeo_kv_shard1_infile_node03, current, seen 713ms ago
              Replica: js_egeo_kv_shard1_infile_node04, current, seen 713ms ago

derekcollison commented 3 months ago

@wallyqs could you look into this a bit and see what we can find?

grrrvahrrr commented 2 months ago

@wallyqs Hi! any news on the issue?

katrinwab commented 2 months ago

we tried https://hub.docker.com/layers/synadia/nats-server/nightly-20240819/images/sha256-8d9cb4a80c5db196b60b69fea68d56651a0e33be8394b51b7fc832d8030fe7c7

but still not working

wallyqs commented 2 months ago

Thanks for the information about this issue, we have a test reproduction for it and are currently investigating how to address.

ricardocabralm commented 2 months ago

We solved it for now following the steps here: https://github.com/nats-io/nats-server/issues/5205#issuecomment-2004181431

katrinwab commented 2 months ago

It works for us too, but we would like more stability))

bfoxstudio commented 2 months ago

We solved it for now following the steps here: #5205 (comment)

If 1 server cannot handle the load - reducing the number of replicas can lead to complete data loss . This is not a solution to the problem, but it does help fix the cluster functionality.

katrinwab commented 1 month ago

we caught it again (on 2.10.20)

derekcollison commented 1 month ago

@katrinwab we have some additional protections going into 2.10.21 release that are landing on main but these are for restoring streams on server start or stream restore.

Is this after restart or just during normal operations?

katrinwab commented 1 month ago

we had problems with the network between nodes ( ok, I will try 2.10.21

katrinwab commented 1 month ago

@derekcollison yesterday I updated to 2.10.21. but today I got the problem again

derekcollison commented 1 month ago

@katrinwab would need quite a bit more information.

Are you a Synadia customer? Is this production impacting?

a-alinichenko commented 1 month ago

We have the same problem. The NATS version is 2.10.21. When we restart NATS cluster we have the next logs and the pod doesn't work:

[7] 2024/10/03 10:36:57.943668 [WRN] RAFT [9R4RvVMj - _meta_] Falling behind in health check, commit 59078 != applied 58988
[7] 2024/10/03 10:36:57.943701 [WRN] Healthcheck failed: "JetStream is not current with the meta leader"
[7] 2024/10/03 10:37:07.943402 [WRN] RAFT [9R4RvVMj - _meta_] Falling behind in health check, commit 59108 != applied 58988
[7] 2024/10/03 10:37:07.943423 [WRN] Healthcheck failed: "JetStream is not current with the meta leader"
[7] 2024/10/03 10:37:17.943138 [WRN] RAFT [9R4RvVMj - _meta_] Falling behind in health check, commit 59108 != applied 58988
[7] 2024/10/03 10:37:17.943163 [WRN] Healthcheck failed: "JetStream is not current with the meta leader"
[7] 2024/10/03 10:37:27.943352 [WRN] RAFT [9R4RvVMj - _meta_] Falling behind in health check, commit 59188 != applied 58988
[7] 2024/10/03 10:37:27.943400 [WRN] Healthcheck failed: "JetStream is not current with the meta leader"
[7] 2024/10/03 10:37:37.942694 [WRN] RAFT [9R4RvVMj - _meta_] Falling behind in health check, commit 59298 != applied 58988
[7] 2024/10/03 10:37:37.942806 [WRN] Healthcheck failed: "JetStream is not current with the meta leader"
[7] 2024/10/03 10:37:47.943473 [WRN] RAFT [9R4RvVMj - _meta_] Falling behind in health check, commit 59308 != applied 58988
[7] 2024/10/03 10:37:47.943498 [WRN] Healthcheck failed: "JetStream is not current with the meta leader"
[7] 2024/10/03 10:37:57.943104 [WRN] RAFT [9R4RvVMj - _meta_] Falling behind in health check, commit 59388 != applied 58988
[7] 2024/10/03 10:37:57.943135 [WRN] Healthcheck failed: "JetStream is not current with the meta leader"

Here is the stream info:

Information for Stream ch_queue6 created 2024-09-18 18:54:04

              Subjects: requests6, antibot6
              Replicas: 1
               Storage: File
     Placement Cluster: nats-sessions-api
        Placement Tags: pod:nats-sessions-api-6

Options:

             Retention: WorkQueue
       Acknowledgments: true
        Discard Policy: Old
      Duplicate Window: 10m0s
     Allows Msg Delete: true
          Allows Purge: true
        Allows Rollups: false

Limits:

      Maximum Messages: 100,000,000,000
   Maximum Per Subject: unlimited
         Maximum Bytes: unlimited
           Maximum Age: 7d0h0m0s
  Maximum Message Size: unlimited
     Maximum Consumers: unlimited

Cluster Information:

                  Name: nats-sessions-api
                Leader: nats-sessions-api-6

State:

              Messages: 961,995
                 Bytes: 1.1 GiB
        First Sequence: 3,884,371,968 @ 2024-10-03 13:11:18 UTC
         Last Sequence: 3,885,335,265 @ 2024-10-03 13:21:41 UTC
      Deleted Messages: 1,303
      Active Consumers: 2
    Number of Subjects: 2

It's quite critical for us as we have the problem if we just scale our cluster on production. Or we have the problem if pod starts to migrate to another k8s node. Each NATS cluster restart triggers this problem.

nats-io / nats-server

Nats server pod gets stuck in KV stream catchup after restart #5205

Observed behavior

Expected behavior

Server and client version

Host environment

Steps to reproduce

Environment information

Symptom

Reproduce

Actual behaviour

Expected behaviour

Mitigation

Notes

Stream configuration

Consumer configuration

01 node

02 node

01 STOP DELETE

02

03

config

we create new clear cluster

5454 https://github.com/nats-io/nats-server/pull/5454

5454