search

nats-io / nats-server

High-Performance server for NATS.io, the cloud and edge native messaging system.
https://nats.io
Apache License 2.0
15.56k stars 1.39k forks source link

stream keep failing behind in health check and replication keeps reinitializing #4695

Closed cvoica closed 10 months ago

cvoica commented 11 months ago

Observed behavior

Busy stream doing replication enters a kind of loop after storing a few GB. We can reproduce this and would like to know what we can do anything to make it finish replicating?

Here is what we did until now.

We have a stream with several hundred million messages which is replicated over 3 replicas in AWS. If we edit the stream configuration to make --replicas=2 and then put it back to --replicas=3 the one replica never recovers, keeps writing couple of GB and then starts over, always staying behind.

Looking at the rate of "operations behind" it is decreasing with 40-50.000 per second.

Cluster Information:

                 Name: ccs-prod-ca
               Leader: nats-a
              Replica: nats-b, outdated, seen 0.08s ago, 271,482,651 operations behind
              Replica: nats-d, current, seen 0.08s ago

The stream is currently not rotating but is relatively busy, adding ~10-20.000 messages per second We use i3en.6xlarge which guarantees 25Gbps network We played with max_outstanding_catchup which is currently set to max_outstanding_catchup: 128MB but it does not help with this stream.

This is the warning that keeps repeating, see more logs at the end. [WRN] RAFT [F526168B - S-R3F-iM31rDwh] Falling behind in health check, commit 80055396 != applied 0

When I activate debug I see these messages (some tens of minutes older logs but the same issue)

Oct 24 12:15:47 ip-10-114-221-31.ca-central-1.compute.internal nats-b[6497]: [6497] 2023/10/24 12:15:47.293751 [DBG] RAFT [F526168B - S-R3F-iM31rDwh] Not saving to append entries pending
Oct 24 12:15:47 ip-10-114-221-31.ca-central-1.compute.internal nats-b[6497]: [6497] 2023/10/24 12:15:47.293757 [DBG] RAFT [F526168B - S-R3F-iM31rDwh] Paused, not applying 73988276
Oct 24 12:15:47 ip-10-114-221-31.ca-central-1.compute.internal nats-b[6497]: [6497] 2023/10/24 12:15:47.293770 [DBG] RAFT [F526168B - S-R3F-iM31rDwh] Paused, not applying 73988277
Oct 24 12:15:47 ip-10-114-221-31.ca-central-1.compute.internal nats-b[6497]: [6497] 2023/10/24 12:15:47.293774 [DBG] RAFT [F526168B - S-R3F-iM31rDwh] Paused, not applying 73988278
Oct 24 12:15:47 ip-10-114-221-31.ca-central-1.compute.internal nats-b[6497]: [6497] 2023/10/24 12:15:47.293780 [DBG] RAFT [F526168B - S-R3F-iM31rDwh] Not saving to append entries pending
Oct 24 12:15:47 ip-10-114-221-31.ca-central-1.compute.internal nats-b[6497]: [6497] 2023/10/24 12:15:47.293782 [DBG] RAFT [F526168B - S-R3F-iM31rDwh] Paused, not applying 73988278
Oct 24 12:15:47 ip-10-114-221-31.ca-central-1.compute.internal nats-b[6497]: [6497] 2023/10/24 12:15:47.293787 [DBG] RAFT [F526168B - S-R3F-iM31rDwh] Not saving to append entries pending
Oct 24 12:15:47 ip-10-114-221-31.ca-central-1.compute.internal nats-b[6497]: [6497] 2023/10/24 12:15:47.293790 [DBG] RAFT [F526168B - S-R3F-iM31rDwh] Paused, not applying 73988278
Oct 24 12:15:47 ip-10-114-221-31.ca-central-1.compute.internal nats-b[6497]: [6497] 2023/10/24 12:15:47.293793 [DBG] RAFT [F526168B - S-R3F-iM31rDwh] Paused, not applying 73988279
Oct 24 12:15:47 ip-10-114-221-31.ca-central-1.compute.internal nats-b[6497]: [6497] 2023/10/24 12:15:47.293798 [DBG] RAFT [F526168B - S-R3F-iM31rDwh] Paused, not applying 73988280

Watch on the stream

Tue Oct 24 13:48:21 CEST 2023
Information for Stream crypto-binc created 2023-04-01T09:32:11+02:00

Configuration:

             Subjects: crypto-binc.>
     Acknowledgements: true
            Retention: File - Limits
             Replicas: 3
       Discard Policy: Old
     Duplicate Window: 2m0s
    Allows Msg Delete: true
         Allows Purge: true
       Allows Rollups: false
     Maximum Messages: unlimited
        Maximum Bytes: unlimited
          Maximum Age: 3d0h0m0s
 Maximum Message Size: unlimited
    Maximum Consumers: unlimited

Cluster Information:

                 Name: ccs-prod-ca
               Leader: nats-a
              Replica: nats-b, outdated, seen 0.00s ago, 263,249,981 operations behind
              Replica: nats-d, current, seen 0.00s ago

State:

             Messages: 301,460,415
                Bytes: 155 GiB
             FirstSeq: 37,461,233,104 @ 2023-10-23T15:16:40 UTC
              LastSeq: 37,762,693,518 @ 2023-10-24T11:48:22 UTC
     Active Consumers: 0

╭───────────────────────────────────────────────────────────────────────────────────────────────────────╮
│                                           JetStream Summary                                           │
├─────────┬─────────────┬─────────┬───────────┬──────────┬───────┬────────┬─────────┬─────────┬─────────┤
│ Server  │ Cluster     │ Streams │ Consumers │ Messages │ Bytes │ Memory │ File    │ API Req │ API Err │
├─────────┼─────────────┼─────────┼───────────┼──────────┼───────┼────────┼─────────┼─────────┼─────────┤
│ nats-d* │ ccs-prod-ca │ 0       │ 0         │ 0        │ 0 B   │ 0 B    │ 343 GiB │ 389     │ 0       │
│ nats-b  │ ccs-prod-ca │ 0       │ 0         │ 0        │ 0 B   │ 0 B    │ 205 GiB │ 0       │ 0       │
│ nats-a  │ ccs-prod-ca │ 0       │ 0         │ 0        │ 0 B   │ 0 B    │ 343 GiB │ 1,936   │ 0       │
├─────────┼─────────────┼─────────┼───────────┼──────────┼───────┼────────┼─────────┼─────────┼─────────┤
│         │             │ 0       │ 0         │ 0        │ 0 B   │ 0 B    │ 890 GiB │ 2,325   │ 0       │
╰─────────┴─────────────┴─────────┴───────────┴──────────┴───────┴────────┴─────────┴─────────┴─────────╯

╭───────────────────────────────────────────────────╮
│            RAFT Meta Group Information            │
├────────┬────────┬─────────┬────────┬────────┬─────┤
│ Name   │ Leader │ Current │ Online │ Active │ Lag │
├────────┼────────┼─────────┼────────┼────────┼─────┤
│ nats-a │        │ true    │ true   │ 0.20s  │ 0   │
│ nats-b │        │ true    │ true   │ 0.20s  │ 0   │
│ nats-d │ yes    │ true    │ true   │ 0.00s  │ 0   │
╰────────┴────────┴─────────┴────────┴────────┴─────╯
│ crypto-btrk      │ File    │ 0         │ 174,475     │ 65 MiB  │ 0    │ 0       │ nats-a*, nats-b!, nats-d │
│ crypto-krkn      │ File    │ 0         │ 82,545,212  │ 33 GiB  │ 0    │ 0       │ nats-a*, nats-b!, nats-d │
│ crypto-binc      │ File    │ 0         │ 301,466,733 │ 155 GiB │ 0    │ 0       │ nats-a*, nats-b!, nats-d │

...
...
...some later point, see the date timestamp and the restart of replication, 297,838,468 operations behind
Tue Oct 24 13:56:39 CEST 2023
Information for Stream crypto-binc created 2023-04-01T09:32:11+02:00

Configuration:

             Subjects: crypto-binc.>
     Acknowledgements: true
            Retention: File - Limits
             Replicas: 3
       Discard Policy: Old
     Duplicate Window: 2m0s
    Allows Msg Delete: true
         Allows Purge: true
       Allows Rollups: false
     Maximum Messages: unlimited
        Maximum Bytes: unlimited
          Maximum Age: 3d0h0m0s
 Maximum Message Size: unlimited
    Maximum Consumers: unlimited

Cluster Information:

                 Name: ccs-prod-ca
               Leader: nats-a
              Replica: nats-b, outdated, seen 0.01s ago, 297,838,468 operations behind
              Replica: nats-d, current, seen 0.01s ago

State:

             Messages: 303,046,659
                Bytes: 156 GiB
             FirstSeq: 37,461,233,104 @ 2023-10-23T15:16:40 UTC
              LastSeq: 37,764,279,762 @ 2023-10-24T11:56:40 UTC
     Active Consumers: 0

╭───────────────────────────────────────────────────────────────────────────────────────────────────────╮
│                                           JetStream Summary                                           │
├─────────┬─────────────┬─────────┬───────────┬──────────┬───────┬────────┬─────────┬─────────┬─────────┤
│ Server  │ Cluster     │ Streams │ Consumers │ Messages │ Bytes │ Memory │ File    │ API Req │ API Err │
├─────────┼─────────────┼─────────┼───────────┼──────────┼───────┼────────┼─────────┼─────────┼─────────┤
│ nats-a  │ ccs-prod-ca │ 0       │ 0         │ 0        │ 0 B   │ 0 B    │ 344 GiB │ 2,083   │ 0       │
│ nats-b  │ ccs-prod-ca │ 0       │ 0         │ 0        │ 0 B   │ 0 B    │ 191 GiB │ 0       │ 0       │
│ nats-d* │ ccs-prod-ca │ 0       │ 0         │ 0        │ 0 B   │ 0 B    │ 344 GiB │ 536     │ 0       │
├─────────┼─────────────┼─────────┼───────────┼──────────┼───────┼────────┼─────────┼─────────┼─────────┤
│         │             │ 0       │ 0         │ 0        │ 0 B   │ 0 B    │ 878 GiB │ 2,619   │ 0       │
╰─────────┴─────────────┴─────────┴───────────┴──────────┴───────┴────────┴─────────┴─────────┴─────────╯

╭───────────────────────────────────────────────────╮
│            RAFT Meta Group Information            │
├────────┬────────┬─────────┬────────┬────────┬─────┤
│ Name   │ Leader │ Current │ Online │ Active │ Lag │
├────────┼────────┼─────────┼────────┼────────┼─────┤
│ nats-a │        │ true    │ true   │ 0.64s  │ 0   │
│ nats-b │        │ true    │ true   │ 0.64s  │ 0   │
│ nats-d │ yes    │ true    │ true   │ 0.00s  │ 0   │
╰────────┴────────┴─────────┴────────┴────────┴─────╯
│ crypto-binc      │ File    │ 0         │ 303,054,805 │ 156 GiB │ 0    │ 0       │ nats-a*, nats-b!, nats-d! │

The logs with debug:false are here

Oct 24 13:26:40 ip-10-114-221-31.ca-central-1.compute.internal nats-b[7261]: [7261] 2023/10/24 13:26:40.224687 [WRN] JetStream has not established contact with a meta leader
Oct 24 13:26:40 ip-10-114-221-31.ca-central-1.compute.internal nats-b[7261]: [7261] 2023/10/24 13:26:40.289179 [INF] 10.114.206.164:34366 - rid:415 - Route connection created
Oct 24 13:26:40 ip-10-114-221-31.ca-central-1.compute.internal nats-b[7261]: [7261] 2023/10/24 13:26:40.289458 [INF] 10.114.206.164:34366 - rid:415 - Router connection closed: Duplicate Route
Oct 24 13:26:40 ip-10-114-221-31.ca-central-1.compute.internal nats-b[7261]: [7261] 2023/10/24 13:26:40.380724 [INF] 10.114.235.205:44490 - rid:416 - Route connection created
Oct 24 13:26:40 ip-10-114-221-31.ca-central-1.compute.internal nats-b[7261]: [7261] 2023/10/24 13:26:40.380927 [INF] 10.114.235.205:44490 - rid:416 - Router connection closed: Duplicate Route
Oct 24 13:26:40 ip-10-114-221-31.ca-central-1.compute.internal nats-b[7261]: [7261] 2023/10/24 13:26:40.486293 [INF] JetStream cluster new metadata leader: nats-d/ccs-prod-ca
Oct 24 13:26:47 ip-10-114-221-31.ca-central-1.compute.internal nats-b[7261]: [7261] 2023/10/24 13:26:47.502114 [WRN] RAFT [F526168B - S-R3F-iM31rDwh] 15000 append entries pending
Oct 24 13:26:50 ip-10-114-221-31.ca-central-1.compute.internal nats-b[7261]: [7261] 2023/10/24 13:26:50.384904 [WRN] RAFT [F526168B - S-R3F-iM31rDwh] 20000 append entries pending
Oct 24 13:28:39 ip-10-114-221-31.ca-central-1.compute.internal nats-b[7261]: [7261] 2023/10/24 13:28:39.871231 [WRN] RAFT [F526168B - S-R3F-iM31rDwh] Falling behind in health check, commit 80055396 != applied 0
Oct 24 13:28:39 ip-10-114-221-31.ca-central-1.compute.internal nats-b[7261]: [7261] 2023/10/24 13:28:39.871250 [WRN] JetStream stream '$G > crypto-binc' is not current
Oct 24 13:30:39 ip-10-114-221-31.ca-central-1.compute.internal nats-b[7261]: [7261] 2023/10/24 13:30:39.870652 [WRN] RAFT [F526168B - S-R3F-iM31rDwh] Falling behind in health check, commit 80055396 != applied 0
Oct 24 13:30:39 ip-10-114-221-31.ca-central-1.compute.internal nats-b[7261]: [7261] 2023/10/24 13:30:39.870669 [WRN] JetStream stream '$G > crypto-binc' is not current
Oct 24 13:32:39 ip-10-114-221-31.ca-central-1.compute.internal nats-b[7261]: [7261] 2023/10/24 13:32:39.870849 [WRN] RAFT [F526168B - S-R3F-iM31rDwh] Falling behind in health check, commit 80055396 != applied 0
Oct 24 13:32:39 ip-10-114-221-31.ca-central-1.compute.internal nats-b[7261]: [7261] 2023/10/24 13:32:39.870864 [WRN] JetStream stream '$G > crypto-binc' is not current
Oct 24 13:34:39 ip-10-114-221-31.ca-central-1.compute.internal nats-b[7261]: [7261] 2023/10/24 13:34:39.870717 [WRN] RAFT [F526168B - S-R3F-iM31rDwh] Falling behind in health check, commit 80055396 != applied 0
Oct 24 13:34:39 ip-10-114-221-31.ca-central-1.compute.internal nats-b[7261]: [7261] 2023/10/24 13:34:39.870733 [WRN] JetStream stream '$G > crypto-binc' is not current
Oct 24 13:36:39 ip-10-114-221-31.ca-central-1.compute.internal nats-b[7261]: [7261] 2023/10/24 13:36:39.871057 [WRN] RAFT [F526168B - S-R3F-iM31rDwh] Falling behind in health check, commit 80055396 != applied 0
Oct 24 13:36:39 ip-10-114-221-31.ca-central-1.compute.internal nats-b[7261]: [7261] 2023/10/24 13:36:39.871075 [WRN] JetStream stream '$G > crypto-binc' is not current
Oct 24 13:38:39 ip-10-114-221-31.ca-central-1.compute.internal nats-b[7261]: [7261] 2023/10/24 13:38:39.870984 [WRN] RAFT [F526168B - S-R3F-iM31rDwh] Falling behind in health check, commit 80055396 != applied 0
Oct 24 13:38:39 ip-10-114-221-31.ca-central-1.compute.internal nats-b[7261]: [7261] 2023/10/24 13:38:39.871006 [WRN] JetStream stream '$G > crypto-binc' is not current
Oct 24 13:40:39 ip-10-114-221-31.ca-central-1.compute.internal nats-b[7261]: [7261] 2023/10/24 13:40:39.870873 [WRN] RAFT [F526168B - S-R3F-iM31rDwh] Falling behind in health check, commit 80055396 != applied 0
Oct 24 13:40:39 ip-10-114-221-31.ca-central-1.compute.internal nats-b[7261]: [7261] 2023/10/24 13:40:39.870888 [WRN] JetStream stream '$G > crypto-binc' is not current
Oct 24 13:42:39 ip-10-114-221-31.ca-central-1.compute.internal nats-b[7261]: [7261] 2023/10/24 13:42:39.871048 [WRN] RAFT [F526168B - S-R3F-iM31rDwh] Falling behind in health check, commit 80055396 != applied 0
Oct 24 13:42:39 ip-10-114-221-31.ca-central-1.compute.internal nats-b[7261]: [7261] 2023/10/24 13:42:39.871070 [WRN] JetStream stream '$G > crypto-binc' is not current
Oct 24 13:44:39 ip-10-114-221-31.ca-central-1.compute.internal nats-b[7261]: [7261] 2023/10/24 13:44:39.871243 [WRN] RAFT [F526168B - S-R3F-iM31rDwh] Falling behind in health check, commit 80055396 != applied 0
Oct 24 13:44:39 ip-10-114-221-31.ca-central-1.compute.internal nats-b[7261]: [7261] 2023/10/24 13:44:39.871261 [WRN] JetStream stream '$G > crypto-binc' is not current
Oct 24 13:46:39 ip-10-114-221-31.ca-central-1.compute.internal nats-b[7261]: [7261] 2023/10/24 13:46:39.870910 [WRN] RAFT [F526168B - S-R3F-iM31rDwh] Falling behind in health check, commit 80055396 != applied 0
Oct 24 13:46:39 ip-10-114-221-31.ca-central-1.compute.internal nats-b[7261]: [7261] 2023/10/24 13:46:39.870944 [WRN] JetStream stream '$G > crypto-binc' is not current
Oct 24 13:48:39 ip-10-114-221-31.ca-central-1.compute.internal nats-b[7261]: [7261] 2023/10/24 13:48:39.870688 [WRN] RAFT [F526168B - S-R3F-iM31rDwh] Falling behind in health check, commit 80055396 != applied 0
Oct 24 13:48:39 ip-10-114-221-31.ca-central-1.compute.internal nats-b[7261]: [7261] 2023/10/24 13:48:39.870707 [WRN] JetStream stream '$G > crypto-binc' is not current
Oct 24 13:50:39 ip-10-114-221-31.ca-central-1.compute.internal nats-b[7261]: [7261] 2023/10/24 13:50:39.870741 [WRN] RAFT [F526168B - S-R3F-iM31rDwh] Falling behind in health check, commit 80055396 != applied 0
Oct 24 13:50:39 ip-10-114-221-31.ca-central-1.compute.internal nats-b[7261]: [7261] 2023/10/24 13:50:39.870759 [WRN] JetStream stream '$G > crypto-binc' is not current
Oct 24 13:52:39 ip-10-114-221-31.ca-central-1.compute.internal nats-b[7261]: [7261] 2023/10/24 13:52:39.870650 [WRN] RAFT [F526168B - S-R3F-iM31rDwh] Falling behind in health check, commit 80055396 != applied 0
Oct 24 13:52:39 ip-10-114-221-31.ca-central-1.compute.internal nats-b[7261]: [7261] 2023/10/24 13:52:39.870678 [WRN] JetStream stream '$G > crypto-binc' is not current
Oct 24 13:54:39 ip-10-114-221-31.ca-central-1.compute.internal nats-b[7261]: [7261] 2023/10/24 13:54:39.871004 [WRN] RAFT [F526168B - S-R3F-iM31rDwh] Falling behind in health check, commit 80055396 != applied 0
Oct 24 13:54:39 ip-10-114-221-31.ca-central-1.compute.internal nats-b[7261]: [7261] 2023/10/24 13:54:39.871020 [WRN] JetStream stream '$G > crypto-binc' is not current
Oct 24 13:55:05 ip-10-114-221-31.ca-central-1.compute.internal nats-b[7261]: [7261] 2023/10/24 13:55:05.748728 [WRN] Catchup for stream '$G > crypto-binc' stalled
Oct 24 13:55:55 ip-10-114-221-31.ca-central-1.compute.internal nats-b[7261]: [7261] 2023/10/24 13:55:55.044529 [WRN] Catchup for stream '$G > crypto-binc' stalled
Oct 24 13:56:03 ip-10-114-221-31.ca-central-1.compute.internal nats-b[7261]: [7261] 2023/10/24 13:56:03.790907 [WRN] Error applying entries to '$G > crypto-binc': first sequence mismatch
Oct 24 13:56:03 ip-10-114-221-31.ca-central-1.compute.internal nats-b[7261]: [7261] 2023/10/24 13:56:03.790975 [WRN] Internal subscription on "$JS.API.STREAM.INFO.crypto-binc" took too long: 8.043710154s
Oct 24 13:56:03 ip-10-114-221-31.ca-central-1.compute.internal nats-b[7261]: [7261] 2023/10/24 13:56:03.790986 [WRN] Internal subscription on "$JS.API.STREAM.INFO.crypto-binc" took too long: 8.043763474s
Oct 24 13:56:03 ip-10-114-221-31.ca-central-1.compute.internal nats-b[7261]: [7261] 2023/10/24 13:56:03.791009 [WRN] Internal subscription on "$JS.API.STREAM.INFO.crypto-binc" took too long: 8.043822619s
Oct 24 13:56:03 ip-10-114-221-31.ca-central-1.compute.internal nats-b[7261]: [7261] 2023/10/24 13:56:03.791015 [WRN] 10.115.107.254:60732 - cid:954 - Readloop processing time: 8.043862777s
Oct 24 13:56:06 ip-10-114-221-31.ca-central-1.compute.internal nats-b[7261]: [7261] 2023/10/24 13:56:06.351464 [WRN] Resetting stream cluster state for '$G > crypto-binc'
Oct 24 13:56:12 ip-10-114-221-31.ca-central-1.compute.internal nats-b[7261]: [7261] 2023/10/24 13:56:12.717580 [WRN] RAFT [F526168B - S-R3F-iM31rDwh] 15000 append entries pending
Oct 24 13:56:16 ip-10-114-221-31.ca-central-1.compute.internal nats-b[7261]: [7261] 2023/10/24 13:56:16.528014 [WRN] RAFT [F526168B - S-R3F-iM31rDwh] 20000 append entries pending
Oct 24 13:56:39 ip-10-114-221-31.ca-central-1.compute.internal nats-b[7261]: [7261] 2023/10/24 13:56:39.872893 [WRN] RAFT [F526168B - S-R3F-iM31rDwh] Falling behind in health check, commit 83439056 != applied 0
Oct 24 13:56:39 ip-10-114-221-31.ca-central-1.compute.internal nats-b[7261]: [7261] 2023/10/24 13:56:39.872910 [WRN] JetStream stream '$G > crypto-binc' is not current
Oct 24 13:56:45 ip-10-114-221-31.ca-central-1.compute.internal nats-b[7261]: [7261] 2023/10/24 13:56:45.280726 [WRN] Catchup for stream '$G > crypto-binc' stalled
Oct 24 13:57:32 ip-10-114-221-31.ca-central-1.compute.internal nats-b[7261]: [7261] 2023/10/24 13:57:32.523850 [WRN] Catchup for stream '$G > crypto-binc' stalled
Oct 24 13:57:32 ip-10-114-221-31.ca-central-1.compute.internal nats-b[7261]: [7261] 2023/10/24 13:57:32.861986 [WRN] Error applying entries to '$G > crypto-binc': first sequence mismatch
Oct 24 13:57:33 ip-10-114-221-31.ca-central-1.compute.internal nats-b[7261]: [7261] 2023/10/24 13:57:33.292750 [WRN] Resetting stream cluster state for '$G > crypto-binc'
Oct 24 13:57:41 ip-10-114-221-31.ca-central-1.compute.internal nats-b[7261]: [7261] 2023/10/24 13:57:41.726744 [WRN] RAFT [F526168B - S-R3F-iM31rDwh] 15000 append entries pending
Oct 24 13:57:47 ip-10-114-221-31.ca-central-1.compute.internal nats-b[7261]: [7261] 2023/10/24 13:57:47.663610 [WRN] RAFT [F526168B - S-R3F-iM31rDwh] 20000 append entries pending
Oct 24 13:58:11 ip-10-114-221-31.ca-central-1.compute.internal nats-b[7261]: [7261] 2023/10/24 13:58:11.514441 [WRN] Catchup for stream '$G > crypto-binc' stalled
Oct 24 13:58:38 ip-10-114-221-31.ca-central-1.compute.internal nats-b[7261]: [7261] 2023/10/24 13:58:38.790320 [WRN] Catchup for stream '$G > crypto-binc' stalled
Oct 24 13:58:39 ip-10-114-221-31.ca-central-1.compute.internal nats-b[7261]: [7261] 2023/10/24 13:58:39.012811 [WRN] Error applying entries to '$G > crypto-binc': first sequence mismatch
Oct 24 13:58:39 ip-10-114-221-31.ca-central-1.compute.internal nats-b[7261]: [7261] 2023/10/24 13:58:39.450353 [WRN] Resetting stream cluster state for '$G > crypto-binc'
Oct 24 13:58:39 ip-10-114-221-31.ca-central-1.compute.internal nats-b[7261]: [7261] 2023/10/24 13:58:39.870729 [WRN] RAFT [F526168B - S-R3F-iM31rDwh] Falling behind in health check, commit 83670022 != applied 0
Oct 24 13:58:39 ip-10-114-221-31.ca-central-1.compute.internal nats-b[7261]: [7261] 2023/10/24 13:58:39.870749 [WRN] JetStream stream '$G > crypto-binc' is not current
Oct 24 13:58:43 ip-10-114-221-31.ca-central-1.compute.internal nats-b[7261]: [7261] 2023/10/24 13:58:43.875211 [WRN] RAFT [F526168B - S-R3F-iM31rDwh] 15000 append entries pending
Oct 24 13:58:46 ip-10-114-221-31.ca-central-1.compute.internal nats-b[7261]: [7261] 2023/10/24 13:58:46.706861 [WRN] RAFT [F526168B - S-R3F-iM31rDwh] 20000 append entries pending

Expected behavior

The stream should eventually make progress with replication and not keep starting from the beginning after couple of GB

Server and client version

nats-server: v2.9.21 $ nats --version 0.0.28

Host environment

i3en.6xlarge

Steps to reproduce

edit the stream configuration to make --replicas=2 and then put it back to --replicas=3 new replica never recovers

cvoica commented 11 months ago

This is the view from the file system during such a "loop"

[root@ip-10-114-221-31 ~]# date; du -khs /data/jetstream/\$G/streams/crypto-binc/* ; ls -l /data/jetstream/\$G/streams/crypto-binc/msgs/ | grep blk | wc -l ; ls -l /data/jetstream/\$G/streams/crypto-binc/msgs/ | grep blk | tail
Tue Oct 24 13:55:42 CEST 2023
4.0K    /data/jetstream/$G/streams/crypto-binc/meta.inf
4.0K    /data/jetstream/$G/streams/crypto-binc/meta.sum
23G     /data/jetstream/$G/streams/crypto-binc/msgs
4.0K    /data/jetstream/$G/streams/crypto-binc/obs
11659
-rw-r----- 1 service_user service_user 2097051 Oct 24 13:55 9993.blk
-rw-r----- 1 service_user service_user 2096807 Oct 24 13:55 9994.blk
-rw-r----- 1 service_user service_user 2096918 Oct 24 13:55 9995.blk
-rw-r----- 1 service_user service_user 2096874 Oct 24 13:55 9996.blk
-rw-r----- 1 service_user service_user 2097064 Oct 24 13:55 9997.blk
-rw-r----- 1 service_user service_user 2096342 Oct 24 13:55 9998.blk
-rw-r----- 1 service_user service_user 2095801 Oct 24 13:55 9999.blk
-rw-r----- 1 service_user service_user 2096922 Oct 24 13:24 999.blk
-rw-r----- 1 service_user service_user 2097106 Oct 24 13:24 99.blk
-rw-r----- 1 service_user service_user 2096772 Oct 24 13:24 9.blk
[root@ip-10-114-221-31 ~]# date; du -khs /data/jetstream/\$G/streams/crypto-binc/* ; ls -l /data/jetstream/\$G/streams/crypto-binc/msgs/ | grep blk | wc -l ; ls -lt /data/jetstream/\$G/streams/crypto-binc/msgs/ | grep blk | tail
Tue Oct 24 13:55:53 CEST 2023
4.0K    /data/jetstream/$G/streams/crypto-binc/meta.inf
4.0K    /data/jetstream/$G/streams/crypto-binc/meta.sum
24G     /data/jetstream/$G/streams/crypto-binc/msgs
4.0K    /data/jetstream/$G/streams/crypto-binc/obs
11745
-rw-r----- 1 service_user service_user 2096776 Oct 24 13:24 10.blk
-rw-r----- 1 service_user service_user 2096772 Oct 24 13:24 9.blk
-rw-r----- 1 service_user service_user 2097085 Oct 24 13:24 8.blk
-rw-r----- 1 service_user service_user 2096984 Oct 24 13:24 7.blk
-rw-r----- 1 service_user service_user 2096526 Oct 24 13:24 6.blk
-rw-r----- 1 service_user service_user 2097079 Oct 24 13:24 5.blk
-rw-r----- 1 service_user service_user 2096785 Oct 24 13:24 4.blk
-rw-r----- 1 service_user service_user 2096992 Oct 24 13:24 3.blk
-rw-r----- 1 service_user service_user 2096585 Oct 24 13:24 2.blk
-rw-r----- 1 service_user service_user 2096581 Oct 24 13:24 1.blk
[root@ip-10-114-221-31 ~]# date; du -khs /data/jetstream/\$G/streams/crypto-binc/* ; ls -l /data/jetstream/\$G/streams/crypto-binc/msgs/ | grep blk | wc -l ; ls -ltr /data/jetstream/\$G/streams/crypto-binc/msgs/ | grep blk | tail
Tue Oct 24 13:56:04 CEST 2023
4.0K    /data/jetstream/$G/streams/crypto-binc/meta.inf
4.0K    /data/jetstream/$G/streams/crypto-binc/meta.sum
8.0K    /data/jetstream/$G/streams/crypto-binc/msgs
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/33.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/56.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/101.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/142.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/372.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/384.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/390.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/395.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/440.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/460.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/500.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/520.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/541.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/561.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/563.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/583.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/595.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/608.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/633.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/665.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/678.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/683.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/742.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/816.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/847.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/867.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/883.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/884.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/906.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/959.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/979.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/990.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/1015.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/1018.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/1032.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/1041.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/1042.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/1081.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/1127.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/1133.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/1150.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/1235.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/1263.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/1276.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/1300.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/1304.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/1319.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/1327.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/1337.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/1356.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/1389.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/1410.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/1426.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/1441.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/1469.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/1531.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/1561.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/1582.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/1602.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/1633.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/1643.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/1647.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/1687.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/1690.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/1716.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/1719.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/1722.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/1732.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/1744.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/1765.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/1815.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/1855.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/1906.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/1936.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/1971.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/1991.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/1994.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/2000.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/2101.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/2113.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/2148.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/2166.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/2169.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/2179.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/2198.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/2223.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/2233.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/2239.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/2240.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/2301.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/2333.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/2352.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/2356.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/2357.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/2375.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/2394.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/2396.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/2405.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/2432.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/2435.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/2437.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/2449.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/2453.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/2459.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/2547.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/2549.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/2552.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/2581.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/2618.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/2633.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/2671.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/2685.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/2706.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/2717.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/2728.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/2743.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/2782.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/2802.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/2812.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/2813.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/2822.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/2823.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/2848.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/2850.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/2851.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/2864.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/2886.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/2886.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/2888.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/2910.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/2917.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/2933.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/2952.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/2957.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/2987.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/2996.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/3004.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/3027.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/3028.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/3033.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/3046.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/3049.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/3058.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/3125.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/3131.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/3187.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/3202.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/3205.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/3210.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/3250.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/3262.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/3265.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/3266.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/3274.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/3284.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/3291.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/3299.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/3305.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/3322.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/3326.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/3339.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/3360.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/3379.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/3382.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/3394.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/3401.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/3408.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/3408.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/3424.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/3469.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/3472.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/3488.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/3493.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/3504.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/3522.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/3523.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/3531.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/3547.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/3564.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/3573.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/3575.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/3580.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/3604.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/3615.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/3627.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/3643.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/3646.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/3664.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/3696.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/3699.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/3704.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/3709.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/3735.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/3751.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/3790.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/3802.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/3805.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/3809.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/3818.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/3833.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/3836.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/3838.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/3848.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/3865.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/3867.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/3873.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/3902.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/3914.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/3935.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/3970.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/3982.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/3999.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/4018.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/4023.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/4024.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/4029.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/4036.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/4061.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/4087.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/4091.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/4091.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/4110.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/4115.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/4125.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/4127.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/4129.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/4150.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/4156.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/4166.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/4179.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/4181.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/4188.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/4198.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/4199.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/4205.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/4206.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/4220.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/4237.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/4239.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/4240.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/4261.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/4271.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/4272.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/4274.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/4286.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/4308.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/4309.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/4326.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/4330.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/4332.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/4337.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/4343.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/4359.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/4364.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/4370.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/4404.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/4413.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/4416.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/4435.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/4457.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/4463.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/4469.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/4477.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/4482.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/4491.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/4501.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/4505.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/4518.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/4522.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/4531.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/4541.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/4550.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/4560.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/4589.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/4597.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/4648.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/4655.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/4663.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/4668.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/4689.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/4705.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/4718.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/4740.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/4750.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/4768.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/4768.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/4769.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/4775.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/4789.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/4795.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/4817.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/4821.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/4837.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/4853.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/4898.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/4900.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/4925.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/4948.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/4962.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/4975.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/4988.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/4992.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/4998.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/5000.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/5002.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/5011.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/5016.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/5026.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/5059.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/5063.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/5108.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/5112.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/5116.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/5117.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/5129.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/5133.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/5140.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/5145.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/5165.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/5168.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/5181.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/5190.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/5221.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/5232.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/5233.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/5238.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/5240.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/5246.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/5249.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/5253.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/5257.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/5259.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/5272.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/5273.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/5276.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/5287.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/5291.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/5331.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/5336.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/5353.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/5355.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/5359.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/5363.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/5366.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/5377.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/5393.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/5397.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/5397.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/5408.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/5419.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/5475.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/5486.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/5491.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/5494.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/5520.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/5521.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/5531.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/5532.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/5535.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/5540.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/5552.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/5555.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/5581.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/5582.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/5585.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/5605.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/5613.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/5620.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/5626.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/5628.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/5630.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/5633.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/5638.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/5646.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/5657.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/5672.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/5699.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/5705.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/5725.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/5732.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/5743.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/5753.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/5796.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/5798.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/5806.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/5809.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/5815.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/5851.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/5875.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/5886.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/5898.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/5903.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/5906.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/5919.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/5921.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/5931.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/5936.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/5944.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/5955.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/5963.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/5964.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/5991.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6003.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6006.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6011.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6019.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6025.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6028.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6058.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6058.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6059.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6059.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6077.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6083.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6086.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6089.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6100.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6100.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6105.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6107.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6107.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6150.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6193.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6199.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6202.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6205.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6209.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6212.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6216.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6220.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6222.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6226.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6242.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6250.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6256.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6263.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6275.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6280.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6302.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6303.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6312.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6315.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6320.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6326.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6351.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6363.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6372.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6375.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6395.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6398.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6399.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6412.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6413.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6413.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6437.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6443.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6448.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6449.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6453.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6454.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6455.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6461.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6463.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6476.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6496.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6503.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6528.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6555.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6559.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6570.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6593.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6605.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6634.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6636.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6647.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6647.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6653.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6666.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6684.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6685.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6689.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6690.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6698.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6716.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6718.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6732.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6735.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6764.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6772.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6775.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6779.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6791.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6805.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6809.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6830.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6837.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6843.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6849.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6853.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6859.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6861.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6865.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6872.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6877.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6884.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6886.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6887.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6897.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6912.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6913.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6935.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6938.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6948.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6954.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6961.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6962.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6969.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6975.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6980.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6987.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/6997.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7004.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7007.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7052.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7060.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7061.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7068.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7083.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7085.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7090.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7097.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7101.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7103.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7125.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7139.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7160.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7166.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7174.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7181.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7190.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7194.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7203.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7206.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7206.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7226.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7232.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7240.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7242.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7245.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7245.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7259.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7264.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7270.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7277.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7294.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7298.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7328.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7329.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7331.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7333.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7344.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7345.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7354.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7367.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7394.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7402.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7421.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7423.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7426.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7427.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7430.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7449.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7481.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7484.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7492.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7499.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7502.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7504.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7508.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7517.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7541.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7572.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7575.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7582.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7594.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7612.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7626.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7629.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7634.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7637.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7643.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7648.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7649.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7666.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7667.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7686.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7689.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7693.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7707.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7714.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7730.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7730.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7734.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7751.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7768.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7772.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7772.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7789.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7795.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7799.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7809.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7809.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7815.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7817.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7819.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7843.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7848.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7849.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7855.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7863.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7869.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7886.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7887.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7889.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7895.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7910.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7912.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7925.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7927.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7927.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7928.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7931.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7932.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7935.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7936.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7941.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7958.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7959.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7970.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7972.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7973.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7978.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7986.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/7988.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8000.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8008.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8019.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8033.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8037.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8039.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8067.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8072.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8073.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8088.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8089.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8090.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8092.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8094.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8095.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8104.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8110.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8122.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8128.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8135.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8135.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8136.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8145.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8159.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8169.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8194.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8216.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8229.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8231.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8233.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8236.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8240.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8244.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8260.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8264.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8269.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8271.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8274.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8274.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8297.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8308.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8309.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8312.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8323.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8327.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8327.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8329.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8330.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8333.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8337.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8354.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8354.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8357.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8364.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8369.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8376.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8385.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8391.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8395.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8410.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8410.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8428.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8438.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8444.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8445.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8448.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8449.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8464.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8464.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8465.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8474.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8478.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8487.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8491.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8493.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8503.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8509.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8533.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8535.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8542.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8544.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8557.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8560.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8561.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8566.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8573.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8586.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8588.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8591.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8596.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8604.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8620.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8621.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8622.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8627.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8628.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8628.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8630.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8633.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8638.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8645.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8649.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8676.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8678.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8682.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8695.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8697.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8698.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8701.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8703.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8712.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8722.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8731.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8732.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8732.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8733.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8740.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8756.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8784.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8790.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8793.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8794.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8795.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8799.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8799.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8801.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8808.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8808.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8813.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8823.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8825.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8863.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8867.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8870.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8910.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8914.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8917.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8922.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8924.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8928.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8929.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8935.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8946.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8951.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8959.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8961.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8962.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8965.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8975.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8975.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8980.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8985.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8986.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/8995.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9002.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9003.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9007.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9008.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9015.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9027.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9034.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9035.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9036.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9041.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9046.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9051.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9053.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9053.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9054.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9055.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9056.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9060.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9061.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9063.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9069.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9072.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9073.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9079.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9085.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9085.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9090.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9093.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9095.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9107.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9111.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9124.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9127.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9136.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9148.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9155.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9157.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9158.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9161.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9168.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9172.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9173.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9177.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9185.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9197.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9197.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9198.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9207.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9217.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9218.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9223.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9225.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9239.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9252.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9252.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9254.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9255.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9262.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9267.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9269.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9271.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9272.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9275.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9277.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9279.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9297.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9301.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9302.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9314.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9315.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9316.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9316.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9326.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9326.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9340.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9345.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9348.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9356.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9357.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9361.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9368.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9368.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9373.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9375.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9384.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9400.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9413.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9418.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9427.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9427.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9433.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9452.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9454.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9465.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9467.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9501.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9505.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9508.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9518.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9528.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9536.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9542.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9547.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9554.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9570.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9584.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9587.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9604.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9608.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9609.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9624.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9627.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9630.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9633.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9660.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9671.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9674.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9680.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9680.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9681.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9685.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9687.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9691.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9699.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9702.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9707.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9717.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9742.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9745.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9746.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9747.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9755.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9762.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9770.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9771.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9785.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9789.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9791.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9794.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9814.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9817.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9818.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9819.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9827.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9828.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9835.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9851.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9857.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9860.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9864.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9870.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9877.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9882.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9884.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9887.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9894.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9895.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9898.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9901.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9916.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9930.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9931.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9932.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9933.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9940.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9944.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9946.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9957.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9973.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9983.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/9994.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10001.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10002.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10006.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10008.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10011.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10016.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10018.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10028.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10035.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10042.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10046.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10052.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10061.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10063.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10063.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10072.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10079.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10081.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10088.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10100.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10101.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10107.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10113.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10114.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10128.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10132.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10134.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10137.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10138.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10162.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10170.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10177.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10181.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10192.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10206.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10210.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10212.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10212.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10216.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10222.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10223.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10223.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10226.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10227.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10228.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10232.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10238.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10253.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10254.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10270.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10270.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10271.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10278.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10299.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10303.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10306.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10312.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10315.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10318.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10331.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10335.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10343.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10345.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10349.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10350.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10352.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10355.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10364.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10365.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10368.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10369.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10373.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10382.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10387.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10389.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10390.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10413.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10418.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10426.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10427.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10429.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10432.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10433.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10435.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10437.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10438.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10440.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10443.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10450.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10460.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10466.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10484.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10491.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10503.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10504.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10505.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10507.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10509.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10510.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10516.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10521.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10525.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10526.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10528.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10529.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10535.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10542.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10565.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10573.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10577.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10579.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10590.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10591.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10594.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10597.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10608.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10610.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10614.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10619.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10633.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10655.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10656.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10663.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10669.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10680.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10692.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10696.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10700.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10706.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10708.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10719.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10721.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10736.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10740.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10741.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10745.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10748.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10749.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10752.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10756.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10761.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10766.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10777.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10783.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10784.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10785.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10800.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10804.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10808.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10810.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10810.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10827.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10830.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10835.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10837.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10842.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10850.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10854.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10859.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10872.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10874.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10877.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10877.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10896.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10897.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10900.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10903.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10912.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10914.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10917.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10920.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10924.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10928.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10932.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10934.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10940.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10943.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10946.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10949.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10962.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10976.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10979.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10981.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10987.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/10992.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11003.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11010.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11023.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11027.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11036.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11049.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11063.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11068.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11080.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11092.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11099.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11100.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11105.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11111.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11112.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11121.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11122.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11123.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11124.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11128.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11129.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11135.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11143.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11144.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11148.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11152.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11153.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11158.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11159.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11160.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11168.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11181.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11183.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11192.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11196.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11200.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11207.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11208.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11209.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11210.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11220.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11223.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11232.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11233.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11241.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11245.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11270.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11272.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11276.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11278.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11282.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11284.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11286.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11287.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11289.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11290.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11294.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11300.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11304.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11309.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11309.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11309.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11312.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11315.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11316.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11317.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11326.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11331.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11336.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11344.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11344.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11356.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11359.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11362.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11365.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11376.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11385.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11386.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11388.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11393.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11393.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11396.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11400.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11403.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11425.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11426.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11437.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11445.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11450.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11453.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11457.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11471.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11472.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11473.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11475.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11481.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11483.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11491.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11503.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11507.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11512.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11515.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11521.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11522.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11524.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11525.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11525.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11527.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11537.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11544.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11556.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11556.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11557.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11563.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11564.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11571.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11574.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11578.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11583.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11590.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11592.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11593.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11597.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11597.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11600.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11606.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11607.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11609.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11617.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11623.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11623.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11643.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11654.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11657.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11673.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11675.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11675.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11678.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11679.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11681.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11683.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11684.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11688.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11691.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11692.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11693.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11696.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11697.idx’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11699.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11709.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11722.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11742.blk’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11743.fss’: No such file or directory
du: cannot access ‘/data/jetstream/$G/streams/crypto-binc/__msgs__/11745.blk’: No such file or directory
17G     /data/jetstream/$G/streams/crypto-binc/__msgs__
4.0K    /data/jetstream/$G/streams/crypto-binc/obs
1
-rw-r----- 1 service_user service_user  0 Oct 24 13:56 1.blk
[root@ip-10-114-221-31 ~]# date; du -khs /data/jetstream/\$G/streams/crypto-binc/* ; ls -l /data/jetstream/\$G/streams/crypto-binc/msgs/ | grep blk | wc -l ; ls -ltr /data/jetstream/\$G/streams/crypto-binc/msgs/ | grep blk | tail
Tue Oct 24 13:56:19 CEST 2023
4.0K    /data/jetstream/$G/streams/crypto-binc/meta.inf
4.0K    /data/jetstream/$G/streams/crypto-binc/meta.sum
1.3G    /data/jetstream/$G/streams/crypto-binc/msgs
4.0K    /data/jetstream/$G/streams/crypto-binc/obs
620
-rw-r----- 1 service_user service_user 2097033 Oct 24 13:56 613.blk
-rw-r----- 1 service_user service_user 2096300 Oct 24 13:56 614.blk
-rw-r----- 1 service_user service_user 2097126 Oct 24 13:56 615.blk
-rw-r----- 1 service_user service_user 2097067 Oct 24 13:56 616.blk
-rw-r----- 1 service_user service_user 2097067 Oct 24 13:56 617.blk
-rw-r----- 1 service_user service_user 2096375 Oct 24 13:56 618.blk
-rw-r----- 1 service_user service_user 2095894 Oct 24 13:56 619.blk
-rw-r----- 1 service_user service_user 2097044 Oct 24 13:56 620.blk
-rw-r----- 1 service_user service_user 1979947 Oct 24 13:56 621.blk
-rw-r----- 1 service_user service_user  212992 Oct 24 13:56 622.blk
[root@ip-10-114-221-31 ~]# date; du -khs /data/jetstream/\$G/streams/crypto-binc/* ; ls -l /data/jetstream/\$G/streams/crypto-binc/msgs/ | grep blk | wc -l ; ls -ltr /data/jetstream/\$G/streams/crypto-binc/msgs/ | grep blk | tail
Tue Oct 24 13:56:32 CEST 2023
4.0K    /data/jetstream/$G/streams/crypto-binc/meta.inf
4.0K    /data/jetstream/$G/streams/crypto-binc/meta.sum
2.5G    /data/jetstream/$G/streams/crypto-binc/msgs
4.0K    /data/jetstream/$G/streams/crypto-binc/obs
1235
-rw-r----- 1 service_user service_user 2097062 Oct 24 13:56 1229.blk
-rw-r----- 1 service_user service_user 2097111 Oct 24 13:56 1230.blk
-rw-r----- 1 service_user service_user 2096733 Oct 24 13:56 1231.blk
-rw-r----- 1 service_user service_user 2095871 Oct 24 13:56 1232.blk
-rw-r----- 1 service_user service_user 2096975 Oct 24 13:56 1233.blk
-rw-r----- 1 service_user service_user 2097036 Oct 24 13:56 1234.blk
-rw-r----- 1 service_user service_user 2097111 Oct 24 13:56 1235.blk
-rw-r----- 1 service_user service_user 2096716 Oct 24 13:56 1236.blk
-rw-r----- 1 service_user service_user 2096736 Oct 24 13:56 1237.blk
-rw-r----- 1 service_user service_user  617585 Oct 24 13:56 1238.blk
[root@ip-10-114-221-31 ~]#
derekcollison commented 11 months ago

Would it be possible to upgrade the system to 2.10.3, or at least 2.9.23?

cvoica commented 11 months ago

will do

derekcollison commented 11 months ago

Thanks, keep us posted if the issue persists.

cvoica commented 11 months ago

The test today seems to be working fine: we've updated to nats-server: v2.10.3 We still get warnings in the logs but the replication finishes

Oct 24 18:48:57 ip-10-112-229-54.ca-central-1.compute.internal nats-d[5948]: [5948] 2023/10/24 18:48:57.144746 [WRN] JetStream stream '$G > crypto-binc' is not current
Oct 25 09:24:39 ip-10-112-229-54.ca-central-1.compute.internal nats-d[5948]: [5948] 2023/10/25 09:24:39.959813 [WRN] RAFT [3JdJR0O6 - S-R3F-Sz1Da20a] 15000 append entries pending
Oct 25 09:24:46 ip-10-112-229-54.ca-central-1.compute.internal nats-d[5948]: [5948] 2023/10/25 09:24:46.007452 [WRN] RAFT [3JdJR0O6 - S-R3F-Sz1Da20a] 20000 append entries pending
Oct 25 09:24:57 ip-10-112-229-54.ca-central-1.compute.internal nats-d[5948]: [5948] 2023/10/25 09:24:57.144668 [WRN] RAFT [3JdJR0O6 - S-R3F-Sz1Da20a] Falling behind in health check, commit 148233206 != applied 0
Oct 25 09:24:57 ip-10-112-229-54.ca-central-1.compute.internal nats-d[5948]: [5948] 2023/10/25 09:24:57.144690 [WRN] JetStream stream '$G > crypto-binc' is not current
Oct 25 09:26:57 ip-10-112-229-54.ca-central-1.compute.internal nats-d[5948]: [5948] 2023/10/25 09:26:57.143612 [WRN] RAFT [3JdJR0O6 - S-R3F-Sz1Da20a] Falling behind in health check, commit 148233206 != applied 0
Oct 25 09:26:57 ip-10-112-229-54.ca-central-1.compute.internal nats-d[5948]: [5948] 2023/10/25 09:26:57.143632 [WRN] JetStream stream '$G > crypto-binc' is not current
Oct 25 09:28:57 ip-10-112-229-54.ca-central-1.compute.internal nats-d[5948]: [5948] 2023/10/25 09:28:57.143616 [WRN] RAFT [3JdJR0O6 - S-R3F-Sz1Da20a] Falling behind in health check, commit 148233206 != applied 0
Oct 25 09:28:57 ip-10-112-229-54.ca-central-1.compute.internal nats-d[5948]: [5948] 2023/10/25 09:28:57.143641 [WRN] JetStream stream '$G > crypto-binc' is not current
Oct 25 09:30:57 ip-10-112-229-54.ca-central-1.compute.internal nats-d[5948]: [5948] 2023/10/25 09:30:57.143476 [WRN] RAFT [3JdJR0O6 - S-R3F-Sz1Da20a] Falling behind in health check, commit 148233206 != applied 0
Oct 25 09:30:57 ip-10-112-229-54.ca-central-1.compute.internal nats-d[5948]: [5948] 2023/10/25 09:30:57.143505 [WRN] JetStream stream '$G > crypto-binc' is not current
Oct 25 09:32:57 ip-10-112-229-54.ca-central-1.compute.internal nats-d[5948]: [5948] 2023/10/25 09:32:57.147570 [WRN] RAFT [3JdJR0O6 - S-R3F-Sz1Da20a] Falling behind in health check, commit 148233206 != applied 0
Oct 25 09:32:57 ip-10-112-229-54.ca-central-1.compute.internal nats-d[5948]: [5948] 2023/10/25 09:32:57.147607 [WRN] JetStream stream '$G > crypto-binc' is not current
Oct 25 09:34:57 ip-10-112-229-54.ca-central-1.compute.internal nats-d[5948]: [5948] 2023/10/25 09:34:57.143542 [WRN] RAFT [3JdJR0O6 - S-R3F-Sz1Da20a] Falling behind in health check, commit 148233206 != applied 0
Oct 25 09:34:57 ip-10-112-229-54.ca-central-1.compute.internal nats-d[5948]: [5948] 2023/10/25 09:34:57.143561 [WRN] JetStream stream '$G > crypto-binc' is not current
Oct 25 09:36:57 ip-10-112-229-54.ca-central-1.compute.internal nats-d[5948]: [5948] 2023/10/25 09:36:57.143586 [WRN] RAFT [3JdJR0O6 - S-R3F-Sz1Da20a] Falling behind in health check, commit 148233206 != applied 0
Oct 25 09:36:57 ip-10-112-229-54.ca-central-1.compute.internal nats-d[5948]: [5948] 2023/10/25 09:36:57.143606 [WRN] JetStream stream '$G > crypto-binc' is not current
Oct 25 09:38:57 ip-10-112-229-54.ca-central-1.compute.internal nats-d[5948]: [5948] 2023/10/25 09:38:57.143882 [WRN] RAFT [3JdJR0O6 - S-R3F-Sz1Da20a] Falling behind in health check, commit 148233206 != applied 0
Oct 25 09:38:57 ip-10-112-229-54.ca-central-1.compute.internal nats-d[5948]: [5948] 2023/10/25 09:38:57.143901 [WRN] JetStream stream '$G > crypto-binc' is not current
Oct 25 09:40:57 ip-10-112-229-54.ca-central-1.compute.internal nats-d[5948]: [5948] 2023/10/25 09:40:57.143421 [WRN] RAFT [3JdJR0O6 - S-R3F-Sz1Da20a] Falling behind in health check, commit 148233206 != applied 0
Oct 25 09:40:57 ip-10-112-229-54.ca-central-1.compute.internal nats-d[5948]: [5948] 2023/10/25 09:40:57.143449 [WRN] JetStream stream '$G > crypto-binc' is not current
Oct 25 09:42:57 ip-10-112-229-54.ca-central-1.compute.internal nats-d[5948]: [5948] 2023/10/25 09:42:57.143702 [WRN] RAFT [3JdJR0O6 - S-R3F-Sz1Da20a] Falling behind in health check, commit 148233206 != applied 0
Oct 25 09:43:00 ip-10-112-229-54.ca-central-1.compute.internal nats-d[5948]: [5948] 2023/10/25 09:43:00.849095 [WRN] JetStream stream '$G > crypto-binc' is not current
Oct 25 09:44:57 ip-10-112-229-54.ca-central-1.compute.internal nats-d[5948]: [5948] 2023/10/25 09:44:57.143631 [WRN] RAFT [3JdJR0O6 - S-R3F-Sz1Da20a] Falling behind in health check, commit 148233206 != applied 0
Oct 25 09:44:57 ip-10-112-229-54.ca-central-1.compute.internal nats-d[5948]: [5948] 2023/10/25 09:44:57.143654 [WRN] JetStream stream '$G > crypto-binc' is not current
Oct 25 09:46:57 ip-10-112-229-54.ca-central-1.compute.internal nats-d[5948]: [5948] 2023/10/25 09:46:57.143455 [WRN] RAFT [3JdJR0O6 - S-R3F-Sz1Da20a] Falling behind in health check, commit 148233206 != applied 0
Oct 25 09:46:57 ip-10-112-229-54.ca-central-1.compute.internal nats-d[5948]: [5948] 2023/10/25 09:46:57.143479 [WRN] JetStream stream '$G > crypto-binc' is not current
Oct 25 09:48:57 ip-10-112-229-54.ca-central-1.compute.internal nats-d[5948]: [5948] 2023/10/25 09:48:57.143521 [WRN] RAFT [3JdJR0O6 - S-R3F-Sz1Da20a] Falling behind in health check, commit 148233206 != applied 0
Oct 25 09:48:57 ip-10-112-229-54.ca-central-1.compute.internal nats-d[5948]: [5948] 2023/10/25 09:48:57.143543 [WRN] JetStream stream '$G > crypto-binc' is not current
Oct 25 09:50:57 ip-10-112-229-54.ca-central-1.compute.internal nats-d[5948]: [5948] 2023/10/25 09:50:57.143667 [WRN] RAFT [3JdJR0O6 - S-R3F-Sz1Da20a] Falling behind in health check, commit 148233206 != applied 0
Oct 25 09:50:57 ip-10-112-229-54.ca-central-1.compute.internal nats-d[5948]: [5948] 2023/10/25 09:50:57.143686 [WRN] JetStream stream '$G > crypto-binc' is not current
Oct 25 09:52:57 ip-10-112-229-54.ca-central-1.compute.internal nats-d[5948]: [5948] 2023/10/25 09:52:57.143490 [WRN] RAFT [3JdJR0O6 - S-R3F-Sz1Da20a] Falling behind in health check, commit 148233206 != applied 0
Oct 25 09:52:57 ip-10-112-229-54.ca-central-1.compute.internal nats-d[5948]: [5948] 2023/10/25 09:52:57.143509 [WRN] JetStream stream '$G > crypto-binc' is not current
Oct 25 09:54:57 ip-10-112-229-54.ca-central-1.compute.internal nats-d[5948]: [5948] 2023/10/25 09:54:57.143822 [WRN] RAFT [3JdJR0O6 - S-R3F-Sz1Da20a] Falling behind in health check, commit 148233206 != applied 0
Oct 25 09:54:57 ip-10-112-229-54.ca-central-1.compute.internal nats-d[5948]: [5948] 2023/10/25 09:54:57.143847 [WRN] JetStream stream '$G > crypto-binc' is not current
Oct 25 09:56:57 ip-10-112-229-54.ca-central-1.compute.internal nats-d[5948]: [5948] 2023/10/25 09:56:57.143761 [WRN] RAFT [3JdJR0O6 - S-R3F-Sz1Da20a] Falling behind in health check, commit 148233206 != applied 0
Oct 25 09:56:57 ip-10-112-229-54.ca-central-1.compute.internal nats-d[5948]: [5948] 2023/10/25 09:56:57.143790 [WRN] JetStream stream '$G > crypto-binc' is not current
Oct 25 09:58:57 ip-10-112-229-54.ca-central-1.compute.internal nats-d[5948]: [5948] 2023/10/25 09:58:57.156602 [WRN] RAFT [3JdJR0O6 - S-R3F-Sz1Da20a] Falling behind in health check, commit 148233206 != applied 0
Oct 25 09:58:57 ip-10-112-229-54.ca-central-1.compute.internal nats-d[5948]: [5948] 2023/10/25 09:58:57.156632 [WRN] JetStream stream '$G > crypto-binc' is not current
Oct 25 10:00:57 ip-10-112-229-54.ca-central-1.compute.internal nats-d[5948]: [5948] 2023/10/25 10:00:57.143450 [WRN] RAFT [3JdJR0O6 - S-R3F-Sz1Da20a] Falling behind in health check, commit 148233206 != applied 0
Oct 25 10:00:57 ip-10-112-229-54.ca-central-1.compute.internal nats-d[5948]: [5948] 2023/10/25 10:00:57.143480 [WRN] JetStream stream '$G > crypto-binc' is not current
Oct 25 10:02:57 ip-10-112-229-54.ca-central-1.compute.internal nats-d[5948]: [5948] 2023/10/25 10:02:57.143689 [WRN] RAFT [3JdJR0O6 - S-R3F-Sz1Da20a] Falling behind in health check, commit 148233206 != applied 0
Oct 25 10:02:57 ip-10-112-229-54.ca-central-1.compute.internal nats-d[5948]: [5948] 2023/10/25 10:02:57.143708 [WRN] JetStream stream '$G > crypto-binc' is not current
Oct 25 10:04:57 ip-10-112-229-54.ca-central-1.compute.internal nats-d[5948]: [5948] 2023/10/25 10:04:57.152566 [WRN] RAFT [3JdJR0O6 - S-R3F-Sz1Da20a] Falling behind in health check, commit 148233206 != applied 0
Oct 25 10:04:57 ip-10-112-229-54.ca-central-1.compute.internal nats-d[5948]: [5948] 2023/10/25 10:04:57.152594 [WRN] JetStream stream '$G > crypto-binc' is not current
Oct 25 10:06:57 ip-10-112-229-54.ca-central-1.compute.internal nats-d[5948]: [5948] 2023/10/25 10:06:57.143790 [WRN] RAFT [3JdJR0O6 - S-R3F-Sz1Da20a] Falling behind in health check, commit 148233206 != applied 0
Oct 25 10:06:57 ip-10-112-229-54.ca-central-1.compute.internal nats-d[5948]: [5948] 2023/10/25 10:06:57.143809 [WRN] JetStream stream '$G > crypto-binc' is not current
Oct 25 10:08:57 ip-10-112-229-54.ca-central-1.compute.internal nats-d[5948]: [5948] 2023/10/25 10:08:57.143549 [WRN] RAFT [3JdJR0O6 - S-R3F-Sz1Da20a] Falling behind in health check, commit 148233206 != applied 0
Oct 25 10:08:57 ip-10-112-229-54.ca-central-1.compute.internal nats-d[5948]: [5948] 2023/10/25 10:08:57.143570 [WRN] JetStream stream '$G > crypto-binc' is not current
Oct 25 10:10:57 ip-10-112-229-54.ca-central-1.compute.internal nats-d[5948]: [5948] 2023/10/25 10:10:57.143687 [WRN] RAFT [3JdJR0O6 - S-R3F-Sz1Da20a] Falling behind in health check, commit 148233206 != applied 0
Oct 25 10:10:57 ip-10-112-229-54.ca-central-1.compute.internal nats-d[5948]: [5948] 2023/10/25 10:10:57.143722 [WRN] JetStream stream '$G > crypto-binc' is not current
Oct 25 10:12:57 ip-10-112-229-54.ca-central-1.compute.internal nats-d[5948]: [5948] 2023/10/25 10:12:57.143745 [WRN] RAFT [3JdJR0O6 - S-R3F-Sz1Da20a] Falling behind in health check, commit 148233206 != applied 0
Oct 25 10:12:57 ip-10-112-229-54.ca-central-1.compute.internal nats-d[5948]: [5948] 2023/10/25 10:12:57.143765 [WRN] JetStream stream '$G > crypto-binc' is not current
Oct 25 10:14:57 ip-10-112-229-54.ca-central-1.compute.internal nats-d[5948]: [5948] 2023/10/25 10:14:57.143451 [WRN] RAFT [3JdJR0O6 - S-R3F-Sz1Da20a] Falling behind in health check, commit 148233206 != applied 0
Oct 25 10:14:57 ip-10-112-229-54.ca-central-1.compute.internal nats-d[5948]: [5948] 2023/10/25 10:14:57.143472 [WRN] JetStream stream '$G > crypto-binc' is not current
Oct 25 10:16:57 ip-10-112-229-54.ca-central-1.compute.internal nats-d[5948]: [5948] 2023/10/25 10:16:57.143622 [WRN] RAFT [3JdJR0O6 - S-R3F-Sz1Da20a] Falling behind in health check, commit 148233206 != applied 0
Oct 25 10:16:57 ip-10-112-229-54.ca-central-1.compute.internal nats-d[5948]: [5948] 2023/10/25 10:16:57.143642 [WRN] JetStream stream '$G > crypto-binc' is not current
Oct 25 10:18:57 ip-10-112-229-54.ca-central-1.compute.internal nats-d[5948]: [5948] 2023/10/25 10:18:57.144263 [WRN] RAFT [3JdJR0O6 - S-R3F-Sz1Da20a] Falling behind in health check, commit 148233206 != applied 0
Oct 25 10:18:57 ip-10-112-229-54.ca-central-1.compute.internal nats-d[5948]: [5948] 2023/10/25 10:18:57.144284 [WRN] JetStream stream '$G > crypto-binc' is not current
Oct 25 10:20:57 ip-10-112-229-54.ca-central-1.compute.internal nats-d[5948]: [5948] 2023/10/25 10:20:57.145954 [WRN] RAFT [3JdJR0O6 - S-R3F-Sz1Da20a] Falling behind in health check, commit 148233206 != applied 0
Oct 25 10:20:57 ip-10-112-229-54.ca-central-1.compute.internal nats-d[5948]: [5948] 2023/10/25 10:20:57.190957 [WRN] JetStream stream '$G > crypto-binc' is not current
Oct 25 10:22:57 ip-10-112-229-54.ca-central-1.compute.internal nats-d[5948]: [5948] 2023/10/25 10:22:57.143443 [WRN] RAFT [3JdJR0O6 - S-R3F-Sz1Da20a] Falling behind in health check, commit 148233206 != applied 0
Oct 25 10:22:57 ip-10-112-229-54.ca-central-1.compute.internal nats-d[5948]: [5948] 2023/10/25 10:22:57.143464 [WRN] JetStream stream '$G > crypto-binc' is not current
Oct 25 10:24:43 ip-10-112-229-54.ca-central-1.compute.internal nats-d[5948]: [5948] 2023/10/25 10:24:43.411547 [WRN] Internal subscription on "$JS.API.STREAM.INFO.crypto-binc" took too long: 23.422857034s
Oct 25 10:24:43 ip-10-112-229-54.ca-central-1.compute.internal nats-d[5948]: [5948] 2023/10/25 10:24:43.411578 [WRN] Internal subscription on "$JS.API.STREAM.INFO.crypto-binc" took too long: 13.160580544s
Oct 25 10:24:43 ip-10-112-229-54.ca-central-1.compute.internal nats-d[5948]: [5948] 2023/10/25 10:24:43.411595 [WRN] Internal subscription on "$JS.API.STREAM.INFO.crypto-binc" took too long: 13.160626234s
Oct 25 10:24:43 ip-10-112-229-54.ca-central-1.compute.internal nats-d[5948]: [5948] 2023/10/25 10:24:43.411625 [WRN] Internal subscription on "$JS.API.STREAM.INFO.crypto-binc" took too long: 13.160681791s
Oct 25 10:24:43 ip-10-112-229-54.ca-central-1.compute.internal nats-d[5948]: [5948] 2023/10/25 10:24:43.411634 [WRN] 10.115.107.254:42894 - cid:1592 - Readloop processing time: 13.160791366s

Will continue to test with a smaller EC2 instance (network up to 25Gbps) to see if network influences it.

I have made some experiments with max_outstanding_catchup: 1G and I have some questions here as we would like to speed-up replication which is currently taking 45 minutes for a bit more than 300GB.

derekcollison commented 11 months ago

That controls the max inflight bytes for the whole server to any other server trying to catchup.. I would leave at default of 64MB or maybe do 256MB, but would not suggest setting higher.

Have you tested the network hop independently of NATS to verify the throughput?

cvoica commented 10 months ago

The network is 5Gbps for one connection so this is limiting us. I can close the ticket now. Thank you!

[root@ip-10-112-222-219 ~]#  iperf3 -c 10.112.234.207 --parallel 40 -i 1 -t 2 | grep SUM
[SUM]   0.00-1.00   sec  2.93 GBytes  25.2 Gbits/sec    0
[SUM]   1.00-2.00   sec  2.89 GBytes  24.8 Gbits/sec    0
[SUM]   0.00-2.00   sec  5.82 GBytes  25.0 Gbits/sec    0             sender
[SUM]   0.00-2.00   sec  5.78 GBytes  24.8 Gbits/sec                  receiver
[root@ip-10-112-222-219 ~]#  iperf3 -c 10.112.234.207  -i 1 -t 10
Connecting to host 10.112.234.207, port 5201
[  4] local 10.112.222.219 port 34322 connected to 10.112.234.207 port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec   581 MBytes  4.87 Gbits/sec    0   1.27 MBytes
[  4]   1.00-2.00   sec   592 MBytes  4.97 Gbits/sec    0   1.27 MBytes
[  4]   2.00-3.00   sec   592 MBytes  4.97 Gbits/sec    0   1.27 MBytes
[  4]   3.00-4.00   sec   592 MBytes  4.97 Gbits/sec    0   1.27 MBytes
[  4]   4.00-5.00   sec   592 MBytes  4.97 Gbits/sec    0   1.27 MBytes
[  4]   5.00-6.00   sec   592 MBytes  4.97 Gbits/sec    0   1.27 MBytes
[  4]   6.00-7.00   sec   592 MBytes  4.97 Gbits/sec    0   1.27 MBytes
[  4]   7.00-8.00   sec   592 MBytes  4.97 Gbits/sec    0   1.27 MBytes
[  4]   8.00-9.00   sec   592 MBytes  4.97 Gbits/sec    0   1.27 MBytes
[  4]   9.00-10.00  sec   592 MBytes  4.97 Gbits/sec    0   1.91 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  5.78 GBytes  4.96 Gbits/sec    0             sender
[  4]   0.00-10.00  sec  5.77 GBytes  4.96 Gbits/sec                  receiver

iperf Done.
cvoica commented 10 months ago

@derekcollison the beauty of the cloud: one must read the fine printing! For instances that are enabled for enhanced networking, traffic between instances within the same Region that is addressed using IPv4 or IPv6 addresses can use up to 5 Gbps for single-flow traffic and up to 25 Gbps for multi-flow traffic. A flow represents a single, point-to-point network connection.

Instances within a cluster placement group can use up to 10 Gbps for single-flow traffic. Instances that are not within a cluster placement group can use up to 5 Gbps for single-flow traffic.

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/placement-groups.html

armangurkan commented 8 months ago

I am running 2.10.5-alpine version, I am trying to open 50k connections, with 50k streams and 50k subjects. After the first 5k connections, I am having a lot of trouble, making the clients create publishers and subscribers. What are the configurations that I need to change or which document should I read to increase the number of connections can be persisted and increase the through put in a stable manner.

derekcollison commented 8 months ago

Best to ask for help on the Slack channels, or reach out to Synadia for support. The answer is very nuanced depending on your application, where the servers are running and on what hardware and under what deployment system etc.