search

nats-io / nats-server

High-Performance server for NATS.io, the cloud and edge native messaging system.
https://nats.io
Apache License 2.0
15.49k stars 1.38k forks source link

Jetstream Sequence numbers different on each replica is that by design? #4174

Closed bet-me closed 1 year ago

bet-me commented 1 year ago

I've got NATS running in Kubernetes with Jetstream enabled

nats:2.9.16 replicas: 3 k8s: v1.25.2

I'm using Jetstream for event sourcing and I'm storing the sequence numbers in a cache to keep track of what messages I've processed.

If I ask for the messages in a subject multiple times, I sometimes get different sequence numbers.

In this case the last sequence numbers alternate between 447953, 447961, 447964

Should NATS Jetstream return the same sequence numbers each time?

seq.txt

bet-me commented 1 year ago

server.txt

derekcollison commented 1 year ago

JetStream tracks 2 sequence numbers, one for the stream, which should never change if a message is redelivered, and the consumer sequence which will always start at 1 and increase by 1 for each message delivered.

bet-me commented 1 year ago

I believe my 3 replicas have different sequence numbers for the same published message

derekcollison commented 1 year ago

We have not seen any reports of that.

How do you access the sequence numbers? Can you share code for that? What does nats stream info show?

If you pull the message directly from the stream with the NATS cli what does it show?

bet-me commented 1 year ago

That's what I've done in the seq.txt above

ripienaar commented 1 year ago

Please show stream info, instead of attachine txt files paste the output here surrounded by 3 backticks.

https://itsfoss.com/markdown-quotes/

bet-me commented 1 year ago

Sequence Info, highlighting the different sequence numbers returned for the same command

nats % nats --user=acc --password=acc -s ws://nats:8443 stream view markets --subject=markets.football.modified.1%2E202512975 | grep -i subject
[19910] Subject: markets.football.modified.1%2E202512975 Received: 2023-04-17T11:23:40+01:00
  Nats-Expected-Last-Subject-Sequence: 0
[35972] Subject: markets.football.modified.1%2E202512975 Received: 2023-04-17T11:29:07+01:00
  Nats-Expected-Last-Subject-Sequence: 19910
[190696] Subject: markets.football.modified.1%2E202512975 Received: 2023-04-18T14:35:33+01:00
  Nats-Expected-Last-Subject-Sequence: 35972
[244149] Subject: markets.football.modified.1%2E202512975 Received: 2023-04-18T22:55:18+01:00
  Nats-Expected-Last-Subject-Sequence: 190696
[390095] Subject: markets.football.modified.1%2E202512975 Received: 2023-05-08T15:05:01+01:00
  Nats-Expected-Last-Subject-Sequence: 244149
[447953] Subject: markets.football.modified.1%2E202512975 Received: 2023-05-15T17:13:42+01:00
  Nats-Expected-Last-Subject-Sequence: 390095
11:57:07 Reached apparent end of data

nats --user=acc --password=acc -s ws://nats:8443 stream view markets --subject=markets.football.modified.1%2E202512975 | grep -i subject
[19910] Subject: markets.football.modified.1%2E202512975 Received: 2023-04-17T11:23:40+01:00
  Nats-Expected-Last-Subject-Sequence: 0
[35972] Subject: markets.football.modified.1%2E202512975 Received: 2023-04-17T11:29:07+01:00
  Nats-Expected-Last-Subject-Sequence: 19910
[190696] Subject: markets.football.modified.1%2E202512975 Received: 2023-04-18T14:35:33+01:00
  Nats-Expected-Last-Subject-Sequence: 35972
[244149] Subject: markets.football.modified.1%2E202512975 Received: 2023-04-18T22:55:18+01:00
  Nats-Expected-Last-Subject-Sequence: 190696
[390088] Subject: markets.football.modified.1%2E202512975 Received: 2023-05-08T15:05:01+01:00
  Nats-Expected-Last-Subject-Sequence: 244149
[447961] Subject: markets.football.modified.1%2E202512975 Received: 2023-05-15T17:13:42+01:00
  Nats-Expected-Last-Subject-Sequence: 390095
11:57:28 Reached apparent end of data

nats --user=acc --password=acc -s ws://nats:8443 stream view markets --subject=markets.football.modified.1%2E202512975 | grep -i subject
[19910] Subject: markets.football.modified.1%2E202512975 Received: 2023-04-17T11:23:40+01:00
  Nats-Expected-Last-Subject-Sequence: 0
[35972] Subject: markets.football.modified.1%2E202512975 Received: 2023-04-17T11:29:07+01:00
  Nats-Expected-Last-Subject-Sequence: 19910
[190696] Subject: markets.football.modified.1%2E202512975 Received: 2023-04-18T14:35:33+01:00
  Nats-Expected-Last-Subject-Sequence: 35972
[244149] Subject: markets.football.modified.1%2E202512975 Received: 2023-04-18T22:55:18+01:00
  Nats-Expected-Last-Subject-Sequence: 190696
[390095] Subject: markets.football.modified.1%2E202512975 Received: 2023-05-08T15:05:01+01:00
  Nats-Expected-Last-Subject-Sequence: 244149
[447964] Subject: markets.football.modified.1%2E202512975 Received: 2023-05-15T17:13:42+01:00
  Nats-Expected-Last-Subject-Sequence: 390095
11:57:32 Reached apparent end of data
bet-me commented 1 year ago

Stream info

nats stream info markets
Information for Stream markets created 2023-04-17 10:00:58

             Subjects: markets.*.*.*.*
             Replicas: 3
              Storage: File

Options:

            Retention: Limits
     Acknowledgements: true
       Discard Policy: Old
     Duplicate Window: 2m0s
    Allows Msg Delete: true
         Allows Purge: true
       Allows Rollups: false

Limits:

     Maximum Messages: unlimited
  Maximum Per Subject: unlimited
        Maximum Bytes: unlimited
          Maximum Age: unlimited
 Maximum Message Size: unlimited
    Maximum Consumers: unlimited

Cluster Information:

                 Name: betfair
               Leader: nats-1
              Replica: nats-0, current, seen 0.62s ago
              Replica: nats-2, current, seen 0.62s ago

State:

             Messages: 464,584
                Bytes: 165 MiB
             FirstSeq: 1 @ 2023-04-17T10:22:56 UTC
              LastSeq: 464,584 @ 2023-05-17T09:12:52 UTC
     Active Consumers: 8
   Number of Subjects: 213,562
bet-me commented 1 year ago

Server report

nats server report jetstream
╭───────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│                                                 JetStream Summary                                                 │
├─────────┬─────────┬─────────┬─────────┬───────────┬────────────┬─────────┬────────┬─────────┬───────────┬─────────┤
│ Server  │ Cluster │ Domain  │ Streams │ Consumers │ Messages   │ Bytes   │ Memory │ File    │ API Req   │ API Err │
├─────────┼─────────┼─────────┼─────────┼───────────┼────────────┼─────────┼────────┼─────────┼───────────┼─────────┤
│ nats-0  │ betfair │ betfair │ 5       │ 3         │ 5,976,511  │ 2.1 GiB │ 0 B    │ 2.1 GiB │ 283,106   │ 0       │
│ nats-1  │ betfair │ betfair │ 5       │ 5         │ 5,976,522  │ 2.1 GiB │ 0 B    │ 2.1 GiB │ 493,324   │ 0       │
│ nats-2* │ betfair │ betfair │ 5       │ 3         │ 5,976,523  │ 2.1 GiB │ 0 B    │ 2.1 GiB │ 501,421   │ 2       │
├─────────┼─────────┼─────────┼─────────┼───────────┼────────────┼─────────┼────────┼─────────┼───────────┼─────────┤
│         │         │         │ 15      │ 11        │ 17,929,556 │ 6.4 GiB │ 0 B    │ 6.4 GiB │ 1,277,851 │ 2       │
╰─────────┴─────────┴─────────┴─────────┴───────────┴────────────┴─────────┴────────┴─────────┴───────────┴─────────╯

╭──────────────────────────────────────────────────────────────╮
│                 RAFT Meta Group Information                  │
├────────┬──────────┬────────┬─────────┬────────┬────────┬─────┤
│ Name   │ ID       │ Leader │ Current │ Online │ Active │ Lag │
├────────┼──────────┼────────┼─────────┼────────┼────────┼─────┤
│ nats-0 │ S1Nunr6R │        │ true    │ true   │ 0.83s  │ 0   │
│ nats-1 │ yrzKKRBu │        │ true    │ true   │ 0.83s  │ 0   │
│ nats-2 │ cnrtt3eg │ yes    │ true    │ true   │ 0.00s  │ 0   │
╰────────┴──────────┴────────┴─────────┴────────┴────────┴─────╯
derekcollison commented 1 year ago

I see what you are pointing out.

Could you run the stream view commands again with --trace on the end?

Thanks.

ripienaar commented 1 year ago 
            meta, err := jsm.ParseJSMsgMetadata(msg)
            if err == nil {
                fmt.Printf("[%d] Subject: %s Received: %s\n", meta.StreamSequence(), msg.Subject, meta.TimeStamp().Format(time.RFC3339))
            } else {
                fmt.Printf("Subject: %s Reply: %s\n", msg.Subject, msg.Reply)
            }

from stream view, so does seem to be a changing sequence there.

bet-me commented 1 year ago

447953

nats --user=acc --password=acc stream view markets --subject=markets.betfair.football.modified.1%2E202512975 --trace
15:54:47 >>> $JS.API.STREAM.INFO.markets
null

15:54:47 <<< $JS.API.STREAM.INFO.markets
{"type":"io.nats.jetstream.api.v1.stream_info_response","total":0,"offset":0,"limit":0,"config":{"name":"markets","subjects":["markets.*.*.*.*"],"retention":"limits","max_consumers":-1,"max_msgs":-1,"max_bytes":-1,"max_age":0,"max_msgs_per_subject":-1,"max_msg_size":-1,"discard":"old","storage":"file","num_replicas":3,"duplicate_window":120000000000,"allow_direct":false,"mirror_direct":false,"sealed":false,"deny_delete":false,"deny_purge":false,"allow_rollup_hdrs":false},"created":"2023-04-17T10:00:58.301389806Z","state":{"messages":464584,"bytes":173279821,"first_seq":1,"first_ts":"2023-04-17T10:22:56.367357515Z","last_seq":464584,"last_ts":"2023-05-17T09:12:52.065857331Z","num_subjects":213562,"consumer_count":8},"domain":"betfair","cluster":{"name":"betfair","leader":"nats-1","replicas":[{"name":"nats-0","current":true,"active":399430902,"peer":"S1Nunr6R"},{"name":"nats-2","current":true,"active":399447734,"peer":"cnrtt3eg"}]}}

15:54:47 >>> $JS.API.CONSUMER.DURABLE.CREATE.markets.stream_pager_8221684338887605086539
{"stream_name":"markets","config":{"description":"JSM Stream Pager","ack_policy":"explicit","ack_wait":30000000000,"deliver_policy":"all","durable_name":"stream_pager_8221684338887605086539","filter_subject":"markets.betfair.football.modified.1%2E202512975","replay_policy":"instant","inactive_threshold":3600000000000,"num_replicas":0}}

15:54:47 <<< $JS.API.CONSUMER.DURABLE.CREATE.markets.stream_pager_8221684338887605086539
{"type":"io.nats.jetstream.api.v1.consumer_create_response","stream_name":"markets","name":"stream_pager_8221684338887605086539","created":"2023-05-17T15:54:47.606019887Z","config":{"durable_name":"stream_pager_8221684338887605086539","description":"JSM Stream Pager","deliver_policy":"all","ack_policy":"explicit","ack_wait":30000000000,"max_deliver":-1,"filter_subject":"markets.betfair.football.modified.1%2E202512975","replay_policy":"instant","max_waiting":512,"max_ack_pending":1000,"inactive_threshold":3600000000000,"num_replicas":0},"delivered":{"consumer_seq":0,"stream_seq":0},"ack_floor":{"consumer_seq":0,"stream_seq":0},"num_ack_pending":0,"num_redelivered":0,"num_waiting":0,"num_pending":6,"cluster":{"name":"betfair","leader":"nats-0","replicas":[{"name":"nats-1","current":true,"active":38674265,"peer":"yrzKKRBu"},{"name":"nats-2","current":true,"active":39236009,"peer":"cnrtt3eg"}]}}

[19910] Subject: markets.betfair.football.modified.1%2E202512975 Received: 2023-04-17T10:23:40Z

  Nats-Expected-Last-Subject-Sequence: 0

R?
78601P??ȯ?????>???????ǡ????????뾊????GIBRALTAR REGULATOR??   UK wallet???<!--Football - Cup - To Reach the Final - Generic --><br> Will the named team reach the Final of this competition?<br> At the start of the tournament all unmatched bets will be cancelled and the market will be reopened.The market will then remain open until the start of the 1st semi-final where unmatched bets will be cancelled and the market turned in play.<br><br></b>For further information please see <a href=http://content.betfair.com/aboutus/content.asp?sWhichKey=Rules%20and%20Regulations#undefined.do style=color:0163ad; text-decoration: underline; target=_blank>Rules & Regs<br><br>
?@CLASSIC?@228?A
                ??

betting_type
competition_id

fixture_id
in_play_enabled

market_time

operator_id
persistence_enabled
price_ladder_type
    published
    regulator
rules

suspend_time
wallet

[35972] Subject: markets.betfair.football.modified.1%2E202512975 Received: 2023-04-17T10:29:07Z

  Nats-Expected-Last-Subject-Sequence: 19910

R#P????ì??>?>??@ ?A??գ?A
0?r
complete
key_line_selections
market_status
number_of_active_runners
number_of_selections
    published
version

[190696] Subject: markets.betfair.football.modified.1%2E202512975 Received: 2023-04-18T13:35:33Z

  Nats-Expected-Last-Subject-Sequence: 35972

RP??????«?A?֋??
    published
version

[244149] Subject: markets.betfair.football.modified.1%2E202512975 Received: 2023-04-18T21:55:18Z

  Nats-Expected-Last-Subject-Sequence: 190696

RP?????ɫ???A?????.
number_of_active_runners
    published
version

[390095] Subject: markets.betfair.football.modified.1%2E202512975 Received: 2023-05-08T14:05:01Z

  Nats-Expected-Last-Subject-Sequence: 244149

RP????̮͂???A????.
number_of_active_runners
    published
version

[447953] Subject: markets.betfair.football.modified.1%2E202512975 Received: 2023-05-15T16:13:42Z

  Nats-Expected-Last-Subject-Sequence: 390095

RP??ݏ??ׯ????A?????(
    bet_delay
in_play
    published
version

15:54:48 Reached apparent end of data
15:54:48 >>> $JS.API.CONSUMER.DELETE.markets.stream_pager_8221684338887605086539

15:54:48 <<< $JS.API.CONSUMER.DELETE.markets.stream_pager_8221684338887605086539
{"type":"io.nats.jetstream.api.v1.consumer_delete_response","success":true}
bet-me commented 1 year ago

447961

nats --user=acc --password=acc stream view markets --subject=markets.betfair.football.modified.1%2E202512975 --trace
15:55:41 >>> $JS.API.STREAM.INFO.markets
null

15:55:41 <<< $JS.API.STREAM.INFO.markets
{"type":"io.nats.jetstream.api.v1.stream_info_response","total":0,"offset":0,"limit":0,"config":{"name":"markets","subjects":["markets.*.*.*.*"],"retention":"limits","max_consumers":-1,"max_msgs":-1,"max_bytes":-1,"max_age":0,"max_msgs_per_subject":-1,"max_msg_size":-1,"discard":"old","storage":"file","num_replicas":3,"duplicate_window":120000000000,"allow_direct":false,"mirror_direct":false,"sealed":false,"deny_delete":false,"deny_purge":false,"allow_rollup_hdrs":false},"created":"2023-04-17T10:00:58.301389806Z","state":{"messages":464584,"bytes":173279821,"first_seq":1,"first_ts":"2023-04-17T10:22:56.367357515Z","last_seq":464584,"last_ts":"2023-05-17T09:12:52.065857331Z","num_subjects":213562,"consumer_count":8},"domain":"betfair","cluster":{"name":"betfair","leader":"nats-1","replicas":[{"name":"nats-0","current":true,"active":549078080,"peer":"S1Nunr6R"},{"name":"nats-2","current":true,"active":549401309,"peer":"cnrtt3eg"}]}}

15:55:41 >>> $JS.API.CONSUMER.DURABLE.CREATE.markets.stream_pager_8571684338941746979172
{"stream_name":"markets","config":{"description":"JSM Stream Pager","ack_policy":"explicit","ack_wait":30000000000,"deliver_policy":"all","durable_name":"stream_pager_8571684338941746979172","filter_subject":"markets.betfair.football.modified.1%2E202512975","replay_policy":"instant","inactive_threshold":3600000000000,"num_replicas":0}}

15:55:42 <<< $JS.API.CONSUMER.DURABLE.CREATE.markets.stream_pager_8571684338941746979172
{"type":"io.nats.jetstream.api.v1.consumer_create_response","stream_name":"markets","name":"stream_pager_8571684338941746979172","created":"2023-05-17T15:55:41.747893747Z","config":{"durable_name":"stream_pager_8571684338941746979172","description":"JSM Stream Pager","deliver_policy":"all","ack_policy":"explicit","ack_wait":30000000000,"max_deliver":-1,"filter_subject":"markets.betfair.football.modified.1%2E202512975","replay_policy":"instant","max_waiting":512,"max_ack_pending":1000,"inactive_threshold":3600000000000,"num_replicas":0},"delivered":{"consumer_seq":0,"stream_seq":0},"ack_floor":{"consumer_seq":0,"stream_seq":0},"num_ack_pending":0,"num_redelivered":0,"num_waiting":0,"num_pending":6,"cluster":{"name":"betfair","leader":"nats-1","replicas":[{"name":"nats-0","current":true,"active":41170693,"peer":"S1Nunr6R"},{"name":"nats-2","current":true,"active":41246692,"peer":"cnrtt3eg"}]}}

[19910] Subject: markets.betfair.football.modified.1%2E202512975 Received: 2023-04-17T10:23:40Z

  Nats-Expected-Last-Subject-Sequence: 0

R?
78601P??ȯ?????>???????ǡ????????뾊????GIBRALTAR REGULATOR??   UK wallet???<!--Football - Cup - To Reach the Final - Generic --><br> Will the named team reach the Final of this competition?<br> At the start of the tournament all unmatched bets will be cancelled and the market will be reopened.The market will then remain open until the start of the 1st semi-final where unmatched bets will be cancelled and the market turned in play.<br><br></b>For further information please see <a href=http://content.betfair.com/aboutus/content.asp?sWhichKey=Rules%20and%20Regulations#undefined.do style=color:0163ad; text-decoration: underline; target=_blank>Rules & Regs<br><br>
?@CLASSIC?@228?A
                ??

betting_type
competition_id

fixture_id
in_play_enabled

market_time

operator_id
persistence_enabled
price_ladder_type
    published
    regulator
rules

suspend_time
wallet

[35972] Subject: markets.betfair.football.modified.1%2E202512975 Received: 2023-04-17T10:29:07Z

  Nats-Expected-Last-Subject-Sequence: 19910

R#P????ì??>?>??@ ?A??գ?A
0?r
complete
key_line_selections
market_status
number_of_active_runners
number_of_selections
    published
version

[190696] Subject: markets.betfair.football.modified.1%2E202512975 Received: 2023-04-18T13:35:33Z

  Nats-Expected-Last-Subject-Sequence: 35972

RP??????«?A?֋??
    published
version

[244149] Subject: markets.betfair.football.modified.1%2E202512975 Received: 2023-04-18T21:55:18Z

  Nats-Expected-Last-Subject-Sequence: 190696

RP?????ɫ???A?????.
number_of_active_runners
    published
version

[390088] Subject: markets.betfair.football.modified.1%2E202512975 Received: 2023-05-08T14:05:01Z

  Nats-Expected-Last-Subject-Sequence: 244149

RP????̮͂???A????.
number_of_active_runners
    published
version

[447961] Subject: markets.betfair.football.modified.1%2E202512975 Received: 2023-05-15T16:13:42Z

  Nats-Expected-Last-Subject-Sequence: 390095

RP??ݏ??ׯ????A?????(
    bet_delay
in_play
    published
version

15:55:42 Reached apparent end of data
15:55:42 >>> $JS.API.CONSUMER.DELETE.markets.stream_pager_8571684338941746979172

15:55:42 <<< $JS.API.CONSUMER.DELETE.markets.stream_pager_8571684338941746979172
{"type":"io.nats.jetstream.api.v1.consumer_delete_response","success":true}
bet-me commented 1 year ago

447964

nats --user=acc --password=acc stream view markets --subject=markets.betfair.football.modified.1%2E202512975 --trace
15:56:54 >>> $JS.API.STREAM.INFO.markets
null

15:56:54 <<< $JS.API.STREAM.INFO.markets
{"type":"io.nats.jetstream.api.v1.stream_info_response","total":0,"offset":0,"limit":0,"config":{"name":"markets","subjects":["markets.*.*.*.*"],"retention":"limits","max_consumers":-1,"max_msgs":-1,"max_bytes":-1,"max_age":0,"max_msgs_per_subject":-1,"max_msg_size":-1,"discard":"old","storage":"file","num_replicas":3,"duplicate_window":120000000000,"allow_direct":false,"mirror_direct":false,"sealed":false,"deny_delete":false,"deny_purge":false,"allow_rollup_hdrs":false},"created":"2023-04-17T10:00:58.301389806Z","state":{"messages":464584,"bytes":173279821,"first_seq":1,"first_ts":"2023-04-17T10:22:56.367357515Z","last_seq":464584,"last_ts":"2023-05-17T09:12:52.065857331Z","num_subjects":213562,"consumer_count":8},"domain":"betfair","cluster":{"name":"betfair","leader":"nats-1","replicas":[{"name":"nats-0","current":true,"active":289126162,"peer":"S1Nunr6R"},{"name":"nats-2","current":true,"active":289306118,"peer":"cnrtt3eg"}]}}

15:56:54 >>> $JS.API.CONSUMER.DURABLE.CREATE.markets.stream_pager_9671684339014512786058
{"stream_name":"markets","config":{"description":"JSM Stream Pager","ack_policy":"explicit","ack_wait":30000000000,"deliver_policy":"all","durable_name":"stream_pager_9671684339014512786058","filter_subject":"markets.betfair.football.modified.1%2E202512975","replay_policy":"instant","inactive_threshold":3600000000000,"num_replicas":0}}

15:56:55 <<< $JS.API.CONSUMER.DURABLE.CREATE.markets.stream_pager_9671684339014512786058
{"type":"io.nats.jetstream.api.v1.consumer_create_response","stream_name":"markets","name":"stream_pager_9671684339014512786058","created":"2023-05-17T15:56:54.51345781Z","config":{"durable_name":"stream_pager_9671684339014512786058","description":"JSM Stream Pager","deliver_policy":"all","ack_policy":"explicit","ack_wait":30000000000,"max_deliver":-1,"filter_subject":"markets.betfair.football.modified.1%2E202512975","replay_policy":"instant","max_waiting":512,"max_ack_pending":1000,"inactive_threshold":3600000000000,"num_replicas":0},"delivered":{"consumer_seq":0,"stream_seq":0},"ack_floor":{"consumer_seq":0,"stream_seq":0},"num_ack_pending":0,"num_redelivered":0,"num_waiting":0,"num_pending":6,"cluster":{"name":"betfair","leader":"nats-2","replicas":[{"name":"nats-0","current":true,"active":42854562,"peer":"S1Nunr6R"},{"name":"nats-1","current":true,"active":42813137,"peer":"yrzKKRBu"}]}}

[19910] Subject: markets.betfair.football.modified.1%2E202512975 Received: 2023-04-17T10:23:40Z

  Nats-Expected-Last-Subject-Sequence: 0

R?
78601P??ȯ?????>???????ǡ????????뾊????GIBRALTAR REGULATOR??   UK wallet???<!--Football - Cup - To Reach the Final - Generic --><br> Will the named team reach the Final of this competition?<br> At the start of the tournament all unmatched bets will be cancelled and the market will be reopened.The market will then remain open until the start of the 1st semi-final where unmatched bets will be cancelled and the market turned in play.<br><br></b>For further information please see <a href=http://content.betfair.com/aboutus/content.asp?sWhichKey=Rules%20and%20Regulations#undefined.do style=color:0163ad; text-decoration: underline; target=_blank>Rules & Regs<br><br>
?@CLASSIC?@228?A
                ??

betting_type
competition_id

fixture_id
in_play_enabled

market_time

operator_id
persistence_enabled
price_ladder_type
    published
    regulator
rules

suspend_time
wallet

[35972] Subject: markets.betfair.football.modified.1%2E202512975 Received: 2023-04-17T10:29:07Z

  Nats-Expected-Last-Subject-Sequence: 19910

R#P????ì??>?>??@ ?A??գ?A
0?r
complete
key_line_selections
market_status
number_of_active_runners
number_of_selections
    published
version

[190696] Subject: markets.betfair.football.modified.1%2E202512975 Received: 2023-04-18T13:35:33Z

  Nats-Expected-Last-Subject-Sequence: 35972

RP??????«?A?֋??
    published
version

[244149] Subject: markets.betfair.football.modified.1%2E202512975 Received: 2023-04-18T21:55:18Z

  Nats-Expected-Last-Subject-Sequence: 190696

RP?????ɫ???A?????.
number_of_active_runners
    published
version

[390095] Subject: markets.betfair.football.modified.1%2E202512975 Received: 2023-05-08T14:05:01Z

  Nats-Expected-Last-Subject-Sequence: 244149

RP????̮͂???A????.
number_of_active_runners
    published
version

[447964] Subject: markets.betfair.football.modified.1%2E202512975 Received: 2023-05-15T16:13:42Z

  Nats-Expected-Last-Subject-Sequence: 390095

RP??ݏ??ׯ????A?????(
    bet_delay
in_play
    published
version

15:56:55 Reached apparent end of data
15:56:55 >>> $JS.API.CONSUMER.DELETE.markets.stream_pager_9671684339014512786058

15:56:55 <<< $JS.API.CONSUMER.DELETE.markets.stream_pager_9671684339014512786058
{"type":"io.nats.jetstream.api.v1.consumer_delete_response","success":true}
derekcollison commented 1 year ago

Will take a closer look later today when I have some more free time.

bet-me commented 1 year ago

Thanks, if you need anything else let me know.

derekcollison commented 1 year ago

Could you do the following for us and send us the output?

nats req "$JS.API.STREAM.INFO.markets" "" --count=3 --trace

bet-me commented 1 year ago 
nats --user=acc --password=acc req "$JS.API.STREAM.INFO.markets" "" --count=3 --trace
19:24:54 Sending request on ".API.STREAM.INFO.markets"
19:24:54 No responders are available
derekcollison commented 1 year ago

nats req "\$JS.API.STREAM.INFO.markets" "" --count=3 --trace

derekcollison commented 1 year ago

Escape the $

bet-me commented 1 year ago 
nats --user=sys --password=sys req "\$JS.API.STREAM.INFO.markets" "" --count=3 --trace
19:36:26 Sending request on "$JS.API.STREAM.INFO.markets"
19:36:31 Sending request on "$JS.API.STREAM.INFO.markets"
19:36:36 Sending request on "$JS.API.STREAM.INFO.markets"
derekcollison commented 1 year ago

Sorry replace --count=3 with `--replies=3

derekcollison commented 1 year ago

And needs to be from user=acc

bet-me commented 1 year ago 
nats --user=acc --password=acc req "\$JS.API.STREAM.INFO.markets" "" --replies=3 --trace
19:42:40 Sending request on "$JS.API.STREAM.INFO.markets"
19:42:40 Received with rtt 1.248412ms
{"type":"io.nats.jetstream.api.v1.stream_info_response","total":0,"offset":0,"limit":0,"config":{"name":"markets","subjects":["markets.*.*.*.*"],"retention":"limits","max_consumers":-1,"max_msgs":-1,"max_bytes":-1,"max_age":0,"max_msgs_per_subject":-1,"max_msg_size":-1,"discard":"old","storage":"file","num_replicas":3,"duplicate_window":120000000000,"allow_direct":false,"mirror_direct":false,"sealed":false,"deny_delete":false,"deny_purge":false,"allow_rollup_hdrs":false},"created":"2023-04-17T10:00:58.301389806Z","state":{"messages":472758,"bytes":175666816,"first_seq":1,"first_ts":"2023-04-17T10:22:56.367357515Z","last_seq":472758,"last_ts":"2023-05-17T19:17:07.847835235Z","num_subjects":221245,"consumer_count":8},"domain":"betfair","cluster":{"name":"betfair","leader":"nats-1","replicas":[{"name":"nats-0","current":true,"active":767913601,"peer":"S1Nunr6R"},{"name":"nats-2","current":true,"active":768030544,"peer":"cnrtt3eg"}]}}

~ #
derekcollison commented 1 year ago

So you only got one response, not three?

bet-me commented 1 year ago

Yes

bet-me commented 1 year ago 
nats --user=acc --password=acc req "\$JS.API.STREAM.INFO.markets" "" --count=3 --replies=3 --trace 
20:04:19 Sending request on "$JS.API.STREAM.INFO.markets"
20:04:19 Received with rtt 45.934387ms
{"type":"io.nats.jetstream.api.v1.stream_info_response","total":0,"offset":0,"limit":0,"config":{"name":"markets","subjects":["markets.*.*.*.*"],"retention":"limits","max_consumers":-1,"max_msgs":-1,"max_bytes":-1,"max_age":0,"max_msgs_per_subject":-1,"max_msg_size":-1,"discard":"old","storage":"file","num_replicas":3,"duplicate_window":120000000000,"allow_direct":false,"mirror_direct":false,"sealed":false,"deny_delete":false,"deny_purge":false,"allow_rollup_hdrs":false},"created":"2023-04-17T10:00:58.301389806Z","state":{"messages":472758,"bytes":175666816,"first_seq":1,"first_ts":"2023-04-17T10:22:56.367357515Z","last_seq":472758,"last_ts":"2023-05-17T19:17:07.847835235Z","num_subjects":221245,"consumer_count":8},"domain":"betfair","cluster":{"name":"betfair","leader":"nats-1","replicas":[{"name":"nats-0","current":true,"active":171676275,"peer":"S1Nunr6R"},{"name":"nats-2","current":true,"active":171673871,"peer":"cnrtt3eg"}]}}

20:04:19 Sending request on "$JS.API.STREAM.INFO.markets"
20:04:19 Received with rtt 1.158968ms
{"type":"io.nats.jetstream.api.v1.stream_info_response","total":0,"offset":0,"limit":0,"config":{"name":"markets","subjects":["markets.*.*.*.*"],"retention":"limits","max_consumers":-1,"max_msgs":-1,"max_bytes":-1,"max_age":0,"max_msgs_per_subject":-1,"max_msg_size":-1,"discard":"old","storage":"file","num_replicas":3,"duplicate_window":120000000000,"allow_direct":false,"mirror_direct":false,"sealed":false,"deny_delete":false,"deny_purge":false,"allow_rollup_hdrs":false},"created":"2023-04-17T10:00:58.301389806Z","state":{"messages":472758,"bytes":175666816,"first_seq":1,"first_ts":"2023-04-17T10:22:56.367357515Z","last_seq":472758,"last_ts":"2023-05-17T19:17:07.847835235Z","num_subjects":221245,"consumer_count":8},"domain":"betfair","cluster":{"name":"betfair","leader":"nats-1","replicas":[{"name":"nats-0","current":true,"active":564485812,"peer":"S1Nunr6R"},{"name":"nats-2","current":true,"active":564483408,"peer":"cnrtt3eg"}]}}

20:04:20 Sending request on "$JS.API.STREAM.INFO.markets"
20:04:20 Received with rtt 849.332µs
{"type":"io.nats.jetstream.api.v1.stream_info_response","total":0,"offset":0,"limit":0,"config":{"name":"markets","subjects":["markets.*.*.*.*"],"retention":"limits","max_consumers":-1,"max_msgs":-1,"max_bytes":-1,"max_age":0,"max_msgs_per_subject":-1,"max_msg_size":-1,"discard":"old","storage":"file","num_replicas":3,"duplicate_window":120000000000,"allow_direct":false,"mirror_direct":false,"sealed":false,"deny_delete":false,"deny_purge":false,"allow_rollup_hdrs":false},"created":"2023-04-17T10:00:58.301389806Z","state":{"messages":472758,"bytes":175666816,"first_seq":1,"first_ts":"2023-04-17T10:22:56.367357515Z","last_seq":472758,"last_ts":"2023-05-17T19:17:07.847835235Z","num_subjects":221245,"consumer_count":8},"domain":"betfair","cluster":{"name":"betfair","leader":"nats-1","replicas":[{"name":"nats-0","current":true,"active":867439175,"peer":"S1Nunr6R"},{"name":"nats-2","current":true,"active":867436771,"peer":"cnrtt3eg"}]}}

~ #
bet-me commented 1 year ago

Added --count

derekcollison commented 1 year ago

No need, just send request once.

derekcollison commented 1 year ago

I will check with @ripienaar that command should work.

One thing to possible retry is to reset cluster state. You can do that by scaling down to 1 and back up to an R3.

nats --user=acc --password=acc stream update markets --replicas=1

then

nats --user=acc --password=acc stream update markets --replicas=3

bet-me commented 1 year ago

Won’t that destroy any evidence as to what caused the issue?

ripienaar commented 1 year ago

Before doing that, please do this:

nats server req jsz --streams

using a system account.

This will produce a result from each of your servers and we can see the per-node state in that output. You can edit it a bit if you like just be sure to give us the stream detail for the stream in question from each server.

bet-me commented 1 year ago 
nats --user=sys --password=sys server req jsz --streams 
{"server":{"name":"nats-2","host":"0.0.0.0","id":"NCPNFREBCNWIA6LRM3FQX2WA2UELDKIEELLS2SAOQA2IAZ7UJPYEG4EL","cluster":"betfair","domain":"betfair","ver":"2.9.16","seq":1512128,"jetstream":true,"time":"2023-05-17T20:37:47.520579364Z"},"data":{"server_id":"NCPNFREBCNWIA6LRM3FQX2WA2UELDKIEELLS2SAOQA2IAZ7UJPYEG4EL","now":"2023-05-17T20:37:47.520478298Z","config":{"max_memory":274877906944,"max_storage":274877906944,"store_dir":"/data/jetstream","domain":"betfair","compress_ok":true},"memory":0,"storage":2296566156,"reserved_memory":0,"reserved_storage":0,"accounts":1,"ha_assets":7,"api":{"total":732547,"errors":5},"streams":5,"consumers":3,"messages":5986788,"bytes":2296566156,"meta_cluster":{"name":"betfair","leader":"nats-2","peer":"cnrtt3eg","replicas":[{"name":"nats-0","current":true,"active":215158552,"peer":"S1Nunr6R"},{"name":"nats-1","current":true,"active":215099140,"peer":"yrzKKRBu"}],"cluster_size":3},"account_details":[{"name":"js","id":"js","memory":0,"storage":6889688786,"reserved_memory":18446744073709551615,"reserved_storage":18446744073709551615,"accounts":0,"ha_assets":0,"api":{"total":1693092,"errors":5},"stream_detail":[{"name":"competitions","created":"2023-04-17T10:00:00.82445078Z","cluster":{"name":"betfair","leader":"nats-1","replicas":[{"name":"nats-0","current":false,"active":103958345516068,"lag":151557,"peer":"S1Nunr6R"},{"name":"nats-1","current":true,"active":333683571,"lag":151557,"peer":"yrzKKRBu"}]},"state":{"messages":707,"bytes":190907,"first_seq":1,"first_ts":"2023-04-17T10:22:56.286020673Z","last_seq":707,"last_ts":"2023-05-17T18:17:43.10541073Z","num_subjects":706,"consumer_count":0}},{"name":"competitors","created":"2023-04-17T10:00:17.485123561Z","cluster":{"name":"betfair","leader":"nats-2","replicas":[{"name":"nats-0","current":true,"active":333153715,"peer":"S1Nunr6R"},{"name":"nats-1","current":true,"active":333206988,"peer":"yrzKKRBu"}]},"state":{"messages":4394,"bytes":1144364,"first_seq":1,"first_ts":"2023-04-17T10:22:56.852403698Z","last_seq":4394,"last_ts":"2023-05-17T17:36:34.25858227Z","num_subjects":4394,"consumer_count":0}},{"name":"fixtures","created":"2023-04-17T10:00:41.391129394Z","cluster":{"name":"betfair","leader":"nats-2","replicas":[{"name":"nats-0","current":true,"active":333173054,"peer":"S1Nunr6R"},{"name":"nats-1","current":true,"active":333204984,"peer":"yrzKKRBu"}]},"state":{"messages":12973,"bytes":3238619,"first_seq":1,"first_ts":"2023-04-17T10:22:56.443136498Z","last_seq":12973,"last_ts":"2023-05-17T19:17:07.814048089Z","num_subjects":10505,"consumer_count":1}},{"name":"markets","created":"2023-04-17T10:00:58.301389806Z","cluster":{"name":"betfair","leader":"nats-1","replicas":[{"name":"nats-0","current":false,"active":103958345580829,"lag":537478,"peer":"S1Nunr6R"},{"name":"nats-1","current":true,"active":333664074,"lag":537478,"peer":"yrzKKRBu"}]},"state":{"messages":472761,"bytes":175667648,"first_seq":1,"first_ts":"2023-04-17T10:22:56.367357515Z","last_seq":472761,"last_ts":"2023-05-17T19:17:07.847835235Z","num_subjects":221247,"consumer_count":2}},{"name":"selections","created":"2023-04-17T10:01:15.569409667Z","cluster":{"name":"betfair","leader":"nats-2","replicas":[{"name":"nats-0","current":true,"active":542234865,"peer":"S1Nunr6R"},{"name":"nats-1","current":true,"active":542337204,"peer":"yrzKKRBu"}]},"state":{"messages":5495953,"bytes":2116324618,"first_seq":1,"first_ts":"2023-04-17T10:22:56.368479378Z","last_seq":5495953,"last_ts":"2023-05-17T18:51:30.878707017Z","num_subjects":1040358,"consumer_count":0}}]}]}}
{"server":{"name":"nats-1","host":"0.0.0.0","id":"NAJPFQLLVOCB5BDZ4NX3CKUVDQLOTCB42FDNHUDASGQXZRL6D3JHTOCC","cluster":"betfair","domain":"betfair","ver":"2.9.16","seq":1513068,"jetstream":true,"time":"2023-05-17T20:37:47.520541169Z"},"data":{"server_id":"NAJPFQLLVOCB5BDZ4NX3CKUVDQLOTCB42FDNHUDASGQXZRL6D3JHTOCC","now":"2023-05-17T20:37:47.520405834Z","config":{"max_memory":274877906944,"max_storage":274877906944,"store_dir":"/data/jetstream","domain":"betfair","compress_ok":true},"memory":0,"storage":2296556392,"reserved_memory":0,"reserved_storage":0,"accounts":1,"ha_assets":7,"api":{"total":585675,"errors":0},"streams":5,"consumers":5,"messages":5986787,"bytes":2296556392,"meta_cluster":{"name":"betfair","leader":"nats-2","peer":"cnrtt3eg","cluster_size":3},"account_details":[{"name":"js","id":"js","memory":0,"storage":6889688786,"reserved_memory":18446744073709551615,"reserved_storage":18446744073709551615,"accounts":0,"ha_assets":0,"api":{"total":1693097,"errors":5},"stream_detail":[{"name":"competitions","created":"2023-04-17T10:00:00.82445078Z","cluster":{"name":"betfair","leader":"nats-1","replicas":[{"name":"nats-0","current":true,"active":333494289,"peer":"S1Nunr6R"},{"name":"nats-2","current":true,"active":333545472,"peer":"cnrtt3eg"}]},"state":{"messages":707,"bytes":190907,"first_seq":1,"first_ts":"2023-04-17T10:22:56.286020673Z","last_seq":707,"last_ts":"2023-05-17T18:17:43.10541073Z","num_subjects":706,"consumer_count":0}},{"name":"competitors","created":"2023-04-17T10:00:17.485123561Z","cluster":{"name":"betfair","leader":"nats-2","replicas":[{"name":"nats-0","current":false,"active":103958345519601,"lag":22416,"peer":"S1Nunr6R"},{"name":"nats-2","current":true,"active":333522140,"lag":22416,"peer":"cnrtt3eg"}]},"state":{"messages":4394,"bytes":1144364,"first_seq":1,"first_ts":"2023-04-17T10:22:56.852403698Z","last_seq":4394,"last_ts":"2023-05-17T17:36:34.25858227Z","num_subjects":4394,"consumer_count":0}},{"name":"fixtures","created":"2023-04-17T10:00:41.391129394Z","cluster":{"name":"betfair","leader":"nats-2","replicas":[{"name":"nats-0","current":false,"active":103958345652044,"lag":15807,"peer":"S1Nunr6R"},{"name":"nats-2","current":true,"active":333499467,"lag":15807,"peer":"cnrtt3eg"}]},"state":{"messages":12973,"bytes":3238562,"first_seq":1,"first_ts":"2023-04-17T10:22:56.443136498Z","last_seq":12973,"last_ts":"2023-05-17T19:17:07.814048089Z","num_subjects":10505,"consumer_count":1}},{"name":"markets","created":"2023-04-17T10:00:58.301389806Z","cluster":{"name":"betfair","leader":"nats-1","replicas":[{"name":"nats-0","current":true,"active":333529791,"peer":"S1Nunr6R"},{"name":"nats-2","current":true,"active":333528317,"peer":"cnrtt3eg"}]},"state":{"messages":472758,"bytes":175666816,"first_seq":1,"first_ts":"2023-04-17T10:22:56.367357515Z","last_seq":472758,"last_ts":"2023-05-17T19:17:07.847835235Z","num_subjects":221245,"consumer_count":4}},{"name":"selections","created":"2023-04-17T10:01:15.569409667Z","cluster":{"name":"betfair","leader":"nats-2","replicas":[{"name":"nats-0","current":false,"active":0,"lag":5499666,"peer":"S1Nunr6R"},{"name":"nats-2","current":true,"active":542617166,"lag":5499666,"peer":"cnrtt3eg"}]},"state":{"messages":5495955,"bytes":2116315743,"first_seq":1,"first_ts":"2023-04-17T10:22:56.368479378Z","last_seq":5495955,"last_ts":"2023-05-17T18:51:30.878707017Z","num_subjects":1040358,"consumer_count":0}}]}]}}
{"server":{"name":"nats-0","host":"0.0.0.0","id":"NDCWTKX2JMEWCRQYFJALJYTIKIOAEIKUXPSM2XMJUW6H5VSLZXLWCC5P","cluster":"betfair","domain":"betfair","ver":"2.9.16","seq":1504372,"jetstream":true,"time":"2023-05-17T20:37:47.521175151Z"},"data":{"server_id":"NDCWTKX2JMEWCRQYFJALJYTIKIOAEIKUXPSM2XMJUW6H5VSLZXLWCC5P","now":"2023-05-17T20:37:47.521036757Z","config":{"max_memory":274877906944,"max_storage":274877906944,"store_dir":"/data/jetstream","domain":"betfair","compress_ok":true},"memory":0,"storage":2296566238,"reserved_memory":0,"reserved_storage":0,"accounts":1,"ha_assets":7,"api":{"total":374884,"errors":0},"streams":5,"consumers":3,"messages":5986776,"bytes":2296566238,"meta_cluster":{"name":"betfair","leader":"nats-2","peer":"cnrtt3eg","cluster_size":3},"account_details":[{"name":"js","id":"js","memory":0,"storage":6889688786,"reserved_memory":18446744073709551615,"reserved_storage":18446744073709551615,"accounts":0,"ha_assets":0,"api":{"total":1693097,"errors":5},"stream_detail":[{"name":"selections","created":"2023-04-17T10:01:15.569409667Z","cluster":{"name":"betfair","leader":"nats-2","replicas":[{"name":"nats-1","current":false,"active":0,"lag":5499666,"peer":"yrzKKRBu"},{"name":"nats-2","current":true,"active":542610718,"lag":5499666,"peer":"cnrtt3eg"}]},"state":{"messages":5495953,"bytes":2116324618,"first_seq":1,"first_ts":"2023-04-17T10:22:56.368479378Z","last_seq":5495953,"last_ts":"2023-05-17T18:51:30.878707017Z","num_subjects":1040358,"consumer_count":0}},{"name":"competitions","created":"2023-04-17T10:00:00.82445078Z","cluster":{"name":"betfair","leader":"nats-1","replicas":[{"name":"nats-1","current":true,"active":333793492,"lag":151557,"peer":"yrzKKRBu"},{"name":"nats-2","current":false,"active":0,"lag":151557,"peer":"cnrtt3eg"}]},"state":{"messages":707,"bytes":190907,"first_seq":1,"first_ts":"2023-04-17T10:22:56.286020673Z","last_seq":707,"last_ts":"2023-05-17T18:17:43.10541073Z","num_subjects":706,"consumer_count":0}},{"name":"competitors","created":"2023-04-17T10:00:17.485123561Z","cluster":{"name":"betfair","leader":"nats-2","replicas":[{"name":"nats-1","current":false,"active":0,"lag":22416,"peer":"yrzKKRBu"},{"name":"nats-2","current":true,"active":333439417,"lag":22416,"peer":"cnrtt3eg"}]},"state":{"messages":4394,"bytes":1144364,"first_seq":1,"first_ts":"2023-04-17T10:22:56.852403698Z","last_seq":4394,"last_ts":"2023-05-17T17:36:34.25858227Z","num_subjects":4394,"consumer_count":0}},{"name":"fixtures","created":"2023-04-17T10:00:41.391129394Z","cluster":{"name":"betfair","leader":"nats-2","replicas":[{"name":"nats-1","current":false,"active":0,"lag":15807,"peer":"yrzKKRBu"},{"name":"nats-2","current":true,"active":333450818,"lag":15807,"peer":"cnrtt3eg"}]},"state":{"messages":12972,"bytes":3238383,"first_seq":1,"first_ts":"2023-04-17T10:22:56.443136498Z","last_seq":12972,"last_ts":"2023-05-17T19:17:07.814048089Z","num_subjects":10505,"consumer_count":1}},{"name":"markets","created":"2023-04-17T10:00:58.301389806Z","cluster":{"name":"betfair","leader":"nats-1","replicas":[{"name":"nats-1","current":true,"active":333807628,"lag":537478,"peer":"yrzKKRBu"},{"name":"nats-2","current":false,"active":0,"lag":537478,"peer":"cnrtt3eg"}]},"state":{"messages":472750,"bytes":175667966,"first_seq":1,"first_ts":"2023-04-17T10:22:56.367357515Z","last_seq":472750,"last_ts":"2023-05-17T19:17:07.847835235Z","num_subjects":221247,"consumer_count":2}}]}]}}
~ # 
~ #
derekcollison commented 1 year ago

So they seemed skewed that is true, unclear as to why this might be.

The replica scale down and back up will sync all replicas to the leader's state.

If you want another state, you can have the leader stepdown to select the leader you want.

derekcollison commented 1 year ago

Will close for now but if needed feel free to reopen.

marc-delpech commented 5 months ago

Hello! I'm adding a few details, because this also happened to us this week. The config is similar:

We shut off our staging environment every night, and put it back on every morning.
On our AKS environment we first encountered what seems to be the same issue as https://github.com/nats-io/nats-server/issues/4410, where we'd come back one morning and every streams of the deployment were gone.
To continue our work, we recreated the streams. But then, we started to run into issues where the sequence number of our consumers for their referred stream were higher than the sequence number of the stream itself, leading to no message consumption.
Hence, we removed our consumers and recreated them again. Everything was fine, at a first look. But then, we noticed one strange behaviour: ephemeral consumers would not have the same sequence number for their referred stream depending on the replica they were created on.
This lead us to the current issue. Running the command above give the following result:

{"server":{"name":"nats-1","host":"0.0.0.0","id":"NBIVM4G76KZKUJCGJFW4V5USRRFG4I7P42EJFOGXY66XVSCWWHEPXPPS","cluster":"nats","ver":"2.9.16","seq":5929,"jetstream":true,"time":"2024-03-28T16:11:30.848918256Z"},"data":{"server_id":"NBIVM4G76KZKUJCGJFW4V5USRRFG4I7P42EJFOGXY66XVSCWWHEPXPPS","now":"2024-03-28T16:11:30.848880846Z","config":{"max_memory":429916160,"max_storage":10737418240,"store_dir":"/data/jetstream","compress_ok":true},"memory":0,"storage":82423,"reserved_memory":0,"reserved_storage":0,"accounts":1,"ha_assets":21,"api":{"total":381,"errors":0},"streams":4,"consumers":34,"messages":54,"bytes":82423,"meta_cluster":{"name":"nats","leader":"nats-0","peer":"S1Nunr6R","cluster_size":3},"account_details":[{"name":"$G","id":"$G","memory":0,"storage":170836,"reserved_memory":18446744073709551615,"reserved_storage":18446744073709551615,"accounts":0,"ha_assets":0,"api":{"total":3925,"errors":1534},"stream_detail":[{"name":"chats","created":"2024-03-26T08:28:38.089896541Z","cluster":{"name":"nats","leader":"nats-2","replicas":[{"name":"nats-0","current":false,"active":0,"lag":274,"peer":"S1Nunr6R"},{"name":"nats-2","current":true,"active":384081563,"lag":274,"peer":"cnrtt3eg"}]},"state":{"messages":48,"bytes":76433,"first_seq":42,"first_ts":"2024-03-28T13:15:53.30222184Z","last_seq":89,"last_ts":"2024-03-28T15:11:50.812816185Z","num_subjects":2,"consumer_count":7}},{"name":"patents","created":"2024-03-26T08:28:39.073905939Z","cluster":{"name":"nats","leader":"nats-0","replicas":[{"name":"nats-0","current":true,"active":16718271,"lag":140,"peer":"S1Nunr6R"},{"name":"nats-2","current":false,"active":0,"lag":140,"peer":"cnrtt3eg"}]},"state":{"messages":2,"bytes":4368,"first_seq":132,"first_ts":"2024-03-28T13:29:15.885738777Z","last_seq":133,"last_ts":"2024-03-28T13:30:16.757064146Z","num_subjects":2,"consumer_count":15}},{"name":"projects","created":"2024-03-26T08:28:40.114880952Z","cluster":{"name":"nats","leader":"nats-0","replicas":[{"name":"nats-0","current":true,"active":105179332,"lag":10,"peer":"S1Nunr6R"},{"name":"nats-2","current":false,"active":0,"lag":10,"peer":"cnrtt3eg"}]},"state":{"messages":0,"bytes":0,"first_seq":3,"first_ts":"0001-01-01T00:00:00Z","last_seq":2,"last_ts":"2024-03-28T09:47:53.891784594Z","consumer_count":1}},{"name":"documents","created":"2024-03-26T08:28:41.189083867Z","cluster":{"name":"nats","leader":"nats-0","replicas":[{"name":"nats-0","current":true,"active":851645761,"lag":132,"peer":"S1Nunr6R"},{"name":"nats-2","current":false,"active":0,"lag":132,"peer":"cnrtt3eg"}]},"state":{"messages":4,"bytes":1622,"first_seq":102,"first_ts":"2024-03-28T13:23:45.96785193Z","last_seq":105,"last_ts":"2024-03-28T13:24:05.277823925Z","num_subjects":2,"consumer_count":11}}]}]}}
{"server":{"name":"nats-0","host":"0.0.0.0","id":"NB4ZZWU2XJJ6SKVQA5OANUWNPOAAQXYD344HNUY3VGTALLYS3CJ5UMWC","cluster":"nats","ver":"2.9.16","seq":5618,"jetstream":true,"time":"2024-03-28T16:11:30.850751323Z"},"data":{"server_id":"NB4ZZWU2XJJ6SKVQA5OANUWNPOAAQXYD344HNUY3VGTALLYS3CJ5UMWC","now":"2024-03-28T16:11:30.850710477Z","config":{"max_memory":429916160,"max_storage":10737418240,"store_dir":"/data/jetstream","compress_ok":true},"memory":0,"storage":5990,"reserved_memory":0,"reserved_storage":0,"accounts":1,"ha_assets":21,"api":{"total":3142,"errors":1534},"streams":4,"consumers":41,"messages":6,"bytes":5990,"meta_cluster":{"name":"nats","leader":"nats-0","peer":"S1Nunr6R","replicas":[{"name":"nats-1","current":true,"active":628168645,"peer":"yrzKKRBu"},{"name":"nats-2","current":true,"active":628356044,"peer":"cnrtt3eg"}],"cluster_size":3},"account_details":[{"name":"$G","id":"$G","memory":0,"storage":170836,"reserved_memory":18446744073709551615,"reserved_storage":18446744073709551615,"accounts":0,"ha_assets":0,"api":{"total":3925,"errors":1534},"stream_detail":[{"name":"projects","created":"2024-03-26T08:28:40.114880952Z","cluster":{"name":"nats","leader":"nats-0","replicas":[{"name":"nats-1","current":true,"active":106509007,"peer":"yrzKKRBu"},{"name":"nats-2","current":true,"active":106562446,"peer":"cnrtt3eg"}]},"state":{"messages":0,"bytes":0,"first_seq":3,"first_ts":"0001-01-01T00:00:00Z","last_seq":2,"last_ts":"2024-03-28T09:47:53.891784594Z","consumer_count":1}},{"name":"chats","created":"2024-03-26T08:28:38.089896541Z","cluster":{"name":"nats","leader":"nats-2","replicas":[{"name":"nats-1","current":false,"active":0,"lag":274,"peer":"yrzKKRBu"},{"name":"nats-2","current":true,"active":385544726,"lag":274,"peer":"cnrtt3eg"}]},"state":{"messages":0,"bytes":0,"first_seq":177,"first_ts":"0001-01-01T00:00:00Z","last_seq":176,"last_ts":"2024-03-27T16:42:31.470009169Z","consumer_count":12}},{"name":"documents","created":"2024-03-26T08:28:41.189083867Z","cluster":{"name":"nats","leader":"nats-0","replicas":[{"name":"nats-1","current":true,"active":852729457,"peer":"yrzKKRBu"},{"name":"nats-2","current":true,"active":852717444,"peer":"cnrtt3eg"}]},"state":{"messages":4,"bytes":1622,"first_seq":102,"first_ts":"2024-03-28T13:23:45.96785193Z","last_seq":105,"last_ts":"2024-03-28T13:24:05.277823925Z","num_subjects":2,"consumer_count":10}},{"name":"patents","created":"2024-03-26T08:28:39.073905939Z","cluster":{"name":"nats","leader":"nats-0","replicas":[{"name":"nats-1","current":true,"active":18025479,"peer":"yrzKKRBu"},{"name":"nats-2","current":true,"active":18151524,"peer":"cnrtt3eg"}]},"state":{"messages":2,"bytes":4368,"first_seq":132,"first_ts":"2024-03-28T13:29:15.885738777Z","last_seq":133,"last_ts":"2024-03-28T13:30:16.757064146Z","num_subjects":2,"consumer_count":18}}]}]}}
{"server":{"name":"nats-2","host":"0.0.0.0","id":"NDLDBCQXSL7KITGNWNS72HL72KWOVZR27RLH3ZJTHZNEOSOBUBTLJVL3","cluster":"nats","ver":"2.9.16","seq":6037,"jetstream":true,"time":"2024-03-28T16:11:30.849909719Z"},"data":{"server_id":"NDLDBCQXSL7KITGNWNS72HL72KWOVZR27RLH3ZJTHZNEOSOBUBTLJVL3","now":"2024-03-28T16:11:30.849868162Z","config":{"max_memory":429916160,"max_storage":10737418240,"store_dir":"/data/jetstream","compress_ok":true},"memory":0,"storage":82423,"reserved_memory":0,"reserved_storage":0,"accounts":1,"ha_assets":21,"api":{"total":402,"errors":0},"streams":4,"consumers":37,"messages":54,"bytes":82423,"meta_cluster":{"name":"nats","leader":"nats-0","peer":"S1Nunr6R","cluster_size":3},"account_details":[{"name":"$G","id":"$G","memory":0,"storage":170836,"reserved_memory":18446744073709551615,"reserved_storage":18446744073709551615,"accounts":0,"ha_assets":0,"api":{"total":3925,"errors":1534},"stream_detail":[{"name":"patents","created":"2024-03-26T08:28:39.073905939Z","cluster":{"name":"nats","leader":"nats-0","replicas":[{"name":"nats-0","current":true,"active":18234253,"lag":140,"peer":"S1Nunr6R"},{"name":"nats-1","current":false,"active":0,"lag":140,"peer":"yrzKKRBu"}]},"state":{"messages":2,"bytes":4368,"first_seq":132,"first_ts":"2024-03-28T13:29:15.885738777Z","last_seq":133,"last_ts":"2024-03-28T13:30:16.757064146Z","num_subjects":2,"consumer_count":22}},{"name":"projects","created":"2024-03-26T08:28:40.114880952Z","cluster":{"name":"nats","leader":"nats-0","replicas":[{"name":"nats-0","current":true,"active":106668419,"lag":10,"peer":"S1Nunr6R"},{"name":"nats-1","current":false,"active":0,"lag":10,"peer":"yrzKKRBu"}]},"state":{"messages":0,"bytes":0,"first_seq":3,"first_ts":"0001-01-01T00:00:00Z","last_seq":2,"last_ts":"2024-03-28T09:47:53.891784594Z","consumer_count":1}},{"name":"documents","created":"2024-03-26T08:28:41.189083867Z","cluster":{"name":"nats","leader":"nats-0","replicas":[{"name":"nats-0","current":true,"active":853234870,"lag":132,"peer":"S1Nunr6R"},{"name":"nats-1","current":false,"active":0,"lag":132,"peer":"yrzKKRBu"}]},"state":{"messages":4,"bytes":1622,"first_seq":102,"first_ts":"2024-03-28T13:23:45.96785193Z","last_seq":105,"last_ts":"2024-03-28T13:24:05.277823925Z","num_subjects":2,"consumer_count":8}},{"name":"chats","created":"2024-03-26T08:28:38.089896541Z","cluster":{"name":"nats","leader":"nats-2","replicas":[{"name":"nats-0","current":true,"active":385420573,"peer":"S1Nunr6R"},{"name":"nats-1","current":true,"active":385413019,"peer":"yrzKKRBu"}]},"state":{"messages":48,"bytes":76433,"first_seq":42,"first_ts":"2024-03-28T13:15:53.30222184Z","last_seq":89,"last_ts":"2024-03-28T15:11:50.812816185Z","num_subjects":2,"consumer_count":6}}]}]}}

As you can see, if I zoom on the chats stream, the state is very different depending on the replica:

After noticing this, we removed the replication from our stream, and put it back, and everything was fine afterwards, at least for now.

To resume:

Even though we managed to get out of it, this is still an issue so I hope this will help you reproduce and then fix the problem.

derekcollison commented 5 months ago

2.9.x is no longer supported. We encourage you to upgrade your system to the latest 2.10 series of the server. If the issue persists please let us know.