Closed bet-me closed 1 year ago
JetStream tracks 2 sequence numbers, one for the stream, which should never change if a message is redelivered, and the consumer sequence which will always start at 1 and increase by 1 for each message delivered.
I believe my 3 replicas have different sequence numbers for the same published message
We have not seen any reports of that.
How do you access the sequence numbers? Can you share code for that?
What does nats stream info
show?
If you pull the message directly from the stream with the NATS cli what does it show?
That's what I've done in the seq.txt above
Please show stream info, instead of attachine txt files paste the output here surrounded by 3 backticks.
Sequence Info, highlighting the different sequence numbers returned for the same command
nats % nats --user=acc --password=acc -s ws://nats:8443 stream view markets --subject=markets.football.modified.1%2E202512975 | grep -i subject
[19910] Subject: markets.football.modified.1%2E202512975 Received: 2023-04-17T11:23:40+01:00
Nats-Expected-Last-Subject-Sequence: 0
[35972] Subject: markets.football.modified.1%2E202512975 Received: 2023-04-17T11:29:07+01:00
Nats-Expected-Last-Subject-Sequence: 19910
[190696] Subject: markets.football.modified.1%2E202512975 Received: 2023-04-18T14:35:33+01:00
Nats-Expected-Last-Subject-Sequence: 35972
[244149] Subject: markets.football.modified.1%2E202512975 Received: 2023-04-18T22:55:18+01:00
Nats-Expected-Last-Subject-Sequence: 190696
[390095] Subject: markets.football.modified.1%2E202512975 Received: 2023-05-08T15:05:01+01:00
Nats-Expected-Last-Subject-Sequence: 244149
[447953] Subject: markets.football.modified.1%2E202512975 Received: 2023-05-15T17:13:42+01:00
Nats-Expected-Last-Subject-Sequence: 390095
11:57:07 Reached apparent end of data
nats --user=acc --password=acc -s ws://nats:8443 stream view markets --subject=markets.football.modified.1%2E202512975 | grep -i subject
[19910] Subject: markets.football.modified.1%2E202512975 Received: 2023-04-17T11:23:40+01:00
Nats-Expected-Last-Subject-Sequence: 0
[35972] Subject: markets.football.modified.1%2E202512975 Received: 2023-04-17T11:29:07+01:00
Nats-Expected-Last-Subject-Sequence: 19910
[190696] Subject: markets.football.modified.1%2E202512975 Received: 2023-04-18T14:35:33+01:00
Nats-Expected-Last-Subject-Sequence: 35972
[244149] Subject: markets.football.modified.1%2E202512975 Received: 2023-04-18T22:55:18+01:00
Nats-Expected-Last-Subject-Sequence: 190696
[390088] Subject: markets.football.modified.1%2E202512975 Received: 2023-05-08T15:05:01+01:00
Nats-Expected-Last-Subject-Sequence: 244149
[447961] Subject: markets.football.modified.1%2E202512975 Received: 2023-05-15T17:13:42+01:00
Nats-Expected-Last-Subject-Sequence: 390095
11:57:28 Reached apparent end of data
nats --user=acc --password=acc -s ws://nats:8443 stream view markets --subject=markets.football.modified.1%2E202512975 | grep -i subject
[19910] Subject: markets.football.modified.1%2E202512975 Received: 2023-04-17T11:23:40+01:00
Nats-Expected-Last-Subject-Sequence: 0
[35972] Subject: markets.football.modified.1%2E202512975 Received: 2023-04-17T11:29:07+01:00
Nats-Expected-Last-Subject-Sequence: 19910
[190696] Subject: markets.football.modified.1%2E202512975 Received: 2023-04-18T14:35:33+01:00
Nats-Expected-Last-Subject-Sequence: 35972
[244149] Subject: markets.football.modified.1%2E202512975 Received: 2023-04-18T22:55:18+01:00
Nats-Expected-Last-Subject-Sequence: 190696
[390095] Subject: markets.football.modified.1%2E202512975 Received: 2023-05-08T15:05:01+01:00
Nats-Expected-Last-Subject-Sequence: 244149
[447964] Subject: markets.football.modified.1%2E202512975 Received: 2023-05-15T17:13:42+01:00
Nats-Expected-Last-Subject-Sequence: 390095
11:57:32 Reached apparent end of data
Stream info
nats stream info markets
Information for Stream markets created 2023-04-17 10:00:58
Subjects: markets.*.*.*.*
Replicas: 3
Storage: File
Options:
Retention: Limits
Acknowledgements: true
Discard Policy: Old
Duplicate Window: 2m0s
Allows Msg Delete: true
Allows Purge: true
Allows Rollups: false
Limits:
Maximum Messages: unlimited
Maximum Per Subject: unlimited
Maximum Bytes: unlimited
Maximum Age: unlimited
Maximum Message Size: unlimited
Maximum Consumers: unlimited
Cluster Information:
Name: betfair
Leader: nats-1
Replica: nats-0, current, seen 0.62s ago
Replica: nats-2, current, seen 0.62s ago
State:
Messages: 464,584
Bytes: 165 MiB
FirstSeq: 1 @ 2023-04-17T10:22:56 UTC
LastSeq: 464,584 @ 2023-05-17T09:12:52 UTC
Active Consumers: 8
Number of Subjects: 213,562
Server report
nats server report jetstream
╭───────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ JetStream Summary │
├─────────┬─────────┬─────────┬─────────┬───────────┬────────────┬─────────┬────────┬─────────┬───────────┬─────────┤
│ Server │ Cluster │ Domain │ Streams │ Consumers │ Messages │ Bytes │ Memory │ File │ API Req │ API Err │
├─────────┼─────────┼─────────┼─────────┼───────────┼────────────┼─────────┼────────┼─────────┼───────────┼─────────┤
│ nats-0 │ betfair │ betfair │ 5 │ 3 │ 5,976,511 │ 2.1 GiB │ 0 B │ 2.1 GiB │ 283,106 │ 0 │
│ nats-1 │ betfair │ betfair │ 5 │ 5 │ 5,976,522 │ 2.1 GiB │ 0 B │ 2.1 GiB │ 493,324 │ 0 │
│ nats-2* │ betfair │ betfair │ 5 │ 3 │ 5,976,523 │ 2.1 GiB │ 0 B │ 2.1 GiB │ 501,421 │ 2 │
├─────────┼─────────┼─────────┼─────────┼───────────┼────────────┼─────────┼────────┼─────────┼───────────┼─────────┤
│ │ │ │ 15 │ 11 │ 17,929,556 │ 6.4 GiB │ 0 B │ 6.4 GiB │ 1,277,851 │ 2 │
╰─────────┴─────────┴─────────┴─────────┴───────────┴────────────┴─────────┴────────┴─────────┴───────────┴─────────╯
╭──────────────────────────────────────────────────────────────╮
│ RAFT Meta Group Information │
├────────┬──────────┬────────┬─────────┬────────┬────────┬─────┤
│ Name │ ID │ Leader │ Current │ Online │ Active │ Lag │
├────────┼──────────┼────────┼─────────┼────────┼────────┼─────┤
│ nats-0 │ S1Nunr6R │ │ true │ true │ 0.83s │ 0 │
│ nats-1 │ yrzKKRBu │ │ true │ true │ 0.83s │ 0 │
│ nats-2 │ cnrtt3eg │ yes │ true │ true │ 0.00s │ 0 │
╰────────┴──────────┴────────┴─────────┴────────┴────────┴─────╯
I see what you are pointing out.
Could you run the stream view commands again with --trace
on the end?
Thanks.
meta, err := jsm.ParseJSMsgMetadata(msg)
if err == nil {
fmt.Printf("[%d] Subject: %s Received: %s\n", meta.StreamSequence(), msg.Subject, meta.TimeStamp().Format(time.RFC3339))
} else {
fmt.Printf("Subject: %s Reply: %s\n", msg.Subject, msg.Reply)
}
from stream view, so does seem to be a changing sequence there.
447953
nats --user=acc --password=acc stream view markets --subject=markets.betfair.football.modified.1%2E202512975 --trace
15:54:47 >>> $JS.API.STREAM.INFO.markets
null
15:54:47 <<< $JS.API.STREAM.INFO.markets
{"type":"io.nats.jetstream.api.v1.stream_info_response","total":0,"offset":0,"limit":0,"config":{"name":"markets","subjects":["markets.*.*.*.*"],"retention":"limits","max_consumers":-1,"max_msgs":-1,"max_bytes":-1,"max_age":0,"max_msgs_per_subject":-1,"max_msg_size":-1,"discard":"old","storage":"file","num_replicas":3,"duplicate_window":120000000000,"allow_direct":false,"mirror_direct":false,"sealed":false,"deny_delete":false,"deny_purge":false,"allow_rollup_hdrs":false},"created":"2023-04-17T10:00:58.301389806Z","state":{"messages":464584,"bytes":173279821,"first_seq":1,"first_ts":"2023-04-17T10:22:56.367357515Z","last_seq":464584,"last_ts":"2023-05-17T09:12:52.065857331Z","num_subjects":213562,"consumer_count":8},"domain":"betfair","cluster":{"name":"betfair","leader":"nats-1","replicas":[{"name":"nats-0","current":true,"active":399430902,"peer":"S1Nunr6R"},{"name":"nats-2","current":true,"active":399447734,"peer":"cnrtt3eg"}]}}
15:54:47 >>> $JS.API.CONSUMER.DURABLE.CREATE.markets.stream_pager_8221684338887605086539
{"stream_name":"markets","config":{"description":"JSM Stream Pager","ack_policy":"explicit","ack_wait":30000000000,"deliver_policy":"all","durable_name":"stream_pager_8221684338887605086539","filter_subject":"markets.betfair.football.modified.1%2E202512975","replay_policy":"instant","inactive_threshold":3600000000000,"num_replicas":0}}
15:54:47 <<< $JS.API.CONSUMER.DURABLE.CREATE.markets.stream_pager_8221684338887605086539
{"type":"io.nats.jetstream.api.v1.consumer_create_response","stream_name":"markets","name":"stream_pager_8221684338887605086539","created":"2023-05-17T15:54:47.606019887Z","config":{"durable_name":"stream_pager_8221684338887605086539","description":"JSM Stream Pager","deliver_policy":"all","ack_policy":"explicit","ack_wait":30000000000,"max_deliver":-1,"filter_subject":"markets.betfair.football.modified.1%2E202512975","replay_policy":"instant","max_waiting":512,"max_ack_pending":1000,"inactive_threshold":3600000000000,"num_replicas":0},"delivered":{"consumer_seq":0,"stream_seq":0},"ack_floor":{"consumer_seq":0,"stream_seq":0},"num_ack_pending":0,"num_redelivered":0,"num_waiting":0,"num_pending":6,"cluster":{"name":"betfair","leader":"nats-0","replicas":[{"name":"nats-1","current":true,"active":38674265,"peer":"yrzKKRBu"},{"name":"nats-2","current":true,"active":39236009,"peer":"cnrtt3eg"}]}}
[19910] Subject: markets.betfair.football.modified.1%2E202512975 Received: 2023-04-17T10:23:40Z
Nats-Expected-Last-Subject-Sequence: 0
R?
78601P??ȯ?????>???????ǡ????????뾊????GIBRALTAR REGULATOR?? UK wallet???<!--Football - Cup - To Reach the Final - Generic --><br> Will the named team reach the Final of this competition?<br> At the start of the tournament all unmatched bets will be cancelled and the market will be reopened.The market will then remain open until the start of the 1st semi-final where unmatched bets will be cancelled and the market turned in play.<br><br></b>For further information please see <a href=http://content.betfair.com/aboutus/content.asp?sWhichKey=Rules%20and%20Regulations#undefined.do style=color:0163ad; text-decoration: underline; target=_blank>Rules & Regs<br><br>
?@CLASSIC?@228?A
??
betting_type
competition_id
fixture_id
in_play_enabled
market_time
operator_id
persistence_enabled
price_ladder_type
published
regulator
rules
suspend_time
wallet
[35972] Subject: markets.betfair.football.modified.1%2E202512975 Received: 2023-04-17T10:29:07Z
Nats-Expected-Last-Subject-Sequence: 19910
R#P????ì??>?>??@ ?A??գ?A
0?r
complete
key_line_selections
market_status
number_of_active_runners
number_of_selections
published
version
[190696] Subject: markets.betfair.football.modified.1%2E202512975 Received: 2023-04-18T13:35:33Z
Nats-Expected-Last-Subject-Sequence: 35972
RP??????«?A???
published
version
[244149] Subject: markets.betfair.football.modified.1%2E202512975 Received: 2023-04-18T21:55:18Z
Nats-Expected-Last-Subject-Sequence: 190696
RP?????ɫ???A?????.
number_of_active_runners
published
version
[390095] Subject: markets.betfair.football.modified.1%2E202512975 Received: 2023-05-08T14:05:01Z
Nats-Expected-Last-Subject-Sequence: 244149
RP????̮͂???A????.
number_of_active_runners
published
version
[447953] Subject: markets.betfair.football.modified.1%2E202512975 Received: 2023-05-15T16:13:42Z
Nats-Expected-Last-Subject-Sequence: 390095
RP??ݏ??ׯ????A?????(
bet_delay
in_play
published
version
15:54:48 Reached apparent end of data
15:54:48 >>> $JS.API.CONSUMER.DELETE.markets.stream_pager_8221684338887605086539
15:54:48 <<< $JS.API.CONSUMER.DELETE.markets.stream_pager_8221684338887605086539
{"type":"io.nats.jetstream.api.v1.consumer_delete_response","success":true}
447961
nats --user=acc --password=acc stream view markets --subject=markets.betfair.football.modified.1%2E202512975 --trace
15:55:41 >>> $JS.API.STREAM.INFO.markets
null
15:55:41 <<< $JS.API.STREAM.INFO.markets
{"type":"io.nats.jetstream.api.v1.stream_info_response","total":0,"offset":0,"limit":0,"config":{"name":"markets","subjects":["markets.*.*.*.*"],"retention":"limits","max_consumers":-1,"max_msgs":-1,"max_bytes":-1,"max_age":0,"max_msgs_per_subject":-1,"max_msg_size":-1,"discard":"old","storage":"file","num_replicas":3,"duplicate_window":120000000000,"allow_direct":false,"mirror_direct":false,"sealed":false,"deny_delete":false,"deny_purge":false,"allow_rollup_hdrs":false},"created":"2023-04-17T10:00:58.301389806Z","state":{"messages":464584,"bytes":173279821,"first_seq":1,"first_ts":"2023-04-17T10:22:56.367357515Z","last_seq":464584,"last_ts":"2023-05-17T09:12:52.065857331Z","num_subjects":213562,"consumer_count":8},"domain":"betfair","cluster":{"name":"betfair","leader":"nats-1","replicas":[{"name":"nats-0","current":true,"active":549078080,"peer":"S1Nunr6R"},{"name":"nats-2","current":true,"active":549401309,"peer":"cnrtt3eg"}]}}
15:55:41 >>> $JS.API.CONSUMER.DURABLE.CREATE.markets.stream_pager_8571684338941746979172
{"stream_name":"markets","config":{"description":"JSM Stream Pager","ack_policy":"explicit","ack_wait":30000000000,"deliver_policy":"all","durable_name":"stream_pager_8571684338941746979172","filter_subject":"markets.betfair.football.modified.1%2E202512975","replay_policy":"instant","inactive_threshold":3600000000000,"num_replicas":0}}
15:55:42 <<< $JS.API.CONSUMER.DURABLE.CREATE.markets.stream_pager_8571684338941746979172
{"type":"io.nats.jetstream.api.v1.consumer_create_response","stream_name":"markets","name":"stream_pager_8571684338941746979172","created":"2023-05-17T15:55:41.747893747Z","config":{"durable_name":"stream_pager_8571684338941746979172","description":"JSM Stream Pager","deliver_policy":"all","ack_policy":"explicit","ack_wait":30000000000,"max_deliver":-1,"filter_subject":"markets.betfair.football.modified.1%2E202512975","replay_policy":"instant","max_waiting":512,"max_ack_pending":1000,"inactive_threshold":3600000000000,"num_replicas":0},"delivered":{"consumer_seq":0,"stream_seq":0},"ack_floor":{"consumer_seq":0,"stream_seq":0},"num_ack_pending":0,"num_redelivered":0,"num_waiting":0,"num_pending":6,"cluster":{"name":"betfair","leader":"nats-1","replicas":[{"name":"nats-0","current":true,"active":41170693,"peer":"S1Nunr6R"},{"name":"nats-2","current":true,"active":41246692,"peer":"cnrtt3eg"}]}}
[19910] Subject: markets.betfair.football.modified.1%2E202512975 Received: 2023-04-17T10:23:40Z
Nats-Expected-Last-Subject-Sequence: 0
R?
78601P??ȯ?????>???????ǡ????????뾊????GIBRALTAR REGULATOR?? UK wallet???<!--Football - Cup - To Reach the Final - Generic --><br> Will the named team reach the Final of this competition?<br> At the start of the tournament all unmatched bets will be cancelled and the market will be reopened.The market will then remain open until the start of the 1st semi-final where unmatched bets will be cancelled and the market turned in play.<br><br></b>For further information please see <a href=http://content.betfair.com/aboutus/content.asp?sWhichKey=Rules%20and%20Regulations#undefined.do style=color:0163ad; text-decoration: underline; target=_blank>Rules & Regs<br><br>
?@CLASSIC?@228?A
??
betting_type
competition_id
fixture_id
in_play_enabled
market_time
operator_id
persistence_enabled
price_ladder_type
published
regulator
rules
suspend_time
wallet
[35972] Subject: markets.betfair.football.modified.1%2E202512975 Received: 2023-04-17T10:29:07Z
Nats-Expected-Last-Subject-Sequence: 19910
R#P????ì??>?>??@ ?A??գ?A
0?r
complete
key_line_selections
market_status
number_of_active_runners
number_of_selections
published
version
[190696] Subject: markets.betfair.football.modified.1%2E202512975 Received: 2023-04-18T13:35:33Z
Nats-Expected-Last-Subject-Sequence: 35972
RP??????«?A???
published
version
[244149] Subject: markets.betfair.football.modified.1%2E202512975 Received: 2023-04-18T21:55:18Z
Nats-Expected-Last-Subject-Sequence: 190696
RP?????ɫ???A?????.
number_of_active_runners
published
version
[390088] Subject: markets.betfair.football.modified.1%2E202512975 Received: 2023-05-08T14:05:01Z
Nats-Expected-Last-Subject-Sequence: 244149
RP????̮͂???A????.
number_of_active_runners
published
version
[447961] Subject: markets.betfair.football.modified.1%2E202512975 Received: 2023-05-15T16:13:42Z
Nats-Expected-Last-Subject-Sequence: 390095
RP??ݏ??ׯ????A?????(
bet_delay
in_play
published
version
15:55:42 Reached apparent end of data
15:55:42 >>> $JS.API.CONSUMER.DELETE.markets.stream_pager_8571684338941746979172
15:55:42 <<< $JS.API.CONSUMER.DELETE.markets.stream_pager_8571684338941746979172
{"type":"io.nats.jetstream.api.v1.consumer_delete_response","success":true}
447964
nats --user=acc --password=acc stream view markets --subject=markets.betfair.football.modified.1%2E202512975 --trace
15:56:54 >>> $JS.API.STREAM.INFO.markets
null
15:56:54 <<< $JS.API.STREAM.INFO.markets
{"type":"io.nats.jetstream.api.v1.stream_info_response","total":0,"offset":0,"limit":0,"config":{"name":"markets","subjects":["markets.*.*.*.*"],"retention":"limits","max_consumers":-1,"max_msgs":-1,"max_bytes":-1,"max_age":0,"max_msgs_per_subject":-1,"max_msg_size":-1,"discard":"old","storage":"file","num_replicas":3,"duplicate_window":120000000000,"allow_direct":false,"mirror_direct":false,"sealed":false,"deny_delete":false,"deny_purge":false,"allow_rollup_hdrs":false},"created":"2023-04-17T10:00:58.301389806Z","state":{"messages":464584,"bytes":173279821,"first_seq":1,"first_ts":"2023-04-17T10:22:56.367357515Z","last_seq":464584,"last_ts":"2023-05-17T09:12:52.065857331Z","num_subjects":213562,"consumer_count":8},"domain":"betfair","cluster":{"name":"betfair","leader":"nats-1","replicas":[{"name":"nats-0","current":true,"active":289126162,"peer":"S1Nunr6R"},{"name":"nats-2","current":true,"active":289306118,"peer":"cnrtt3eg"}]}}
15:56:54 >>> $JS.API.CONSUMER.DURABLE.CREATE.markets.stream_pager_9671684339014512786058
{"stream_name":"markets","config":{"description":"JSM Stream Pager","ack_policy":"explicit","ack_wait":30000000000,"deliver_policy":"all","durable_name":"stream_pager_9671684339014512786058","filter_subject":"markets.betfair.football.modified.1%2E202512975","replay_policy":"instant","inactive_threshold":3600000000000,"num_replicas":0}}
15:56:55 <<< $JS.API.CONSUMER.DURABLE.CREATE.markets.stream_pager_9671684339014512786058
{"type":"io.nats.jetstream.api.v1.consumer_create_response","stream_name":"markets","name":"stream_pager_9671684339014512786058","created":"2023-05-17T15:56:54.51345781Z","config":{"durable_name":"stream_pager_9671684339014512786058","description":"JSM Stream Pager","deliver_policy":"all","ack_policy":"explicit","ack_wait":30000000000,"max_deliver":-1,"filter_subject":"markets.betfair.football.modified.1%2E202512975","replay_policy":"instant","max_waiting":512,"max_ack_pending":1000,"inactive_threshold":3600000000000,"num_replicas":0},"delivered":{"consumer_seq":0,"stream_seq":0},"ack_floor":{"consumer_seq":0,"stream_seq":0},"num_ack_pending":0,"num_redelivered":0,"num_waiting":0,"num_pending":6,"cluster":{"name":"betfair","leader":"nats-2","replicas":[{"name":"nats-0","current":true,"active":42854562,"peer":"S1Nunr6R"},{"name":"nats-1","current":true,"active":42813137,"peer":"yrzKKRBu"}]}}
[19910] Subject: markets.betfair.football.modified.1%2E202512975 Received: 2023-04-17T10:23:40Z
Nats-Expected-Last-Subject-Sequence: 0
R?
78601P??ȯ?????>???????ǡ????????뾊????GIBRALTAR REGULATOR?? UK wallet???<!--Football - Cup - To Reach the Final - Generic --><br> Will the named team reach the Final of this competition?<br> At the start of the tournament all unmatched bets will be cancelled and the market will be reopened.The market will then remain open until the start of the 1st semi-final where unmatched bets will be cancelled and the market turned in play.<br><br></b>For further information please see <a href=http://content.betfair.com/aboutus/content.asp?sWhichKey=Rules%20and%20Regulations#undefined.do style=color:0163ad; text-decoration: underline; target=_blank>Rules & Regs<br><br>
?@CLASSIC?@228?A
??
betting_type
competition_id
fixture_id
in_play_enabled
market_time
operator_id
persistence_enabled
price_ladder_type
published
regulator
rules
suspend_time
wallet
[35972] Subject: markets.betfair.football.modified.1%2E202512975 Received: 2023-04-17T10:29:07Z
Nats-Expected-Last-Subject-Sequence: 19910
R#P????ì??>?>??@ ?A??գ?A
0?r
complete
key_line_selections
market_status
number_of_active_runners
number_of_selections
published
version
[190696] Subject: markets.betfair.football.modified.1%2E202512975 Received: 2023-04-18T13:35:33Z
Nats-Expected-Last-Subject-Sequence: 35972
RP??????«?A???
published
version
[244149] Subject: markets.betfair.football.modified.1%2E202512975 Received: 2023-04-18T21:55:18Z
Nats-Expected-Last-Subject-Sequence: 190696
RP?????ɫ???A?????.
number_of_active_runners
published
version
[390095] Subject: markets.betfair.football.modified.1%2E202512975 Received: 2023-05-08T14:05:01Z
Nats-Expected-Last-Subject-Sequence: 244149
RP????̮͂???A????.
number_of_active_runners
published
version
[447964] Subject: markets.betfair.football.modified.1%2E202512975 Received: 2023-05-15T16:13:42Z
Nats-Expected-Last-Subject-Sequence: 390095
RP??ݏ??ׯ????A?????(
bet_delay
in_play
published
version
15:56:55 Reached apparent end of data
15:56:55 >>> $JS.API.CONSUMER.DELETE.markets.stream_pager_9671684339014512786058
15:56:55 <<< $JS.API.CONSUMER.DELETE.markets.stream_pager_9671684339014512786058
{"type":"io.nats.jetstream.api.v1.consumer_delete_response","success":true}
Will take a closer look later today when I have some more free time.
Thanks, if you need anything else let me know.
Could you do the following for us and send us the output?
nats req "$JS.API.STREAM.INFO.markets" "" --count=3 --trace
nats --user=acc --password=acc req "$JS.API.STREAM.INFO.markets" "" --count=3 --trace
19:24:54 Sending request on ".API.STREAM.INFO.markets"
19:24:54 No responders are available
nats req "\$JS.API.STREAM.INFO.markets" "" --count=3 --trace
Escape the $
nats --user=sys --password=sys req "\$JS.API.STREAM.INFO.markets" "" --count=3 --trace
19:36:26 Sending request on "$JS.API.STREAM.INFO.markets"
19:36:31 Sending request on "$JS.API.STREAM.INFO.markets"
19:36:36 Sending request on "$JS.API.STREAM.INFO.markets"
Sorry replace --count=3
with `--replies=3
And needs to be from user=acc
nats --user=acc --password=acc req "\$JS.API.STREAM.INFO.markets" "" --replies=3 --trace
19:42:40 Sending request on "$JS.API.STREAM.INFO.markets"
19:42:40 Received with rtt 1.248412ms
{"type":"io.nats.jetstream.api.v1.stream_info_response","total":0,"offset":0,"limit":0,"config":{"name":"markets","subjects":["markets.*.*.*.*"],"retention":"limits","max_consumers":-1,"max_msgs":-1,"max_bytes":-1,"max_age":0,"max_msgs_per_subject":-1,"max_msg_size":-1,"discard":"old","storage":"file","num_replicas":3,"duplicate_window":120000000000,"allow_direct":false,"mirror_direct":false,"sealed":false,"deny_delete":false,"deny_purge":false,"allow_rollup_hdrs":false},"created":"2023-04-17T10:00:58.301389806Z","state":{"messages":472758,"bytes":175666816,"first_seq":1,"first_ts":"2023-04-17T10:22:56.367357515Z","last_seq":472758,"last_ts":"2023-05-17T19:17:07.847835235Z","num_subjects":221245,"consumer_count":8},"domain":"betfair","cluster":{"name":"betfair","leader":"nats-1","replicas":[{"name":"nats-0","current":true,"active":767913601,"peer":"S1Nunr6R"},{"name":"nats-2","current":true,"active":768030544,"peer":"cnrtt3eg"}]}}
~ #
So you only got one response, not three?
Yes
nats --user=acc --password=acc req "\$JS.API.STREAM.INFO.markets" "" --count=3 --replies=3 --trace
20:04:19 Sending request on "$JS.API.STREAM.INFO.markets"
20:04:19 Received with rtt 45.934387ms
{"type":"io.nats.jetstream.api.v1.stream_info_response","total":0,"offset":0,"limit":0,"config":{"name":"markets","subjects":["markets.*.*.*.*"],"retention":"limits","max_consumers":-1,"max_msgs":-1,"max_bytes":-1,"max_age":0,"max_msgs_per_subject":-1,"max_msg_size":-1,"discard":"old","storage":"file","num_replicas":3,"duplicate_window":120000000000,"allow_direct":false,"mirror_direct":false,"sealed":false,"deny_delete":false,"deny_purge":false,"allow_rollup_hdrs":false},"created":"2023-04-17T10:00:58.301389806Z","state":{"messages":472758,"bytes":175666816,"first_seq":1,"first_ts":"2023-04-17T10:22:56.367357515Z","last_seq":472758,"last_ts":"2023-05-17T19:17:07.847835235Z","num_subjects":221245,"consumer_count":8},"domain":"betfair","cluster":{"name":"betfair","leader":"nats-1","replicas":[{"name":"nats-0","current":true,"active":171676275,"peer":"S1Nunr6R"},{"name":"nats-2","current":true,"active":171673871,"peer":"cnrtt3eg"}]}}
20:04:19 Sending request on "$JS.API.STREAM.INFO.markets"
20:04:19 Received with rtt 1.158968ms
{"type":"io.nats.jetstream.api.v1.stream_info_response","total":0,"offset":0,"limit":0,"config":{"name":"markets","subjects":["markets.*.*.*.*"],"retention":"limits","max_consumers":-1,"max_msgs":-1,"max_bytes":-1,"max_age":0,"max_msgs_per_subject":-1,"max_msg_size":-1,"discard":"old","storage":"file","num_replicas":3,"duplicate_window":120000000000,"allow_direct":false,"mirror_direct":false,"sealed":false,"deny_delete":false,"deny_purge":false,"allow_rollup_hdrs":false},"created":"2023-04-17T10:00:58.301389806Z","state":{"messages":472758,"bytes":175666816,"first_seq":1,"first_ts":"2023-04-17T10:22:56.367357515Z","last_seq":472758,"last_ts":"2023-05-17T19:17:07.847835235Z","num_subjects":221245,"consumer_count":8},"domain":"betfair","cluster":{"name":"betfair","leader":"nats-1","replicas":[{"name":"nats-0","current":true,"active":564485812,"peer":"S1Nunr6R"},{"name":"nats-2","current":true,"active":564483408,"peer":"cnrtt3eg"}]}}
20:04:20 Sending request on "$JS.API.STREAM.INFO.markets"
20:04:20 Received with rtt 849.332µs
{"type":"io.nats.jetstream.api.v1.stream_info_response","total":0,"offset":0,"limit":0,"config":{"name":"markets","subjects":["markets.*.*.*.*"],"retention":"limits","max_consumers":-1,"max_msgs":-1,"max_bytes":-1,"max_age":0,"max_msgs_per_subject":-1,"max_msg_size":-1,"discard":"old","storage":"file","num_replicas":3,"duplicate_window":120000000000,"allow_direct":false,"mirror_direct":false,"sealed":false,"deny_delete":false,"deny_purge":false,"allow_rollup_hdrs":false},"created":"2023-04-17T10:00:58.301389806Z","state":{"messages":472758,"bytes":175666816,"first_seq":1,"first_ts":"2023-04-17T10:22:56.367357515Z","last_seq":472758,"last_ts":"2023-05-17T19:17:07.847835235Z","num_subjects":221245,"consumer_count":8},"domain":"betfair","cluster":{"name":"betfair","leader":"nats-1","replicas":[{"name":"nats-0","current":true,"active":867439175,"peer":"S1Nunr6R"},{"name":"nats-2","current":true,"active":867436771,"peer":"cnrtt3eg"}]}}
~ #
Added --count
No need, just send request once.
I will check with @ripienaar that command should work.
One thing to possible retry is to reset cluster state. You can do that by scaling down to 1 and back up to an R3.
nats --user=acc --password=acc stream update markets --replicas=1
then
nats --user=acc --password=acc stream update markets --replicas=3
Won’t that destroy any evidence as to what caused the issue?
Before doing that, please do this:
nats server req jsz --streams
using a system account.
This will produce a result from each of your servers and we can see the per-node state in that output. You can edit it a bit if you like just be sure to give us the stream detail for the stream in question from each server.
nats --user=sys --password=sys server req jsz --streams
{"server":{"name":"nats-2","host":"0.0.0.0","id":"NCPNFREBCNWIA6LRM3FQX2WA2UELDKIEELLS2SAOQA2IAZ7UJPYEG4EL","cluster":"betfair","domain":"betfair","ver":"2.9.16","seq":1512128,"jetstream":true,"time":"2023-05-17T20:37:47.520579364Z"},"data":{"server_id":"NCPNFREBCNWIA6LRM3FQX2WA2UELDKIEELLS2SAOQA2IAZ7UJPYEG4EL","now":"2023-05-17T20:37:47.520478298Z","config":{"max_memory":274877906944,"max_storage":274877906944,"store_dir":"/data/jetstream","domain":"betfair","compress_ok":true},"memory":0,"storage":2296566156,"reserved_memory":0,"reserved_storage":0,"accounts":1,"ha_assets":7,"api":{"total":732547,"errors":5},"streams":5,"consumers":3,"messages":5986788,"bytes":2296566156,"meta_cluster":{"name":"betfair","leader":"nats-2","peer":"cnrtt3eg","replicas":[{"name":"nats-0","current":true,"active":215158552,"peer":"S1Nunr6R"},{"name":"nats-1","current":true,"active":215099140,"peer":"yrzKKRBu"}],"cluster_size":3},"account_details":[{"name":"js","id":"js","memory":0,"storage":6889688786,"reserved_memory":18446744073709551615,"reserved_storage":18446744073709551615,"accounts":0,"ha_assets":0,"api":{"total":1693092,"errors":5},"stream_detail":[{"name":"competitions","created":"2023-04-17T10:00:00.82445078Z","cluster":{"name":"betfair","leader":"nats-1","replicas":[{"name":"nats-0","current":false,"active":103958345516068,"lag":151557,"peer":"S1Nunr6R"},{"name":"nats-1","current":true,"active":333683571,"lag":151557,"peer":"yrzKKRBu"}]},"state":{"messages":707,"bytes":190907,"first_seq":1,"first_ts":"2023-04-17T10:22:56.286020673Z","last_seq":707,"last_ts":"2023-05-17T18:17:43.10541073Z","num_subjects":706,"consumer_count":0}},{"name":"competitors","created":"2023-04-17T10:00:17.485123561Z","cluster":{"name":"betfair","leader":"nats-2","replicas":[{"name":"nats-0","current":true,"active":333153715,"peer":"S1Nunr6R"},{"name":"nats-1","current":true,"active":333206988,"peer":"yrzKKRBu"}]},"state":{"messages":4394,"bytes":1144364,"first_seq":1,"first_ts":"2023-04-17T10:22:56.852403698Z","last_seq":4394,"last_ts":"2023-05-17T17:36:34.25858227Z","num_subjects":4394,"consumer_count":0}},{"name":"fixtures","created":"2023-04-17T10:00:41.391129394Z","cluster":{"name":"betfair","leader":"nats-2","replicas":[{"name":"nats-0","current":true,"active":333173054,"peer":"S1Nunr6R"},{"name":"nats-1","current":true,"active":333204984,"peer":"yrzKKRBu"}]},"state":{"messages":12973,"bytes":3238619,"first_seq":1,"first_ts":"2023-04-17T10:22:56.443136498Z","last_seq":12973,"last_ts":"2023-05-17T19:17:07.814048089Z","num_subjects":10505,"consumer_count":1}},{"name":"markets","created":"2023-04-17T10:00:58.301389806Z","cluster":{"name":"betfair","leader":"nats-1","replicas":[{"name":"nats-0","current":false,"active":103958345580829,"lag":537478,"peer":"S1Nunr6R"},{"name":"nats-1","current":true,"active":333664074,"lag":537478,"peer":"yrzKKRBu"}]},"state":{"messages":472761,"bytes":175667648,"first_seq":1,"first_ts":"2023-04-17T10:22:56.367357515Z","last_seq":472761,"last_ts":"2023-05-17T19:17:07.847835235Z","num_subjects":221247,"consumer_count":2}},{"name":"selections","created":"2023-04-17T10:01:15.569409667Z","cluster":{"name":"betfair","leader":"nats-2","replicas":[{"name":"nats-0","current":true,"active":542234865,"peer":"S1Nunr6R"},{"name":"nats-1","current":true,"active":542337204,"peer":"yrzKKRBu"}]},"state":{"messages":5495953,"bytes":2116324618,"first_seq":1,"first_ts":"2023-04-17T10:22:56.368479378Z","last_seq":5495953,"last_ts":"2023-05-17T18:51:30.878707017Z","num_subjects":1040358,"consumer_count":0}}]}]}}
{"server":{"name":"nats-1","host":"0.0.0.0","id":"NAJPFQLLVOCB5BDZ4NX3CKUVDQLOTCB42FDNHUDASGQXZRL6D3JHTOCC","cluster":"betfair","domain":"betfair","ver":"2.9.16","seq":1513068,"jetstream":true,"time":"2023-05-17T20:37:47.520541169Z"},"data":{"server_id":"NAJPFQLLVOCB5BDZ4NX3CKUVDQLOTCB42FDNHUDASGQXZRL6D3JHTOCC","now":"2023-05-17T20:37:47.520405834Z","config":{"max_memory":274877906944,"max_storage":274877906944,"store_dir":"/data/jetstream","domain":"betfair","compress_ok":true},"memory":0,"storage":2296556392,"reserved_memory":0,"reserved_storage":0,"accounts":1,"ha_assets":7,"api":{"total":585675,"errors":0},"streams":5,"consumers":5,"messages":5986787,"bytes":2296556392,"meta_cluster":{"name":"betfair","leader":"nats-2","peer":"cnrtt3eg","cluster_size":3},"account_details":[{"name":"js","id":"js","memory":0,"storage":6889688786,"reserved_memory":18446744073709551615,"reserved_storage":18446744073709551615,"accounts":0,"ha_assets":0,"api":{"total":1693097,"errors":5},"stream_detail":[{"name":"competitions","created":"2023-04-17T10:00:00.82445078Z","cluster":{"name":"betfair","leader":"nats-1","replicas":[{"name":"nats-0","current":true,"active":333494289,"peer":"S1Nunr6R"},{"name":"nats-2","current":true,"active":333545472,"peer":"cnrtt3eg"}]},"state":{"messages":707,"bytes":190907,"first_seq":1,"first_ts":"2023-04-17T10:22:56.286020673Z","last_seq":707,"last_ts":"2023-05-17T18:17:43.10541073Z","num_subjects":706,"consumer_count":0}},{"name":"competitors","created":"2023-04-17T10:00:17.485123561Z","cluster":{"name":"betfair","leader":"nats-2","replicas":[{"name":"nats-0","current":false,"active":103958345519601,"lag":22416,"peer":"S1Nunr6R"},{"name":"nats-2","current":true,"active":333522140,"lag":22416,"peer":"cnrtt3eg"}]},"state":{"messages":4394,"bytes":1144364,"first_seq":1,"first_ts":"2023-04-17T10:22:56.852403698Z","last_seq":4394,"last_ts":"2023-05-17T17:36:34.25858227Z","num_subjects":4394,"consumer_count":0}},{"name":"fixtures","created":"2023-04-17T10:00:41.391129394Z","cluster":{"name":"betfair","leader":"nats-2","replicas":[{"name":"nats-0","current":false,"active":103958345652044,"lag":15807,"peer":"S1Nunr6R"},{"name":"nats-2","current":true,"active":333499467,"lag":15807,"peer":"cnrtt3eg"}]},"state":{"messages":12973,"bytes":3238562,"first_seq":1,"first_ts":"2023-04-17T10:22:56.443136498Z","last_seq":12973,"last_ts":"2023-05-17T19:17:07.814048089Z","num_subjects":10505,"consumer_count":1}},{"name":"markets","created":"2023-04-17T10:00:58.301389806Z","cluster":{"name":"betfair","leader":"nats-1","replicas":[{"name":"nats-0","current":true,"active":333529791,"peer":"S1Nunr6R"},{"name":"nats-2","current":true,"active":333528317,"peer":"cnrtt3eg"}]},"state":{"messages":472758,"bytes":175666816,"first_seq":1,"first_ts":"2023-04-17T10:22:56.367357515Z","last_seq":472758,"last_ts":"2023-05-17T19:17:07.847835235Z","num_subjects":221245,"consumer_count":4}},{"name":"selections","created":"2023-04-17T10:01:15.569409667Z","cluster":{"name":"betfair","leader":"nats-2","replicas":[{"name":"nats-0","current":false,"active":0,"lag":5499666,"peer":"S1Nunr6R"},{"name":"nats-2","current":true,"active":542617166,"lag":5499666,"peer":"cnrtt3eg"}]},"state":{"messages":5495955,"bytes":2116315743,"first_seq":1,"first_ts":"2023-04-17T10:22:56.368479378Z","last_seq":5495955,"last_ts":"2023-05-17T18:51:30.878707017Z","num_subjects":1040358,"consumer_count":0}}]}]}}
{"server":{"name":"nats-0","host":"0.0.0.0","id":"NDCWTKX2JMEWCRQYFJALJYTIKIOAEIKUXPSM2XMJUW6H5VSLZXLWCC5P","cluster":"betfair","domain":"betfair","ver":"2.9.16","seq":1504372,"jetstream":true,"time":"2023-05-17T20:37:47.521175151Z"},"data":{"server_id":"NDCWTKX2JMEWCRQYFJALJYTIKIOAEIKUXPSM2XMJUW6H5VSLZXLWCC5P","now":"2023-05-17T20:37:47.521036757Z","config":{"max_memory":274877906944,"max_storage":274877906944,"store_dir":"/data/jetstream","domain":"betfair","compress_ok":true},"memory":0,"storage":2296566238,"reserved_memory":0,"reserved_storage":0,"accounts":1,"ha_assets":7,"api":{"total":374884,"errors":0},"streams":5,"consumers":3,"messages":5986776,"bytes":2296566238,"meta_cluster":{"name":"betfair","leader":"nats-2","peer":"cnrtt3eg","cluster_size":3},"account_details":[{"name":"js","id":"js","memory":0,"storage":6889688786,"reserved_memory":18446744073709551615,"reserved_storage":18446744073709551615,"accounts":0,"ha_assets":0,"api":{"total":1693097,"errors":5},"stream_detail":[{"name":"selections","created":"2023-04-17T10:01:15.569409667Z","cluster":{"name":"betfair","leader":"nats-2","replicas":[{"name":"nats-1","current":false,"active":0,"lag":5499666,"peer":"yrzKKRBu"},{"name":"nats-2","current":true,"active":542610718,"lag":5499666,"peer":"cnrtt3eg"}]},"state":{"messages":5495953,"bytes":2116324618,"first_seq":1,"first_ts":"2023-04-17T10:22:56.368479378Z","last_seq":5495953,"last_ts":"2023-05-17T18:51:30.878707017Z","num_subjects":1040358,"consumer_count":0}},{"name":"competitions","created":"2023-04-17T10:00:00.82445078Z","cluster":{"name":"betfair","leader":"nats-1","replicas":[{"name":"nats-1","current":true,"active":333793492,"lag":151557,"peer":"yrzKKRBu"},{"name":"nats-2","current":false,"active":0,"lag":151557,"peer":"cnrtt3eg"}]},"state":{"messages":707,"bytes":190907,"first_seq":1,"first_ts":"2023-04-17T10:22:56.286020673Z","last_seq":707,"last_ts":"2023-05-17T18:17:43.10541073Z","num_subjects":706,"consumer_count":0}},{"name":"competitors","created":"2023-04-17T10:00:17.485123561Z","cluster":{"name":"betfair","leader":"nats-2","replicas":[{"name":"nats-1","current":false,"active":0,"lag":22416,"peer":"yrzKKRBu"},{"name":"nats-2","current":true,"active":333439417,"lag":22416,"peer":"cnrtt3eg"}]},"state":{"messages":4394,"bytes":1144364,"first_seq":1,"first_ts":"2023-04-17T10:22:56.852403698Z","last_seq":4394,"last_ts":"2023-05-17T17:36:34.25858227Z","num_subjects":4394,"consumer_count":0}},{"name":"fixtures","created":"2023-04-17T10:00:41.391129394Z","cluster":{"name":"betfair","leader":"nats-2","replicas":[{"name":"nats-1","current":false,"active":0,"lag":15807,"peer":"yrzKKRBu"},{"name":"nats-2","current":true,"active":333450818,"lag":15807,"peer":"cnrtt3eg"}]},"state":{"messages":12972,"bytes":3238383,"first_seq":1,"first_ts":"2023-04-17T10:22:56.443136498Z","last_seq":12972,"last_ts":"2023-05-17T19:17:07.814048089Z","num_subjects":10505,"consumer_count":1}},{"name":"markets","created":"2023-04-17T10:00:58.301389806Z","cluster":{"name":"betfair","leader":"nats-1","replicas":[{"name":"nats-1","current":true,"active":333807628,"lag":537478,"peer":"yrzKKRBu"},{"name":"nats-2","current":false,"active":0,"lag":537478,"peer":"cnrtt3eg"}]},"state":{"messages":472750,"bytes":175667966,"first_seq":1,"first_ts":"2023-04-17T10:22:56.367357515Z","last_seq":472750,"last_ts":"2023-05-17T19:17:07.847835235Z","num_subjects":221247,"consumer_count":2}}]}]}}
~ #
~ #
So they seemed skewed that is true, unclear as to why this might be.
The replica scale down and back up will sync all replicas to the leader's state.
If you want another state, you can have the leader stepdown to select the leader you want.
Will close for now but if needed feel free to reopen.
Hello! I'm adding a few details, because this also happened to us this week. The config is similar:
We shut off our staging environment every night, and put it back on every morning.
On our AKS environment we first encountered what seems to be the same issue as https://github.com/nats-io/nats-server/issues/4410, where we'd come back one morning and every streams of the deployment were gone.
To continue our work, we recreated the streams. But then, we started to run into issues where the sequence number of our consumers for their referred stream were higher than the sequence number of the stream itself, leading to no message consumption.
Hence, we removed our consumers and recreated them again. Everything was fine, at a first look. But then, we noticed one strange behaviour: ephemeral consumers would not have the same sequence number for their referred stream depending on the replica they were created on.
This lead us to the current issue. Running the command above give the following result:
{"server":{"name":"nats-1","host":"0.0.0.0","id":"NBIVM4G76KZKUJCGJFW4V5USRRFG4I7P42EJFOGXY66XVSCWWHEPXPPS","cluster":"nats","ver":"2.9.16","seq":5929,"jetstream":true,"time":"2024-03-28T16:11:30.848918256Z"},"data":{"server_id":"NBIVM4G76KZKUJCGJFW4V5USRRFG4I7P42EJFOGXY66XVSCWWHEPXPPS","now":"2024-03-28T16:11:30.848880846Z","config":{"max_memory":429916160,"max_storage":10737418240,"store_dir":"/data/jetstream","compress_ok":true},"memory":0,"storage":82423,"reserved_memory":0,"reserved_storage":0,"accounts":1,"ha_assets":21,"api":{"total":381,"errors":0},"streams":4,"consumers":34,"messages":54,"bytes":82423,"meta_cluster":{"name":"nats","leader":"nats-0","peer":"S1Nunr6R","cluster_size":3},"account_details":[{"name":"$G","id":"$G","memory":0,"storage":170836,"reserved_memory":18446744073709551615,"reserved_storage":18446744073709551615,"accounts":0,"ha_assets":0,"api":{"total":3925,"errors":1534},"stream_detail":[{"name":"chats","created":"2024-03-26T08:28:38.089896541Z","cluster":{"name":"nats","leader":"nats-2","replicas":[{"name":"nats-0","current":false,"active":0,"lag":274,"peer":"S1Nunr6R"},{"name":"nats-2","current":true,"active":384081563,"lag":274,"peer":"cnrtt3eg"}]},"state":{"messages":48,"bytes":76433,"first_seq":42,"first_ts":"2024-03-28T13:15:53.30222184Z","last_seq":89,"last_ts":"2024-03-28T15:11:50.812816185Z","num_subjects":2,"consumer_count":7}},{"name":"patents","created":"2024-03-26T08:28:39.073905939Z","cluster":{"name":"nats","leader":"nats-0","replicas":[{"name":"nats-0","current":true,"active":16718271,"lag":140,"peer":"S1Nunr6R"},{"name":"nats-2","current":false,"active":0,"lag":140,"peer":"cnrtt3eg"}]},"state":{"messages":2,"bytes":4368,"first_seq":132,"first_ts":"2024-03-28T13:29:15.885738777Z","last_seq":133,"last_ts":"2024-03-28T13:30:16.757064146Z","num_subjects":2,"consumer_count":15}},{"name":"projects","created":"2024-03-26T08:28:40.114880952Z","cluster":{"name":"nats","leader":"nats-0","replicas":[{"name":"nats-0","current":true,"active":105179332,"lag":10,"peer":"S1Nunr6R"},{"name":"nats-2","current":false,"active":0,"lag":10,"peer":"cnrtt3eg"}]},"state":{"messages":0,"bytes":0,"first_seq":3,"first_ts":"0001-01-01T00:00:00Z","last_seq":2,"last_ts":"2024-03-28T09:47:53.891784594Z","consumer_count":1}},{"name":"documents","created":"2024-03-26T08:28:41.189083867Z","cluster":{"name":"nats","leader":"nats-0","replicas":[{"name":"nats-0","current":true,"active":851645761,"lag":132,"peer":"S1Nunr6R"},{"name":"nats-2","current":false,"active":0,"lag":132,"peer":"cnrtt3eg"}]},"state":{"messages":4,"bytes":1622,"first_seq":102,"first_ts":"2024-03-28T13:23:45.96785193Z","last_seq":105,"last_ts":"2024-03-28T13:24:05.277823925Z","num_subjects":2,"consumer_count":11}}]}]}}
{"server":{"name":"nats-0","host":"0.0.0.0","id":"NB4ZZWU2XJJ6SKVQA5OANUWNPOAAQXYD344HNUY3VGTALLYS3CJ5UMWC","cluster":"nats","ver":"2.9.16","seq":5618,"jetstream":true,"time":"2024-03-28T16:11:30.850751323Z"},"data":{"server_id":"NB4ZZWU2XJJ6SKVQA5OANUWNPOAAQXYD344HNUY3VGTALLYS3CJ5UMWC","now":"2024-03-28T16:11:30.850710477Z","config":{"max_memory":429916160,"max_storage":10737418240,"store_dir":"/data/jetstream","compress_ok":true},"memory":0,"storage":5990,"reserved_memory":0,"reserved_storage":0,"accounts":1,"ha_assets":21,"api":{"total":3142,"errors":1534},"streams":4,"consumers":41,"messages":6,"bytes":5990,"meta_cluster":{"name":"nats","leader":"nats-0","peer":"S1Nunr6R","replicas":[{"name":"nats-1","current":true,"active":628168645,"peer":"yrzKKRBu"},{"name":"nats-2","current":true,"active":628356044,"peer":"cnrtt3eg"}],"cluster_size":3},"account_details":[{"name":"$G","id":"$G","memory":0,"storage":170836,"reserved_memory":18446744073709551615,"reserved_storage":18446744073709551615,"accounts":0,"ha_assets":0,"api":{"total":3925,"errors":1534},"stream_detail":[{"name":"projects","created":"2024-03-26T08:28:40.114880952Z","cluster":{"name":"nats","leader":"nats-0","replicas":[{"name":"nats-1","current":true,"active":106509007,"peer":"yrzKKRBu"},{"name":"nats-2","current":true,"active":106562446,"peer":"cnrtt3eg"}]},"state":{"messages":0,"bytes":0,"first_seq":3,"first_ts":"0001-01-01T00:00:00Z","last_seq":2,"last_ts":"2024-03-28T09:47:53.891784594Z","consumer_count":1}},{"name":"chats","created":"2024-03-26T08:28:38.089896541Z","cluster":{"name":"nats","leader":"nats-2","replicas":[{"name":"nats-1","current":false,"active":0,"lag":274,"peer":"yrzKKRBu"},{"name":"nats-2","current":true,"active":385544726,"lag":274,"peer":"cnrtt3eg"}]},"state":{"messages":0,"bytes":0,"first_seq":177,"first_ts":"0001-01-01T00:00:00Z","last_seq":176,"last_ts":"2024-03-27T16:42:31.470009169Z","consumer_count":12}},{"name":"documents","created":"2024-03-26T08:28:41.189083867Z","cluster":{"name":"nats","leader":"nats-0","replicas":[{"name":"nats-1","current":true,"active":852729457,"peer":"yrzKKRBu"},{"name":"nats-2","current":true,"active":852717444,"peer":"cnrtt3eg"}]},"state":{"messages":4,"bytes":1622,"first_seq":102,"first_ts":"2024-03-28T13:23:45.96785193Z","last_seq":105,"last_ts":"2024-03-28T13:24:05.277823925Z","num_subjects":2,"consumer_count":10}},{"name":"patents","created":"2024-03-26T08:28:39.073905939Z","cluster":{"name":"nats","leader":"nats-0","replicas":[{"name":"nats-1","current":true,"active":18025479,"peer":"yrzKKRBu"},{"name":"nats-2","current":true,"active":18151524,"peer":"cnrtt3eg"}]},"state":{"messages":2,"bytes":4368,"first_seq":132,"first_ts":"2024-03-28T13:29:15.885738777Z","last_seq":133,"last_ts":"2024-03-28T13:30:16.757064146Z","num_subjects":2,"consumer_count":18}}]}]}}
{"server":{"name":"nats-2","host":"0.0.0.0","id":"NDLDBCQXSL7KITGNWNS72HL72KWOVZR27RLH3ZJTHZNEOSOBUBTLJVL3","cluster":"nats","ver":"2.9.16","seq":6037,"jetstream":true,"time":"2024-03-28T16:11:30.849909719Z"},"data":{"server_id":"NDLDBCQXSL7KITGNWNS72HL72KWOVZR27RLH3ZJTHZNEOSOBUBTLJVL3","now":"2024-03-28T16:11:30.849868162Z","config":{"max_memory":429916160,"max_storage":10737418240,"store_dir":"/data/jetstream","compress_ok":true},"memory":0,"storage":82423,"reserved_memory":0,"reserved_storage":0,"accounts":1,"ha_assets":21,"api":{"total":402,"errors":0},"streams":4,"consumers":37,"messages":54,"bytes":82423,"meta_cluster":{"name":"nats","leader":"nats-0","peer":"S1Nunr6R","cluster_size":3},"account_details":[{"name":"$G","id":"$G","memory":0,"storage":170836,"reserved_memory":18446744073709551615,"reserved_storage":18446744073709551615,"accounts":0,"ha_assets":0,"api":{"total":3925,"errors":1534},"stream_detail":[{"name":"patents","created":"2024-03-26T08:28:39.073905939Z","cluster":{"name":"nats","leader":"nats-0","replicas":[{"name":"nats-0","current":true,"active":18234253,"lag":140,"peer":"S1Nunr6R"},{"name":"nats-1","current":false,"active":0,"lag":140,"peer":"yrzKKRBu"}]},"state":{"messages":2,"bytes":4368,"first_seq":132,"first_ts":"2024-03-28T13:29:15.885738777Z","last_seq":133,"last_ts":"2024-03-28T13:30:16.757064146Z","num_subjects":2,"consumer_count":22}},{"name":"projects","created":"2024-03-26T08:28:40.114880952Z","cluster":{"name":"nats","leader":"nats-0","replicas":[{"name":"nats-0","current":true,"active":106668419,"lag":10,"peer":"S1Nunr6R"},{"name":"nats-1","current":false,"active":0,"lag":10,"peer":"yrzKKRBu"}]},"state":{"messages":0,"bytes":0,"first_seq":3,"first_ts":"0001-01-01T00:00:00Z","last_seq":2,"last_ts":"2024-03-28T09:47:53.891784594Z","consumer_count":1}},{"name":"documents","created":"2024-03-26T08:28:41.189083867Z","cluster":{"name":"nats","leader":"nats-0","replicas":[{"name":"nats-0","current":true,"active":853234870,"lag":132,"peer":"S1Nunr6R"},{"name":"nats-1","current":false,"active":0,"lag":132,"peer":"yrzKKRBu"}]},"state":{"messages":4,"bytes":1622,"first_seq":102,"first_ts":"2024-03-28T13:23:45.96785193Z","last_seq":105,"last_ts":"2024-03-28T13:24:05.277823925Z","num_subjects":2,"consumer_count":8}},{"name":"chats","created":"2024-03-26T08:28:38.089896541Z","cluster":{"name":"nats","leader":"nats-2","replicas":[{"name":"nats-0","current":true,"active":385420573,"peer":"S1Nunr6R"},{"name":"nats-1","current":true,"active":385413019,"peer":"yrzKKRBu"}]},"state":{"messages":48,"bytes":76433,"first_seq":42,"first_ts":"2024-03-28T13:15:53.30222184Z","last_seq":89,"last_ts":"2024-03-28T15:11:50.812816185Z","num_subjects":2,"consumer_count":6}}]}]}}
As you can see, if I zoom on the chats
stream, the state is very different depending on the replica:
"state": {
"messages": 0,
"bytes": 0,
"first_seq": 177,
"first_ts": "0001-01-01T00:00:00Z",
"last_seq": 176,
"last_ts": "2024-03-27T16:42:31.470009169Z",
"consumer_count": 12
}
"state": {
"messages": 48,
"bytes": 76433,
"first_seq": 42,
"first_ts": "2024-03-28T13:15:53.30222184Z",
"last_seq": 89,
"last_ts": "2024-03-28T15:11:50.812816185Z",
"num_subjects": 2,
"consumer_count": 7
}
"state": {
"messages": 48,
"bytes": 76433,
"first_seq": 42,
"first_ts": "2024-03-28T13:15:53.30222184Z",
"last_seq": 89,
"last_ts": "2024-03-28T15:11:50.812816185Z",
"num_subjects": 2,
"consumer_count": 6
}
After noticing this, we removed the replication from our stream, and put it back, and everything was fine afterwards, at least for now.
To resume:
Even though we managed to get out of it, this is still an issue so I hope this will help you reproduce and then fix the problem.
2.9.x is no longer supported. We encourage you to upgrade your system to the latest 2.10 series of the server. If the issue persists please let us know.
I've got NATS running in Kubernetes with Jetstream enabled
nats:2.9.16 replicas: 3 k8s: v1.25.2
I'm using Jetstream for event sourcing and I'm storing the sequence numbers in a cache to keep track of what messages I've processed.
If I ask for the messages in a subject multiple times, I sometimes get different sequence numbers.
In this case the last sequence numbers alternate between 447953, 447961, 447964
Should NATS Jetstream return the same sequence numbers each time?
seq.txt