search

nats-io / natscli

The NATS Command Line Interface
Apache License 2.0
479 stars 95 forks source link

"nats str ls" shows duplicate stream names #825

Closed goku321 closed 1 year ago

goku321 commented 1 year ago

What happened

  1. I created 10 R3 streams, stream1... stream10 using the below command 
    for i in $(seq 1 1 10)
do
nats str add "stream""$i" \
--subjects "stream""$i.*" \
--replicas 3 \
--storage file \
--retention limits \
--max-msg-size=-1 \
--discard old \
--max-msgs=-1 \
--max-bytes=-1 \
--max-age="2h" \
--max-msgs-per-subject=-1 \
--dupe-window="1m" \
--no-allow-rollup \
--no-deny-delete \
--deny-purge \
--server tls://nats-dev.nats-dev.svc.cluster.local:443 \
--creds /etc/nats-config/creds/cpaas-stream.creds
done
  2. nats str ls showed duplicate streams for a brief period of time before showing the correct output Screenshot 2023-07-13 at 1 43 58 PM

Expected nats str ls should have showed "stream3" and "stream6" in the output as well instead of showing "stream10" and "stream9" twice.

I believe that the issue is in the nats-server because the CLI is simply showing what it receives from the server.

NATS server version: 2.9.19 NATS CLI version: 0.0.35 3-nodes cluster

ripienaar commented 1 year ago

Indeed, if you can show us output of adding --trace to the nats invocation that would help the server team. This issue should probably be in the server repo, I'll move it once you sent --trace output.

goku321 commented 1 year ago

This happened only a few times for a very brief duration. Does the --trace output still help?

goku321 commented 1 year ago 
12:32:46 >>> $JS.API.STREAM.LIST
{"offset":0}

12:32:46 <<< $JS.API.STREAM.LIST
{"type":"io.nats.jetstream.api.v1.stream_list_response","total":12,"offset":0,"limit":256,"streams":[{"config":{"name":"stream1","subjects":["stream1.*"],"retention":"limits","max_consumers":-1,"max_msgs":-1,"max_bytes":-1,"max_age":7200000000000,"max_msgs_per_subject":-1,"max_msg_size":-1,"discard":"old","storage":"file","num_replicas":3,"duplicate_window":60000000000,"allow_direct":false,"mirror_direct":false,"sealed":false,"deny_delete":false,"deny_purge":true,"allow_rollup_hdrs":false},"created":"2023-07-13T10:30:34.930528408Z","state":{"messages":713953,"bytes":7168016688,"first_seq":1,"first_ts":"2023-07-13T10:33:03.982241672Z","last_seq":713953,"last_ts":"2023-07-13T12:32:46.747406525Z","num_subjects":100,"consumer_count":1},"cluster":{"name":"nats-dev","leader":"nats-dev-2","replicas":[{"name":"nats-dev-0","current":true,"active":2150785,"peer":"wkq8TXaT"},{"name":"nats-dev-1","current":true,"active":2070243,"peer":"fKv1R187"}]}},{"config":{"name":"stream10","subjects":["stream10.*"],"retention":"limits","max_consumers":-1,"max_msgs":-1,"max_bytes":-1,"max_age":7200000000000,"max_msgs_per_subject":-1,"max_msg_size":-1,"discard":"old","storage":"file","num_replicas":3,"duplicate_window":60000000000,"allow_direct":false,"mirror_direct":false,"sealed":false,"deny_delete":false,"deny_purge":true,"allow_rollup_hdrs":false},"created":"2023-07-13T10:30:39.599480217Z","state":{"messages":689697,"bytes":6925178577,"first_seq":1,"first_ts":"2023-07-13T10:37:07.967634276Z","last_seq":689697,"last_ts":"2023-07-13T12:32:46.731373571Z","num_subjects":100,"consumer_count":1},"cluster":{"name":"nats-dev","leader":"nats-dev-2","replicas":[{"name":"nats-dev-0","current":true,"active":18339556,"peer":"wkq8TXaT"},{"name":"nats-dev-1","current":true,"active":18246628,"peer":"fKv1R187"}]}},{"config":{"name":"stream11","subjects":["stream11.*"],"retention":"limits","max_consumers":-1,"max_msgs":-1,"max_bytes":-1,"max_age":7200000000000,"max_msgs_per_subject":-1,"max_msg_size":-1,"discard":"old","storage":"file","num_replicas":3,"duplicate_window":60000000000,"allow_direct":false,"mirror_direct":false,"sealed":false,"deny_delete":false,"deny_purge":true,"allow_rollup_hdrs":false},"created":"2023-07-13T10:45:06.615951349Z","state":{"messages":1000000,"bytes":10040900000,"first_seq":1,"first_ts":"2023-07-13T10:45:56.310006447Z","last_seq":1000000,"last_ts":"2023-07-13T11:27:55.310377056Z","num_subjects":100,"consumer_count":0},"cluster":{"name":"nats-dev","leader":"nats-dev-2","replicas":[{"name":"nats-dev-0","current":true,"active":834527165,"peer":"wkq8TXaT"},{"name":"nats-dev-1","current":true,"active":850536179,"peer":"fKv1R187"}]}},{"config":{"name":"stream12","subjects":["stream12.*"],"retention":"limits","max_consumers":-1,"max_msgs":-1,"max_bytes":-1,"max_age":7200000000000,"max_msgs_per_subject":-1,"max_msg_size":-1,"discard":"old","storage":"file","num_replicas":3,"duplicate_window":60000000000,"allow_direct":false,"mirror_direct":false,"sealed":false,"deny_delete":false,"deny_purge":true,"allow_rollup_hdrs":false},"created":"2023-07-13T10:58:41.47181809Z","state":{"messages":610000,"bytes":6124949000,"first_seq":1,"first_ts":"2023-07-13T10:59:10.373998158Z","last_seq":610000,"last_ts":"2023-07-13T12:25:27.018041885Z","num_subjects":100,"consumer_count":0},"cluster":{"name":"nats-dev","leader":"nats-dev-1","replicas":[{"name":"nats-dev-0","current":true,"active":531249556,"peer":"wkq8TXaT"},{"name":"nats-dev-2","current":true,"active":531155678,"peer":"H0ki4mNO"}]}},{"config":{"name":"stream2","subjects":["stream2.*"],"retention":"limits","max_consumers":-1,"max_msgs":-1,"max_bytes":-1,"max_age":7200000000000,"max_msgs_per_subject":-1,"max_msg_size":-1,"discard":"old","storage":"file","num_replicas":3,"duplicate_window":60000000000,"allow_direct":false,"mirror_direct":false,"sealed":false,"deny_delete":false,"deny_purge":true,"allow_rollup_hdrs":false},"created":"2023-07-13T10:30:35.594055832Z","state":{"messages":711112,"bytes":7139493320,"first_seq":1,"first_ts":"2023-07-13T10:33:33.295762607Z","last_seq":711112,"last_ts":"2023-07-13T12:32:46.746393297Z","num_subjects":100,"consumer_count":1},"cluster":{"name":"nats-dev","leader":"nats-dev-2","replicas":[{"name":"nats-dev-0","current":true,"active":2889610,"peer":"wkq8TXaT"},{"name":"nats-dev-1","current":true,"active":3168079,"peer":"fKv1R187"}]}},{"config":{"name":"stream3","subjects":["stream3.*"],"retention":"limits","max_consumers":-1,"max_msgs":-1,"max_bytes":-1,"max_age":7200000000000,"max_msgs_per_subject":-1,"max_msg_size":-1,"discard":"old","storage":"file","num_replicas":3,"duplicate_window":60000000000,"allow_direct":false,"mirror_direct":false,"sealed":false,"deny_delete":false,"deny_purge":true,"allow_rollup_hdrs":false},"created":"2023-07-13T10:30:36.336120594Z","state":{"messages":708489,"bytes":7113158660,"first_seq":1,"first_ts":"2023-07-13T10:34:00.459629992Z","last_seq":708489,"last_ts":"2023-07-13T12:32:46.748969537Z","num_subjects":100,"consumer_count":1},"cluster":{"name":"nats-dev","leader":"nats-dev-2","replicas":[{"name":"nats-dev-0","current":true,"active":610304,"peer":"wkq8TXaT"},{"name":"nats-dev-1","current":false,"active":42126722,"lag":2,"peer":"fKv1R187"}]}},{"config":{"name":"stream4","subjects":["stream4.*"],"retention":"limits","max_consumers":-1,"max_msgs":-1,"max_bytes":-1,"max_age":7200000000000,"max_msgs_per_subject":-1,"max_msg_size":-1,"discard":"old","storage":"file","num_replicas":3,"duplicate_window":60000000000,"allow_direct":false,"mirror_direct":false,"sealed":false,"deny_delete":false,"deny_purge":true,"allow_rollup_hdrs":false},"created":"2023-07-13T10:30:37.152499491Z","state":{"messages":698867,"bytes":7016554744,"first_seq":1,"first_ts":"2023-07-13T10:35:33.659831725Z","last_seq":698867,"last_ts":"2023-07-13T12:32:46.665421931Z","num_subjects":100,"consumer_count":1},"cluster":{"name":"nats-dev","leader":"nats-dev-0","replicas":[{"name":"nats-dev-1","current":true,"active":81769382,"peer":"fKv1R187"},{"name":"nats-dev-2","current":true,"active":72527492,"peer":"H0ki4mNO"}]}},{"config":{"name":"stream5","subjects":["stream5.*"],"retention":"limits","max_consumers":-1,"max_msgs":-1,"max_bytes":-1,"max_age":7200000000000,"max_msgs_per_subject":-1,"max_msg_size":-1,"discard":"old","storage":"file","num_replicas":3,"duplicate_window":60000000000,"allow_direct":false,"mirror_direct":false,"sealed":false,"deny_delete":false,"deny_purge":true,"allow_rollup_hdrs":false},"created":"2023-07-13T10:30:37.421008954Z","state":{"messages":697203,"bytes":6999848320,"first_seq":1,"first_ts":"2023-07-13T10:35:52.004543025Z","last_seq":697203,"last_ts":"2023-07-13T12:32:46.711247478Z","num_subjects":100,"consumer_count":1},"cluster":{"name":"nats-dev","leader":"nats-dev-1","replicas":[{"name":"nats-dev-0","current":true,"active":38908360,"peer":"wkq8TXaT"},{"name":"nats-dev-2","current":true,"active":47834715,"peer":"H0ki4mNO"}]}},{"config":{"name":"stream6","subjects":["stream6.*"],"retention":"limits","max_consumers":-1,"max_msgs":-1,"max_bytes":-1,"max_age":7200000000000,"max_msgs_per_subject":-1,"max_msg_size":-1,"discard":"old","storage":"file","num_replicas":3,"duplicate_window":60000000000,"allow_direct":false,"mirror_direct":false,"sealed":false,"deny_delete":false,"deny_purge":true,"allow_rollup_hdrs":false},"created":"2023-07-13T10:30:38.006371288Z","state":{"messages":695298,"bytes":6980722320,"first_seq":1,"first_ts":"2023-07-13T10:36:10.525497819Z","last_seq":695298,"last_ts":"2023-07-13T12:32:46.667782047Z","num_subjects":100,"consumer_count":1},"cluster":{"name":"nats-dev","leader":"nats-dev-1","replicas":[{"name":"nats-dev-0","current":true,"active":83846079,"peer":"wkq8TXaT"},{"name":"nats-dev-2","current":true,"active":74492088,"peer":"H0ki4mNO"}]}},{"config":{"name":"stream7","subjects":["stream7.*"],"retention":"limits","max_consumers":-1,"max_msgs":-1,"max_bytes":-1,"max_age":7200000000000,"max_msgs_per_subject":-1,"max_msg_size":-1,"discard":"old","storage":"file","num_replicas":3,"duplicate_window":60000000000,"allow_direct":false,"mirror_direct":false,"sealed":false,"deny_delete":false,"deny_purge":true,"allow_rollup_hdrs":false},"created":"2023-07-13T10:30:38.270176789Z","state":{"messages":693428,"bytes":6961947720,"first_seq":1,"first_ts":"2023-07-13T10:36:30.120163741Z","last_seq":693428,"last_ts":"2023-07-13T12:32:46.68471164Z","num_subjects":100,"consumer_count":1},"cluster":{"name":"nats-dev","leader":"nats-dev-2","replicas":[{"name":"nats-dev-0","current":true,"active":61716078,"peer":"wkq8TXaT"},{"name":"nats-dev-1","current":true,"active":42389311,"peer":"fKv1R187"}]}},{"config":{"name":"stream8","subjects":["stream8.*"],"retention":"limits","max_consumers":-1,"max_msgs":-1,"max_bytes":-1,"max_age":7200000000000,"max_msgs_per_subject":-1,"max_msg_size":-1,"discard":"old","storage":"file","num_replicas":3,"duplicate_window":60000000000,"allow_direct":false,"mirror_direct":false,"sealed":false,"deny_delete":false,"deny_purge":true,"allow_rollup_hdrs":false},"created":"2023-07-13T10:30:38.483510123Z","state":{"messages":691995,"bytes":6947560559,"first_seq":1,"first_ts":"2023-07-13T10:36:43.708683304Z","last_seq":691995,"last_ts":"2023-07-13T12:32:46.686156714Z","num_subjects":100,"consumer_count":1},"cluster":{"name":"nats-dev","leader":"nats-dev-0","replicas":[{"name":"nats-dev-1","current":true,"active":55253822,"peer":"fKv1R187"},{"name":"nats-dev-2","current":true,"active":61404487,"peer":"H0ki4mNO"}]}},{"config":{"name":"stream9","subjects":["stream9.*"],"retention":"limits","max_consumers":-1,"max_msgs":-1,"max_bytes":-1,"max_age":7200000000000,"max_msgs_per_subject":-1,"max_msg_size":-1,"discard":"old","storage":"file","num_replicas":3,"duplicate_window":60000000000,"allow_direct":false,"mirror_direct":false,"sealed":false,"deny_delete":false,"deny_purge":true,"allow_rollup_hdrs":false},"created":"2023-07-13T10:30:38.982616252Z","state":{"messages":690899,"bytes":6936556828,"first_seq":1,"first_ts":"2023-07-13T10:36:55.939677436Z","last_seq":690899,"last_ts":"2023-07-13T12:32:46.743343764Z","num_subjects":100,"consumer_count":1},"cluster":{"name":"nats-dev","leader":"nats-dev-0","replicas":[{"name":"nats-dev-1","current":false,"active":101341145,"lag":5,"peer":"fKv1R187"},{"name":"nats-dev-2","current":true,"active":3038274,"peer":"H0ki4mNO"}]}}]}
wallyqs commented 1 year ago

under what server version did this happen? how many nodes do you have in your cluster?

goku321 commented 1 year ago

NATS server: 2.9.19 NATS CLI: 0.0.35 3-nodes cluster (will edit the post and add these details)

wallyqs commented 1 year ago

@goku321 when this happens again, you can try to capture the following too to see the state stream on each replica:

# using the system account:
nats server request jetstream --streams
goku321 commented 1 year ago

@wallyqs sure, will do. Thanks for the suggestion 🤝

ripienaar commented 1 year ago

Closing this as not CLI related.