Closed kylebernhardy closed 2 years ago
@kylebernhardy Sorry for the delay and thank you so much for the detailed issue report. We are able to reproduce the issue and will let you know as soon as possible our findings.
Thanks so much!
On Mon, Mar 21, 2022 at 11:40 AM Ivan Kozlovic @.***> wrote:
@kylebernhardy https://github.com/kylebernhardy Sorry for the delay and thank you so much for the detailed issue report. We are able to reproduce the issue and will let you know as soon as possible our findings.
— Reply to this email directly, view it on GitHub https://github.com/nats-io/nats-server/issues/2920#issuecomment-1074056905, or unsubscribe https://github.com/notifications/unsubscribe-auth/AGCHZ2ONCHF2JZKWCFSP2OLVBCJ73ANCNFSM5QP6DPNQ . You are receiving this because you were mentioned.Message ID: @.***>
--
Kyle Bernhardy CTO m: 303.587.2736 w: harperdb.io e: @.** Simplicity without Sacrifice*
--
This transmission may contain information that is proprietary, privileged, confidential, and/or exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or use of the information contained herein (including any reliance thereon) is STRICTLY PROHIBITED. If you received this transmission in error, please immediately contact the sender and destroy the material in its entirety, whether in electronic or hard copy format.
Defect
A node that has a remote source removed from a local stream still receives messages from the remote stream until full stop & restart. We need to be able to dynamically remove sourced streams without forcing a server outage to no longer receive unwanted messages. Important to note that a reload does not stop receiving messages, only a full stop & restart.
Make sure that these boxes are checked before submitting your issue -- thank you!
nats-server -DV
outputnats-sourcing-issue.zip The above zip has all configs and -DV output for all servers.
Versions of
nats-server
and affected client libraries used:using nats-server v2.7.4
OS/Container environment:
Ubuntu 20.04
Steps or code to reproduce the issue:
Launch 3 servers based on configs from attached zip:
nats-server -c hub.json
nats-server -c aleaf.json
nats-server -c bleaf.json
Create queue stream on a-leaf:
nats --server nats://localhost:4111 str add --config ./queue.json
Create test stream on b-leaf:
nats --server nats://localhost:2111 str add --config ./test.json
Add entries to test stream:
nats --server nats://localhost:2111 pub test hello --count 10
See entries in test stream:
nats --server nats://localhost:2111 str report
Add test as source to queue:
nats --server nats://localhost:4111 pub '$JS.a-leaf.API.STREAM.UPDATE.queue' '{ "name": "queue", "subjects": [ "queue" ], "retention": "limits", "max_consumers": -1, "max_msgs": -1, "max_bytes": -1, "max_age": 0, "max_msgs_per_subject": -1, "max_msg_size": -1, "discard": "old", "storage": "file", "num_replicas": 1, "duplicate_window": 120000000000, "sealed": false, "deny_delete": false, "deny_purge": false, "allow_rollup_hdrs": false, "sources": [ { "name": "test", "external": { "api": "$JS.b-leaf.API", "deliver": "" } } ] }'
Verify entries are in queue:
nats --server nats://localhost:4111 str report
Add more entries to test:
nats --server nats://localhost:2111 pub test hello --count 10
Verify entries are in queue:
nats --server nats://localhost:4111 str report
Remove source:
nats --server nats://hdb:admin@localhost:4111 pub '$JS.a-leaf.API.STREAM.UPDATE.queue' '{ "name": "queue", "subjects": [ "queue" ], "retention": "limits", "max_consumers": -1, "max_msgs": -1, "max_bytes": -1, "max_age": 0, "max_msgs_per_subject": -1, "max_msg_size": -1, "discard": "old", "storage": "file", "num_replicas": 1, "duplicate_window": 120000000000, "sealed": false, "deny_delete": false, "deny_purge": false, "allow_rollup_hdrs": false }'
Add more entries to test
nats --server nats://localhost:2111 pub test hello --count 10
See entries are incorrectly queue
nats --server nats://localhost:4111 str report
Stop / start a-leaf server
Add more entries to test:
nats --server nats://localhost:2111 pub test hello --count 10
Now new entries are not syncing:
nats --server nats://localhost:4111 str report
Expected result:
We would expect the stream sourcing from a remote stream to no longer receive messages after a stream update that removes the remote stream without full stop & restart.
Actual result:
We continue to receive messages until the server is fully stopped & restarted.