A node that has a remote source removed from a local stream still receives messages from the remote stream until full stop & restart

[kylebernhardy]

Defect

A node that has a remote source removed from a local stream still receives messages from the remote stream until full stop & restart. We need to be able to dynamically remove sourced streams without forcing a server outage to no longer receive unwanted messages. Important to note that a reload does not stop receiving messages, only a full stop & restart.

Make sure that these boxes are checked before submitting your issue -- thank you!

• [ X] Included nats-server -DV output
• [ X] Included a [Minimal, Complete, and Verifiable example] (https://stackoverflow.com/help/mcve)

nats-sourcing-issue.zip The above zip has all configs and -DV output for all servers.

Versions of nats-server and affected client libraries used:

using nats-server v2.7.4

OS/Container environment:

Ubuntu 20.04

Steps or code to reproduce the issue:

Launch 3 servers based on configs from attached zip: nats-server -c hub.json nats-server -c aleaf.json nats-server -c bleaf.json

Create queue stream on a-leaf: nats --server nats://localhost:4111 str add --config ./queue.json

Create test stream on b-leaf: nats --server nats://localhost:2111 str add --config ./test.json

Add entries to test stream: nats --server nats://localhost:2111 pub test hello --count 10

See entries in test stream: nats --server nats://localhost:2111 str report

Add test as source to queue: nats --server nats://localhost:4111 pub '$JS.a-leaf.API.STREAM.UPDATE.queue' '{ "name": "queue", "subjects": [ "queue" ], "retention": "limits", "max_consumers": -1, "max_msgs": -1, "max_bytes": -1, "max_age": 0, "max_msgs_per_subject": -1, "max_msg_size": -1, "discard": "old", "storage": "file", "num_replicas": 1, "duplicate_window": 120000000000, "sealed": false, "deny_delete": false, "deny_purge": false, "allow_rollup_hdrs": false, "sources": [ { "name": "test", "external": { "api": "$JS.b-leaf.API", "deliver": "" } } ] }'

Verify entries are in queue: nats --server nats://localhost:4111 str report

Add more entries to test: nats --server nats://localhost:2111 pub test hello --count 10

Verify entries are in queue: nats --server nats://localhost:4111 str report

Remove source: nats --server nats://hdb:admin@localhost:4111 pub '$JS.a-leaf.API.STREAM.UPDATE.queue' '{ "name": "queue", "subjects": [ "queue" ], "retention": "limits", "max_consumers": -1, "max_msgs": -1, "max_bytes": -1, "max_age": 0, "max_msgs_per_subject": -1, "max_msg_size": -1, "discard": "old", "storage": "file", "num_replicas": 1, "duplicate_window": 120000000000, "sealed": false, "deny_delete": false, "deny_purge": false, "allow_rollup_hdrs": false }'

Add more entries to test nats --server nats://localhost:2111 pub test hello --count 10

See entries are incorrectly queue nats --server nats://localhost:4111 str report

Stop / start a-leaf server

Add more entries to test: nats --server nats://localhost:2111 pub test hello --count 10

Now new entries are not syncing: nats --server nats://localhost:4111 str report

Expected result:

We would expect the stream sourcing from a remote stream to no longer receive messages after a stream update that removes the remote stream without full stop & restart.

Actual result:

We continue to receive messages until the server is fully stopped & restarted.

[kozlovic]

@kylebernhardy Sorry for the delay and thank you so much for the detailed issue report. We are able to reproduce the issue and will let you know as soon as possible our findings.

[kylebernhardy]

Thanks so much!

On Mon, Mar 21, 2022 at 11:40 AM Ivan Kozlovic @.***> wrote:

@kylebernhardy https://github.com/kylebernhardy Sorry for the delay and thank you so much for the detailed issue report. We are able to reproduce the issue and will let you know as soon as possible our findings.

— Reply to this email directly, view it on GitHub https://github.com/nats-io/nats-server/issues/2920#issuecomment-1074056905, or unsubscribe https://github.com/notifications/unsubscribe-auth/AGCHZ2ONCHF2JZKWCFSP2OLVBCJ73ANCNFSM5QP6DPNQ . You are receiving this because you were mentioned.Message ID: @.***>

--

Kyle Bernhardy CTO m: 303.587.2736 w: harperdb.io e: @.** Simplicity without Sacrifice*

--

This transmission may contain information that is proprietary, privileged, confidential, and/or exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or use of the information contained herein (including any reliance thereon) is STRICTLY PROHIBITED. If you received this transmission in error, please immediately contact the sender and destroy the material in its entirety, whether in electronic or hard copy format.