Messages getting dropped during network outage when leaf node stream with work queue policy is aggregated on remote stream

bjurr commented 2 years ago

Defect

When a leaf node has a stream configured with work queue policy and this stream is aggregated on a remote stream, there is a chance messages will be silently dropped if a network outage occurs between the leaf node and the cluster (especially obvious using wifi).

Versions of `nats-server` and affected client libraries used:

nats-server: 2.8.4
natscli: 0.0.33

OS/Container environment:

Server:

Fedora36 X86_64
Docker version 20.10.17, build 100c701
using official nats:latest docker hub image

Client/Leaf node:

Tried both MacOS BigSur 11.5.2 and Windows11 WSL Ubuntu 20.04
Docker version 20.10.14, build a224086
using official nats:latest docker hub image

Steps or code to reproduce the issue:

Testing this is a somewhat lengthy manual process.

Start a main server that has JetStream enabled, and can accept leaf node connections with the following sample configuration (save as nats.config):
```
server_name: "hub-server"
jetstream {
domain=hub
}
port: 4222
leafnodes {
port: 7422
}
monitor_port: 8222
```

docker run --rm -ti -v $(pwd)/nats.config:/etc/nats/nats-server.conf --name=nats-server -p 4222:4222 -p 6222:6222 -p 8222:8222 -p 7422:7422 nats -m 8222 -c /etc/nats/nats-server.conf

On a separate computer, launch a leaf node with JetStream enabled using the following sample configuration (save as nats-leaf.config):

listen: "0.0.0.0:4222"
server_name: leaf-server1
jetstream {
        domain = leaf1
}
leafnodes {
    remotes = [
        {
          url: "nats-leaf://<ip-of-remote-server>:7422"
        },
    ]
}

docker run --rm -ti -v $(pwd)/nats-leaf.config:/etc/nats/nats-server.conf --name=nats-leaf1 -p 4222:4222 -p 6222:6222 -p 8222:8222 nats -m 8222 -c /etc/nats/nats-server.conf

Create the leaf node work queue stream using the following configuration (save it as leaf-stream.json):

{
"name": "test1",
"subjects": [
"sub1"
],
"retention": "workqueue",
"max_consumers": -1,
"max_msgs_per_subject": -1,
"max_msgs": -1,
"max_bytes": -1,
"max_age": 0,
"max_msg_size": -1,
"storage": "memory",
"discard": "old",
"num_replicas": 1,
"duplicate_window": 120000000000,
"sealed": false,
"deny_delete": false,
"deny_purge": false,
"allow_rollup_hdrs": false
}

nats --server=127.0.0.1 stream add --config=leaf-stream.json

Create the remote aggregating stream using the following configuration (save it as remote-stream.json):

{
"name": "agg-test1",
"subjects": [
"sub1"
],
"retention": "workqueue",
"max_consumers": -1,
"max_msgs_per_subject": -1,
"max_msgs": -1,
"max_bytes": -1,
"max_age": 0,
"max_msg_size": -1,
"storage": "memory",
"discard": "old",
"num_replicas": 1,
"duplicate_window": 120000000000,
"sources": [
{
  "name": "test1",
  "external": {
    "api": "$JS.leaf1.API",
    "deliver": ""
  }
}
],
"sealed": false,
"deny_delete": false,
"deny_purge": false,
"allow_rollup_hdrs": false
}

nats --server=<ip-of-remote-server> stream add --config=remote-stream.json

From the same computer running the leaf node, publish a test message and confirm that the message has been synchronized on the remote stream:
```
nats --server=127.0.0.1 pub sub1 "message 1"
nats --server=127.0.0.1 stream report
nats --server=<ip-of-remote-server> stream report
```
Simulate network outage. This is best done using a wifi enabled computer. Assuming the computer running the leaf node and publisher is connected on wifi, shut off wifi, and quickly publish a new message and confirm message is NOT persisted in the stream:
```
nats --server=127.0.0.1 pub sub1 "message 2"
nats --server=127.0.0.1 stream report
```
After some time (up to about 30 seconds), the leaf node will start printing error messages ([ERR] Error trying to connect as leafnode to remote server). At that point, publish a new message and confirm that the message has been persisted in the stream:
```
nats --server=127.0.0.1 pub sub1 "message 3"
nats --server=127.0.0.1 stream report
```
Reconnect wifi, wait for connectivity, verify that the persisted message is synchronized on the remote stream but notice the total message count=2 (when it should be 3):
```
nats --server=127.0.0.1 stream report
nats --server=<ip-of-remote-server> stream report
```

Expected result:

Messages should be persisted on the leaf node stream when the remote stream is not reachable.

Actual result:

Messages destined for the leaf node stream are dropped until the leaf node realize it is disconnected from the cluster. Only at that point will messages start being persisted in the stream.

derekcollison commented 2 years ago

Currently we use R1 ephemerals which default to an idle cleanup of 5s IIRC.

My thought here would be to allow this to be configurable for downstream streams when configuring a mirror or source.

PKuchibhatla commented 1 year ago

has this been reolved

derekcollison commented 1 year ago

No extended outages of the leafnode connection could cause issues since the message may be deleted locally if interest policy based.

kuriboww commented 8 months ago

This appears to be an issue I am facing as well. Please investigate with network latency and jitter.

leandrofars commented 1 month ago

Facing the same issue here

leandrofars commented 1 month ago

Is there a way to keep messages in the stream of a leafnode even when lost connection with remote nats-server?

jnmoyne commented 1 month ago

Is there a way to keep messages in the stream of a leafnode even when lost connection with remote nats-server?

Yes: use a limit (rather than work queue or interest) stream to source from, e.g. set a 'max age' limit that is as long as the longest network outage that you want to be able to recover from.

nats-io / nats-server