Mirrored stream stops when an invalid source Nats-Expected-Last-Subject-Sequence appears [v2.10.20]

VeryStrongFingers commented 2 months ago

Observed behavior

Creating a Mirrored stream using a source that contains messages with Nats-Expected-Last-Subject-Sequence headers still performs sequence checks as per the source stream once performed.

Expected behavior

Mirrored streams to ignore Nats-Expected-Last-Subject-Sequence headers (also possibly other 'Expect' headers?).

Existence of the message in the source alone means the original assertion was valid. There is no need for a mirror to have the same expectations and checks as the original source stream.

Alternatively an option for mirrored streams to remove source headers would work too for a resolution. (https://github.com/nats-io/nats-server/pull/5409)

Server and client version

nats-cli: 0.1.5 nats-server: v2.10.20

Host environment

Tested in both Linux/amd64 & Darwin/arm64

Steps to reproduce

# Create stream
nats stream create cool_stream \
    --subjects="cool.*" --storage=file \
    --replicas=1 --retention=limits \
    --discard=old --max-msgs=-1 --max-msgs-per-subject=-1 \
    --max-bytes=-1 --ack --max-age=-1 --max-msg-size=-1 \
    --dupe-window=2m0s --allow-rollup \
    --no-deny-delete --no-deny-purge

# Publish a bunch of messages
nats pub "cool.aaa" --header "Nats-Expected-Last-Subject-Sequence:0"  "a"
nats pub "cool.aaa" --header "Nats-Expected-Last-Subject-Sequence:1"  "b"
nats pub "cool.aaa" --header "Nats-Expected-Last-Subject-Sequence:2"  "c"
nats pub "cool.aaa" --header "Nats-Expected-Last-Subject-Sequence:3"  "d"
nats pub "cool.aaa" --header "Nats-Expected-Last-Subject-Sequence:4"  "e"
nats pub "cool.aaa" --header "Nats-Expected-Last-Subject-Sequence:5"  "f"
nats pub "cool.aaa" --header "Nats-Expected-Last-Subject-Sequence:6"  "g"
nats pub "cool.aaa" --header "Nats-Expected-Last-Subject-Sequence:7"  "h"

# Remove sequence number 5 from stream (message body: e)
nats stream rmm cool_stream 5 -f

# Create mirrored stream (after messages are already published)
nats stream create cool_stream_mirrored \
    --storage=file --replicas=1 \
    --retention=limits --discard=old \
    --max-msgs=-1 --max-msgs-per-subject=-1 \
    --max-bytes=-1 --ack --max-age=-1 \
    --max-msg-size=-1 --dupe-window=2m0s \
    --allow-rollup --no-deny-delete \
    --no-deny-purge --mirror=cool_stream

# Publish some more messages

nats pub "cool.aaa" --header "Nats-Expected-Last-Subject-Sequence:8"  "i"
nats pub "cool.aaa" --header "Nats-Expected-Last-Subject-Sequence:9"  "j"
nats pub "cool.aaa" --header "Nats-Expected-Last-Subject-Sequence:10"  "k"

# View streams
nats stream ls

╭────────────────────────────────────────────────────────────────────────────────────────────╮
│                                           Streams                                          │
├──────────────────────┬─────────────┬─────────────────────┬──────────┬───────┬──────────────┤
│ Name                 │ Description │ Created             │ Messages │ Size  │ Last Message │
├──────────────────────┼─────────────┼─────────────────────┼──────────┼───────┼──────────────┤
│ cool_stream_mirrored │             │ 2024-09-08 21:35:20 │ 4        │ 376 B │ 15m12s       │
│ cool_stream          │             │ 2024-09-08 19:37:21 │ 10       │ 943 B │ 2.25s        │
╰──────────────────────┴─────────────┴─────────────────────┴──────────┴───────┴──────────────╯

cool_stream_mirrored stream will no longer mirror any new messages because of the invalid Nats-Expected-Last-Subject-Sequence header

[36649] 2024/09/08 21:35:20.258473 [WRN] Error processing inbound mirror message for '$G' > 'cool_stream_mirrored': last sequence by subject mismatch: 5 vs 4

Relevant code seems to be

derekcollison commented 2 months ago

Yes this is a known issue we plan on improving in 2.11.

m3nowak commented 1 month ago

There is a similar issue with messages containing the 'Nats-Expected-Stream' header. They won’t be sent to the mirrored stream unless it is named the same as the source stream.

nats-io / nats-server