Closed MauriceVanVeen closed 1 year ago
On our k8s cluster (so not running locally with kind) we also see similar behaviour. With less overall messages, but with way more deleted messages.
╭──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ Stream Report │
├────────────────────┬─────────┬───────────┬───────────┬──────────┬─────────┬──────┬─────────┬─────────────────────────┤
│ Stream │ Storage │ Placement │ Consumers │ Messages │ Bytes │ Lost │ Deleted │ Replicas │
├────────────────────┼─────────┼───────────┼───────────┼──────────┼─────────┼──────┼─────────┼─────────────────────────┤
│ interest-stream │ File │ │ 3 │ 4 │ 1.7 KiB │ 0 │ 3334851 │ nats-0, nats-1*, nats-2 │
╰────────────────────┴─────────┴───────────┴───────────┴──────────┴─────────┴──────┴─────────┴─────────────────────────╯
file-based R3 interest stream with max-age=1h
, ~900mps
those messages still get cleaned up with the max-age
this way, but would be better if they would be removed once they have been consumed (as all consumers have consumed every message already)
What server version? Latest fixed a related bug.
What server version? Latest fixed a related bug.
(latest) 2.9.14
Is fixed since 2.9.16
Defect
Make sure that these boxes are checked before submitting your issue -- thank you!
nats-server -DV
outputVersions of
nats-server
and affected client libraries used:OS/Container environment:
Kubernetes (reproduced with kind)
Steps or code to reproduce the issue:
Summary:
It seems to be some kind of de-sync, where some replicas keep the message, but others don't.
To reproduce:
Cluster setup:
Stream setup & bench (with nats-box/cli):
nats str report
:nats str info interest
:nats con ls interest
:Expected result:
Stream is empty, all messages are consumed / no consumers left.
Actual result:
Stream contains some messages, and some deleted.
There doesn't seem to be a consistent pattern, the amount of message that remain in the stream aren't the same when retrying from scratch.
nats-0.zip nats-1.zip nats-2.zip