search

nats-io / nats-server

High-Performance server for NATS.io, the cloud and edge native messaging system.
https://nats.io
Apache License 2.0
16k stars 1.41k forks source link

[FIXED] Ghost consumers after failed meta proposal #6088

Closed MauriceVanVeen closed 2 weeks ago

MauriceVanVeen commented 2 weeks ago

If the cc.meta.Propose fails, for example due to restarts, the sa.consumers[ca.Name] = ca would have staged a consumer that will never be successfully added. That consumer would then become part of the next snapshot (for example during shutdown), which meant that that server had a ghost consumer it was unable to clean up.

This does not fully solve the ghost consumers issue, but this should contribute to the final solution.

Signed-off-by: Maurice van Veen github@mauricevanveen.com

derekcollison commented 2 weeks ago

This test seems to fail now on this PR - TestJetStreamClusterConsumerLeak

MauriceVanVeen commented 2 weeks ago

Should there be a test here that shows the change in behavior?

There was already TestJetStreamClusterMaxConsumersMultipleConcurrentRequests that partially covers this. Have extended it a little to also check sa.pendingConsumers gets properly cleared.

MauriceVanVeen commented 2 weeks ago

This test seems to fail now on this PR - TestJetStreamClusterConsumerLeak

At least in CI it seems, locally it passes consistently for me.