search

nats-io / nats-server

High-Performance server for NATS.io, the cloud and edge native messaging system.
https://nats.io
Apache License 2.0
16.01k stars 1.41k forks source link

NATS consumers not pulling messages from Jetstream. Restarting NATS server solved the problem #6017

Open funkye opened 1 month ago

funkye commented 1 month ago

Observed behavior

There are messages for consumers in the stream (confirmed by analyzing the stream file and publisher logging), but the messages do not reach the consumers. NATS CLI also shows that there are no messages for these consumers. BeforeNatsReboot

Restarting the consumers does not change anything. Restarting the server led to the messages being sent to consumers and the NATS CLI began to show other information: AfterNatsReboot

And after consuming the message: AfterNatsRebootMessageProcessed

The problem occurred for 13 out of 100 consumers of one stream. There is nothing suspicious in the server logs for this period (around 15.10.2024 20:30:00) nats_logs.zip

Expected behavior

Messages must be sent to consumers consistently.

Server and client version

NATS server version: 2.10.20, not a cluster .NET client version: 2.2.3

Host environment

OS: Ubuntu 22.04.1

Steps to reproduce

No stable reproduce.

neilalexander commented 1 month ago

Can you please post the nats stream info for the stream too?

It looks like before the restart, the stream sequence moves back down, which makes me think that this might be something to do with #6014. Can you please test v2.10.22 and let me know if that improves things or if the same thing happens?