Closed debajyoti-truefoundry closed 2 months ago
We found this issue while debugging a potential orphan message scenario. We were not receiving the messages in the stream from the interested consumer. Even though there were messages in the stream, the consumer was not showing any outstanding acks or unprocessed messages.
On the Slack thread, @ripienaar suggested that there can be some edge cases due to using the rollup feature in a work queue stream, but I have not verified this yet.
We use rollups to replace existing messages from the stream for some subjects if they have not been processed yet.
We could not model this using the per-subject message limit as, in some cases, I need to keep multiple messages with the same subject.
Sometimes, we want a message of a particular subject to replace any existing unprocessed messages in the stream with the same subject.
We decided to use rollups to model this.
We have made some fixes in the area of stream sync. I would suggest trying 2.10.17-RC3 (formal release will be next week).
Cool. We will try it once the release is done on our devtest environment.
On 2.10.17 either ordering is broken or orphans even when rollup is used.
@sammy007 please provide details and insights that lead you to this conclusion.
We are running 2.10.17
and have not noticed this issue again yet. We noticed that 2.10.18
also had related fixes and upgraded on one of our environments. I will close this issue.
Observed behavior
I started a discussion on the Slack channel but opened an issue, too.
I am using NATS
2.10.12-alpine3.19
. I have a stream like this below,As you can see,
1891
messages are in the stream. If I look at filter subjecttfy-agent-state-buffer.tfy-devtest-euwe1.>
But the consumer for the above filter subject shows no pending messages.
Nak
orTerm
in our system on messages.consumer report
command, I do not see any pending or unprocessed messages on any consumer.Our consumption flow looks something like this,
Fetch
. b. We process these 50 messages serially and in order. c. After processing each message, we executeDoubleAck
. We do not process messageN
ifN - 1
is not double-acked successfully. d. Go back to (a) after the batch is processed completely.In a different environment, we are running
nats:2.10.14-alpine3.19
I am noticing the same issue there, too.Expected behavior
The stream will not have any orphan messages.
Server and client version
The server version is mentioned above.
CLI
On the consumption side, we use the Go client
v1.34.1
. On the publisher side, we use"nats.ws": "^1.21.0"
.Host environment
No response
Steps to reproduce
No response