Closed jrovira-kumori closed 2 weeks ago
On a plane atm so hard to run your scripts and docker with not so great wifi. However, I will once back on the ground.
If possible could you swap in the 2.10.18-RC2 image we have prepped and see if the issue still is there?
docker pull synadia/nats-server:2.10.18-RC.2-alpine3.20
I have run the repro multiple times and the issue persists with synadia/nats-server:2.10.18-RC.2-alpine3.20
. I also checked with nats:2.9.25-alpine
with the same results.
Any update on this issue?
Thank you for your time.
we have the same problem last test - nats-server:2.10.18
we have the same problem last test - nats-server:2.10.18
Have been using your reproducible example extensively in tracking down (part of) the issue. Huge kudos, @jrovira-kumori! :tada:
With the above example code and fix I'm not able to reproduce the issue anymore. However, we're continuing the investigation since not all cases of desynced replicas are solved. With the fix it should be more reliable though.
I have tried the latest nightly synadia/nats-server:nightly-20240819
and it works like a charm!
Thank you very much for taking the time to deal with this issue.
Observed behavior
JetStream sometimes does not automatically recover correctly from a forced termination. The original error was found after an OOM kill. Restarting the corrupted instance does not fix the issue. Only removing the persistent volume and restarting the affected instance fixed the issue.
Expected behavior
After a forced termination of the stream leader, the KV bucket should always remain consistent.
Server and client version
Server:
2.10.17
Client:0.1.4
Host environment
Steps to reproduce
I uploaded a minimal reproducible example at
jrovira-kumori/NATS-KV-Corruption
. Requiresdocker
andbash
.