stefanLeo
commented
11 months ago And an update - we fixed a few config issues but are now at a point where the client succesfully reconnects and esubscribes, but with the following results: PULL Consumer: Stuck after Resubscribe PUSH Consumer: SOMETIMES! failover is as expected - we immediately receive messages after the res-subscribe, but SOMETIMES the consumer is stuck waiting for the next message after re-subscriptoin for 22seconds until it receives the next message.
The NATS Server logs only that the "consumer is slow" during the 22 sec blackout.
derekcollison
scottf
What version were you using?
Server: 2.9.22 & 2.10 Java Client: 2.16.14 & 2.17.0
What environment was the server running in?
We used the official NATS container images and the HELM charts for deployment.
Is this defect reproducible?
Setup 3 node cluster on RHAT Openshift or any other Kubernetes Cluster Start producer with settings as described > https://github.com/nats-io/nats.java/blob/main/src/examples/java/io/nats/examples/jetstream/NatsJsPub.java Start consumer with settnigs as adescribed > https://github.com/nats-io/nats.java/blob/main/src/examples/java/io/nats/examples/jetstream/NatsJsPushSubBasicAsync.java Kill Jetstream leader node (find stream leader via using nats cli) The Consumer may not even be directly connected to the jestream leader node.
Given the capability you are leveraging, describe your expectation?
The Consuming client continues consuming messages after a leader has been elected.
Given the expectation, what is the defect you are observing?
We re-used the NATS JetStream Producer and PUSH Consumer examples from https://github.com/nats-io/nats.java/blob/main/src/examples/java/io/nats/examples/jetstream/NatsJsPushSubBasicAsync.java and then killed the NATS leader node of the stream (we forcefully killed the VM hosting the Kube Worker Node).
Setup: In Memory Storage Option, 3 Replicas.
Config of createStream was changed to: StreamConfiguration sc = StreamConfiguration.builder() .name(streamName) .storageType(storageType) .subjects(subjects) .replicas(3) .description("LifeX-Test") .build();
and connection builder to Options.Builder builder = new Options.Builder() .server(servers) .connectionTimeout(Duration.ofMillis(500)) .pingInterval(Duration.ofSeconds(3)) .maxPingsOut(2) .reconnectWait(Duration.ofMillis(500)) .connectionListener(EXAMPLE_CONNECTION_LISTENER) .traceConnection() .errorListener(EXAMPLE_ERROR_LISTENER) .maxReconnects(-1) .reconnectDelayHandler(new PsReconnectDelayHandler()) .reconnectJitter(Duration.ofMillis(500));
When connecting, we configure all 3 servers of the cluster and register connection, error and delay handlers (basically just logging the callbacks).
Setup Environment: NATS Cluster with 3 NATS Pods on top of RHAT Openshift Kubernetes cluster.
After the failure of the NATS master node the following happens:
Producer detects the failure, reconnects and continues sending after ~15 seconds. Consumer DOES NOT detect the failure and is stuck. It does not log any error or disconnection info nor any new message reception in the message handler. Note that if we restart the consumer it can consume ALL message sent by the producer incl. the ones after the producer reconnected. Note as well that the consumer aborts once the producer is done and deletes the stream. Then some disconnect log is printed. Logs of NATS nodes are attached... I cannot really add logs of the java client as there are none as it is seems to just remain stuck indefinitely. UPDATE: Added java client logs with traceConnection settings and now we see more details. The Client seems to reconnect and resubscribe, but still does NOT get any further messages pushed...
See also https://github.com/nats-io/nats.java/issues/997
We think now this is a NATS Server issue.