This is a regression caused by the implementation of the reactive stream support for the stream read operations. (Shame on me. :-) )
Situation
EventStoreDBClient performs a read stream operation and the underlying GrpcClient receives:
a StatusRuntimeExceptionwith the code UNAVAILABLE or ABORTED
a NotLeaderException
Expected bevahior:
The existing gRPC connection (aka the ManagedChannel) is dropped a new conection to a node is created. In case of the NotLeaderException the new connection should be to the node specified in the Exception.
Current behavior:
The exceptions are not received by the GrpcClient and therefore the reconnect is not triggered properly.
Impact
After a leader change in a cluster the client will try to reconnect to the old node and will not switch to the new master, causing operations to fail.
Root cause
Inside the implementation of AbstractRead the error signals of the gRPC connection are only forward to the reactive subscriber but not to the GrpcClient since the result of AbstractRead is always a successfully completed CompletableFuture containing the subscription.
It is necessary to forward the error signals to the reactive subscriber and provide a CompletableFuture for the GrpcClient to receive error signals.
This is a regression caused by the implementation of the reactive stream support for the stream read operations. (Shame on me. :-) )
Situation
EventStoreDBClientperforms a read stream operation and the underlyingGrpcClientreceives:StatusRuntimeExceptionwith the codeUNAVAILABLEorABORTEDNotLeaderExceptionExpected bevahior:
The existing gRPC connection (aka the
ManagedChannel) is dropped a new conection to a node is created. In case of theNotLeaderExceptionthe new connection should be to the node specified in the Exception.Current behavior:
GrpcClientand therefore the reconnect is not triggered properly.Impact
After a leader change in a cluster the client will try to reconnect to the old node and will not switch to the new master, causing operations to fail.
Root cause
Inside the implementation of
AbstractReadthe error signals of the gRPC connection are only forward to the reactive subscriber but not to theGrpcClientsince the result ofAbstractReadis always a successfully completedCompletableFuturecontaining the subscription. It is necessary to forward the error signals to the reactive subscriber and provide aCompletableFuturefor theGrpcClientto receive error signals.