sakno
commented
11 months ago Currently, the only reason for OperationCanceledException is a token cancellation. According to sources of DataModifier, we have:
var source = CancellationTokenSource.CreateLinkedTokenSource(stoppingToken, leadershipToken);stoppingToken is not a reason I think, because the token represents the lifetime of the hosted service. Inside of ReplicateAsync implementation, passed token is linked with another token associated with RaftCluster lifetime. Moreover, the node doesn't print Saving value {0} generated by the leader node. According to sources, this is possible only if the node is no longer a leader.
freddyrios
This is about the issues being discussed here https://github.com/dotnet/dotNext/issues/184#issuecomment-1660673297.
The issue has been reproduced both in linux arm 32 bit in raspberry os (debian based) and linux debian 64 bit in wsl. At least one reproduction had all 3 nodes running, but due to the investigation of a different issue the cluster was often running with 2 out of the 3 nodes.
There are 2 observations surrounding the issue:
Presumably the frequent leader changes make the issue easier to reproduce, as there is a chance it only happens during leader changes.
The issue has been reproduced in the 14.13, 14.12.3 and 4.4.1 and in all cases it has taken under a day to get to the failure. The specific settings + entries + behavior + even console output shown over ssh could play a part in making the issue more likely to reproduce, as we have systems that run for months with 4.4.1 writing entries to raft 10 times x second.
In 14.13 and 14.12.3 the issue was observed with similar output to https://github.com/dotnet/dotNext/issues/184#issuecomment-1660235577.
I only made one run in 4.4.1 where progress did stop but the output was different. While in the later versions things suddenly stop after the leader change and potentially a single TaskCanceledException (not sure if it always gets into the screen), in the 4.4.1 run there were a various TaskCanceledException that happened before the stop + some "fail" log entries with "EndOfStreamException: Attempted to read past the end of the stream" by the leader.
I share the detailed 4.4.1 in case the issue was still the same and the old version logs extra info. Of course there is a risk the old version failure was not the same type of failure and is irrelevant for the issue present in the latest versions.
Leader:
Follower