Closed freddyrios closed 2 years ago
Assuming you use TCP transport:
TransmissionBlockSize
to a approx size of a single log entryBufferSize
to approx log entry size
Incremental
compaction mode. In this case, the size of partition doesn't matter and has no effect on performance.InitialPartitionSize
to recordsPerPartition X approx log entry size
MaxConcurrentReads
to the expected number of cluster membersCacheEvictionPolicy
property to LogEntryCacheEvictionPolicy.OnSnapshot
ConnectTimeout < RequestTimeout < LowerElectionTimeout
. ConnectTimeout is a small value that should be enough to establish TCP connectionSome exploration done with the example using fast speed branch + flushoncommit + single replication x 16 entries + connection timeout of 5 ms (times recorded is the time for 1k entries being applied to a node):
Did not explore MaxConcurrentReads because for this setup it should already be set to 3.
Side question: is this type of CPU usage difference expected on the leader vs. followers?
Regarding the errors in point 9, some of the errors said fail in red with this:
fail: DotNext.Net.Cluster.Consensus.Raft.Tcp.TcpServer[74028]
Failed to process request from 169.254.241.37:40800
System.IO.EndOfStreamException: Attempted to read past the end of the stream.
at RaftNode.SimplePersistentState.UpdateValue(LogEntry entry) in C:\Users\fredd\source\repos\dotNext\src\examples\RaftNode\SimplePersistentState.cs:line 87
at DotNext.Net.Cluster.Consensus.Raft.MemoryBasedStateMachine.ApplyAsync(Int32 sessionId, Int64 startIndex, CancellationToken token) in C:\Users\fredd\source\repos\dotNext\src\cluster\DotNext.Net.Cluster\Net\Cluster\Consensus\Raft\MemoryBasedStateMachine.cs:line 456
Regarding the actual entries:
That's normal CPU utilization if the loop doesn't have a delay.
re CPU utilization, I just mean the 2 followers seem to have 1 core at 100%, while the leader does not. That picture of CPU usage remains if I change the leader, its always the leader without any super busy core vs. the followers with a core at 100%.
So another way to phrase the question is whether its normal that the followers only seem to be using a single core for some significant part of the work.
No, I'm not using CPU affinity. Just two things: await and ThreadPool.QueueUserWorkItem
.
Additional findings based on shared and individual additional explorations:
Conclusions: it does not seem currently possible in this version to use parallelization to achieve an order of magnitude more throughput. Having 1 bigger entry instead of multiple smaller entries is thus preferable, which for us means an unified entry with the inputs of all nodes instead of a smaller entry x node. This approach is already the one considered in the pull based approach recommended by @sakno, so no further action is necesary for this.
To get the modified example running stable, we have modified the following parameters:
Note these parameter changes were needed for stability before any changes to the original example write speed, that is, writting an entry once x second.
The thought behind the update for the example was simple: we have entries that are 1k times bigger than the original example, so the values were increased by 1k. Does that make any sense? if not, what are good ways to reason about these parameters and what would be some good values to try for the example.
If one uses ping between the devices there is sub millisecond latency. Any ideas what could be behind needing to run with an election timeout range of 300 - 600ms?
In the 3 node deployment, the nodes are seeing approx. 700 big entries / second (1k every 1.4 seconds). In reality, this is the flow of messages we expect to see:
So, 26 of the big entries + commands/application state + future flow for fault tolerance should end up well below 70 big entries, which is 10 times less than the rate we are seeing on the modified test. Based on this info, how would it make sense to tune the parameters for this smaller data flow (with a bigger amount of nodes, which also means nodes the leader would be sending info to many more systems).