[Tech Request]: Discrepancy in Commit Wait Times Across tn and cn Nodes

sukki37 commented 1 week ago

Is there an existing issue for the same tech request?

[X] I have checked the existing issues.

Does this tech request not affect user experience?

[X] This tech request doesn't affect user experience.

What would you like to be added ?

During a recent performance profiling session, we identified significant discrepancies in commit wait times across different nodes in the commit chain. Specifically:

The tn node’s call to the logservice client takes approximately 25 seconds, while the overall tn commit process takes around 40 seconds.
The cn commit process takes more than 70 seconds, with a large portion of this time spent on the GetTimestamp operation.

Request：

[ ] Investigate the cause of the high GetTimestamp overhead in the cn commit process.
[ ] Analyze the additional latency in the tn commit process, focusing on identifying potential optimizations in the transaction commit path.
[ ] Explore solutions to reduce the overall commit time across the system, ensuring that each stage of the commit process is optimized for performance.

Why is this needed ?

The 15-second difference between the logservice client call and the overall tn commit time, as well as the additional latency in the cn commit, suggests there are other inefficiencies in the commit chain that need to be addressed.

Additional information

out

tool: https://github.com/iovisor/bcc https://github.com/brendangregg/FlameGraph

command:

/bcc/tools/offcputime.py -df -p "$(pgrep -d ',' test)" 30 > out.stacks
./FlameGraph/flamegraph.pl --color=io --title="Off-CPU Time Flame Graph" --countname=us < out.stacks > out.svg

XuPeng-SH commented 1 week ago

as discussed in the last meeting, logtail serial apply may lead to the accumulation of latency

XuPeng-SH commented 7 hours ago

目前有更高优先级的事情在处理

matrixorigin / matrixone