search

EspressoSystems / hotshot-query-service

Generic query service for HotShot applications
https://espressosystems.github.io/hotshot-query-service/
GNU General Public License v3.0
5 stars 1 forks source link

Profile with high transaction load #615

Closed jbearer closed 4 months ago

jbearer commented 4 months ago

We've noticed in staging that when the throughput gets around 100-200kb/s, we start seeing very high CPU load on query nodes only. A CPU profiler should give some insights as to why

jbearer commented 4 months ago

Profiling shows the query nodes using ~70% of their total CPU consumption computing VID commitments and the remaining ~30% verifying VID shares. Whereas regular nodes use about 85% verifying disperse and the remainder on various miscellaneous small tasks, with no significant contribution from computing commitments. This actually makes perfect sense: it's not the query service that's expensive; it's just that the query nodes are also DA nodes, and DA nodes have to compute an extra VID commitment each time they receive a DA proposal.

This is confirmed by running a DA node which is not a query node (sequencer2 in the native demo); the CPU profile of this node looks like the other DA nodes, not like the other non-query nodes.

jbearer commented 4 months ago

We did test this in staging with larger instances on the DA nodes, and CPU reached only about 40%, putting the bottleneck back on the builder. So it seems like the issue is just computation of the VID commitments, and it's not a blocker