Investigate performance of Mina node before bootstrap

[georgeee]

Context

In most runs bootstrap and catchup are the longest routines in the process of synchronizing a just-started node.

Bootstrap happens only due to one of the following circumstances:

• when Mina node was offline for a significant amount of time (more than 290 blocks off the main chain)
• when it wasn't possible to load the frontier (e.g. due to abrupt stop of Mina node)

All in all, it is not expected to happen on every run.

Catchup algorithm is known for its inefficiencies and there's a separate stream of work (around integrating Bitswap) to improve catchup time.

However a distinctly visible amount of time is spent by Mina node before bootstrap/catchup start. It's worth to investigate it separately.

Sneak-peak into numbers

While synchronization time varies significantly due to such factors as connectivity (how many healthy peers Mina node was able to establish connection to), hardware (CPU/RAM configuration) and network state (is synchronization being approached at a moment of high transaction traffic), it's useful to look at near-perfect conditions to see what are numbers there.

Using log explorer it's possible to check how different code parts perform when being executed on o1-managed mainnet nodes. These nodes are known to be healthy, long-running, well-resourced etc.

Some observations based on querying the logs:

• It takes around 1m30s to perform computations between log messages Initializing with runtime configuration and Daemon will use chain id <..>
• Initialization of prover and verifier processes takes around 1m each
• Around 20s is spent between messages Initializing transition router and Already connected to enough peers
  • Likely due to waiting for networking component to initialize
• It seems like getting best tip is reasonably capped at 1m (in code) and takes no more than 20s in practice
• It seems like loading transition frontier takes up to 9m
  • Evidenced by the time spent between Already connected to enough peers and Network best tip is recent enough to catchup

TL;DR

Mina node spends ~13min (measured in perfect conditions on mainnet) performing some initialization even before the actual synchronization (bootstrap/catchup) starts.

[mrmr1993]

• It takes around 1m30s to perform computations between log messages Initializing with runtime configuration and Daemon will use chain id <..>

We've found that the biggest input to this is likely to be generating the SRS (and the lagrange basis thereof) to initialise the proof system. We can cache this (and were planning to anyway to speed up unit tests).

• Initialization of prover and verifier processes takes around 1m each

This is the same issue: at this point we use the cached SRS, but still recompute the lagrange basis using a big FFT.

• It seems like loading transition frontier takes up to 9m

Yikes, that seems slow. Is this coming from applying all transactions from the frontier's blocks?